[rsyslog-notify] Forum Thread: Re: Dequeue Perfomance - (Mode 'reply')

[noreply at adiscon.com]

User: kamermans 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25482#p25482

Message: 
----------
Here is my current config (this gets pulled in via an include from the main
rsyslog.conf now):

[code:32jh74uc]
###########################
### Ruleset Partition 1 ###
###########################
ruleset(name="forwardPartition1"
        queue.type="LinkedList"

        # The number of messages that can be stored in RAM
        # Theoretical MAX: (15GB RAM / 1.5k msg) == 10,000,000
        # System has crashed with 1 queue at 7.8M
        # Keep this UNDER 6M
        queue.size="500000"

        # rsyslog warns you if this is less than 10% of queue.size
        queue.workerThreadMinimumMessages="50000"
        queue.dequeueBatchSize="750"
        queue.saveOnShutdown="on"
        queue.workerThreads="4"


        # Setting these enables Disk-Assisted mode
        queue.fileName="mycompany_queue"
        queue.spoolDirectory="/var/spool/rsyslog/mycompany_part1"
        # The total amount of disk space that can be used to store
        #  messages that overflow the memory-based LinkedList queue
        # NOTE: queue.size has no effect on the disk-assisted
storage
        queue.maxDiskSpace="35g"
        queue.maxFileSize="1g"
) {
        action( type="omrelp"
                target="10.1.2.3"
                port="15140"
#               action.retryCount="-1"
        )

# Use this instead of the above to log locally
#       action( type="omfile"
#               file="/mnt/log_backup/local-backup.log"
#               ioBufferSize="64k"
#               flushOnTXEnd="off"
#               asyncWriting="on"
#       )

}

###########################
### Ruleset Partition 2 ###
###########################
ruleset(name="forwardPartition2"
        queue.type="LinkedList"
        queue.size="500000"
        queue.workerThreadMinimumMessages="50000"
        queue.dequeueBatchSize="750"
        queue.saveOnShutdown="on"
        queue.workerThreads="4"
        queue.fileName="mycompany_queue"
        queue.spoolDirectory="/var/spool/rsyslog/mycompany_part2"
        queue.maxDiskSpace="35g"
        queue.maxFileSize="1g"
) {
        action( type="omrelp"
                target="10.1.2.3"
                port="15140"
        )

}


######################
### Logging Inputs ###
######################

input(  type="imudp"
        port="514"
        ruleset="forwardPartition1"
)

input(  type="imudp"
        port="515"
        ruleset="forwardPartition2"
)
[/code:32jh74uc]

I am currently using only [b:32jh74uc]forwardPartition1[/b:32jh74uc].  I
have reduced the threads from 6 to 4 per your original recommendation that
6 was too high.  I don't mind decreasing it to 1 if you think that's a good
idea.

Right now I am doing 3400 req/sec and it's holding strong (the dip was
expected from the application):


[img:32jh74uc]http://s9.postimg.org/3wvmxyqsv/stats4.png[/img:32jh74uc]

You can see that the queue does regularly push the max up, but do far it is
clearing it quickly.