[rsyslog-notify] Forum Thread: Re: Dequeue Perfomance - (Mode 'reply')

[noreply at adiscon.com]

User: kamermans 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25497#p25497

Message: 
----------
Well, I've hit a new record on this server, pushing over 4300 req/sec.  I'm
still using the single-thread config that I posted last.

It's not a great test day since the traffic was so high, but in any case,
I'm still queuing like crazy.  This graph shows the LinkedList cache.  I'm
writing a full-blown stats querying tool that will let me pull out a graph
to track Disk Assisted queues as well so I can see them on a single graph.


[img:2xfw3twg]http://s7.postimg.org/bm3mhu1h7/stats5.png[/img:2xfw3twg]

As you can see, the queue is regularly hitting the 500k LinkedList limit
and sending things into the DA queue (which is currently at 0, but enqueued
6557250 messages today).

Here's the CPU usage:

[img:2xfw3twg]http://s8.postimg.org/u70a5l69h/stats6.png[/img:2xfw3twg]

And this gets me back to my original point, that all my CPU cores are
basically idle, but rsyslog is falling way behind on the dequeue process.

Here's my most recent pstats output in case you're curious:
[code:2xfw3twg]
Wed May  6 01:31:39 2015: action 1: origin=core.action
processed=140544218 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 2: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 3: origin=core.action
processed=1560 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 4: origin=core.action
processed=156 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 5: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 6: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: action 7: origin=core.action
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed May  6 01:31:39 2015: imudp(*:514): origin=imudp
submitted=148105883
Wed May  6 01:31:39 2015: imudp(*:514): origin=imudp
submitted=0
Wed May  6 01:31:39 2015: imudp(*:515): origin=imudp
submitted=0
Wed May  6 01:31:39 2015: imudp(*:515): origin=imudp
submitted=0
Wed May  6 01:31:39 2015: resource-usage: origin=impstats
utime=5447328185 stime=3497084375 maxrss=10909376 minflt=2526070 majflt=0
inblock=0 oublock=4816 nvcsw=397703354 nivcsw=33643
Wed May  6 01:31:39 2015: forwardPartition1[DA]:
origin=core.queue size=0 enqueued=7170750 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed May  6 01:31:39 2015: forwardPartition1:
origin=core.queue size=391665 enqueued=148105883 full=0
discarded.full=0 discarded.nf=0 maxqsize=450849
Wed May  6 01:31:39 2015: forwardPartition2[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed May  6 01:31:39 2015: forwardPartition2:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0
discarded.nf=0 maxqsize=0
Wed May  6 01:31:39 2015: main Q: origin=core.queue
size=0 enqueued=1716 full=0 discarded.full=0 discarded.nf=0
maxqsize=2
Wed May  6 01:31:39 2015: imudp(w1): origin=imudp
called.recvmmsg=129364726 called.recvmsg=0
msgs.received=30442520
Wed May  6 01:31:39 2015: imudp(w0): origin=imudp
called.recvmmsg=253019822 called.recvmsg=0
msgs.received=117663363
[/code:2xfw3twg]