[rsyslog] Reducing the time a msg spends in the main unreliablemain queue to a minimum.

Rainer Gerhards rgerhards at hq.adiscon.com
Wed Sep 1 15:51:23 CEST 2010


hmmm... If I look at the debug log, starting with line 1763, it says:

====
9222.964403152:b57bfb90: main Q: entry deleted, state 0, size now 4 entries
9222.964464222:b57bfb90: strm 0x9663c18: file -1(mainq) flush, buflen 420
9222.964531873:b57bfb90: file '/root/rsyslog/spool/mainq.00000001' opened as
#8 with mode 384
9222.964553588:b57bfb90: strm 0x9663c18: opened file
'/root/rsyslog/spool/mainq.00000001' for WRITE as 8
9222.964554042:b57bfb90: strm 0x9663c18: file 8 write wrote 420 bytes
9222.964651904:b57bfb90: main Q[DA]: write wrote 420 octets to disk, queue
disk size now 420 octets
9222.964674507:b57bfb90: main Q[DA]: entry added, size now 1 entries
9222.964696768:b57bfb90: wtpAdviseMaxWorkers signals busy
9222.965203228:b57bfb90: main Q[DA]: EnqueueMsg advised worker start
9222.965226061:b57bfb90: main Q: DAConsumer returns with iRet 0
9222.965249390:b57bfb90: main Q: entry deleted, state 0, size now 3 entries
9222.965285743:b57bfb90: strm 0x9663c18: file 8(mainq) flush, buflen 420
9222.965311318:b57bfb90: strm 0x9663c18: file 8 write wrote 420 bytes
9222.965335196:b57bfb90: main Q[DA]: write wrote 420 octets to disk, queue
disk size now 840 octets
9222.965357551:b57bfb90: main Q[DA]: entry added, size now 2 entries
9222.965379709:b57bfb90: wtpAdviseMaxWorkers signals bus
====

So it starts to write to disk. In line 2797 disk is turned off again, because
there is no more need to use it (empty queue). The it restarts and so on...

Looks like it works as expected. Your producer is too slow to keep rsyslog
busy for a sustained time. You could try the tcpflood utility from the
testbench. It is not documented, but has all relevant information in the file
header comment.

I also guess that the tool you use to monitor the disk is too slow to keep up
with the rate at which rsyslog creates, processes and deletes queue files.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> Sent: Wednesday, September 01, 2010 3:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Reducing the time a msg spends in the main
> unreliablemain queue to a minimum.
> 
> The debug log is here: http://www.omniqueue.com/debug.tar.gz
> 
> The configuration snippet that seems to not able to trigger disk
> assisted queening:
> 
>  $WorkDirectory /root/rsyslog/spool
> $MainMsgQueueSaveOnShutDown on
> $MainMsgQueueType LinkedList
> $MainMsgQueueFileName mainq
> $MainMsgQueueSize 5
> $MainMsgQueueHighWatermark 4
> $MainMsgQueueLowWatermark  2
> $MainMsgQueueMaxDiskSpace 1g
> 
> This is the command I try to stress the main queue to do disk buffering
> with:
> for i in $(seq 1 1000); do `logger test$i &` ; done
> or:
> for i in $(seq 1 1000); do logger test$i  ; done
> 
> 
> Thanks,
> 
> Sivan
> 
> On Wed, Sep 1, 2010 at 2:21 PM, Rainer Gerhards
> <rgerhards at hq.adiscon.com> wrote:
> > This sounds like a bug. Could you provide me a debug log?
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> >> Sent: Wednesday, September 01, 2010 1:20 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Reducing the time a msg spends in the main
> >> unreliablemain queue to a minimum.
> >>
> >> Hi Rainer,
> >>
> >> So I tried following your suggestions but can't seem to make the
> >> MainMsgQueue go to disk at no condition, I want to make it use the
> >> disk for some of the msgs at least , so if power goes off, some msgs
> >> are kept.
> >>
> >> However, this does not seem to happen even when setting the
> >> MainMsgQueue to be DA or Disk only, in DA mode I used quite
> >> restrictive watermarks that I hope will cause it to spool to disk:
> >>
> >> $WorkDirectory /root/rsyslog/spool
> >> # $MainMsgQueueType Disk
> >> $MainMsgQueueSaveOnShutDown on
> >> $MainMsgQueueType LinkedList
> >> $MainMsgQueueFileName mainq
> >> $MainMsgQueueSize 5
> >> $MainMsgQueueHighWatermark 4
> >> $MainMsgQueueLowWatermark  2
> >> $MainMsgQueueMaxDiskSpace 1g
> >>
> >>
> >> When commenting the "type" : "disk" there's still no change in
> >> behavior. To test it I do something like:
> >> for i in $(seq 1 10000); do logger test$i; done
> >>
> >> Any idea what needs to be done to make it use the disk for at least
> >> 50%-70% of the msgs?
> >>
> >> -Sivan
> >>
> >> On Mon, Aug 30, 2010 at 9:19 AM, Rainer Gerhards
> >> <rgerhards at hq.adiscon.com> wrote:
> >> > Hi Sivan,
> >> >
> >> > the main message queue supports the same set of properties and
> >> capabilities
> >> > like the action queue. The directives just start with $MainMsg...
> >> instead of
> >> > $Action...
> >> >
> >> > So you can set the main queue to pure disk mode as well.
> >> >
> >> > Rainer
> >> >
> >> >> -----Original Message-----
> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> >> bounces at lists.adiscon.com] On Behalf Of Sivan Greenberg
> >> >> Sent: Sunday, August 29, 2010 10:44 AM
> >> >> To: rsyslog at lists.adiscon.com
> >> >> Subject: [rsyslog] Reducing the time a msg spends in the main
> >> >> unreliablemain queue to a minimum.
> >> >>
> >> >> Hi List,
> >> >>
> >> >>  I have some critical msgs being transferred using rsyslog.
> Hence, I
> >> >> would like to minimize the time a msg spends in the unreliable by
> >> >> definition main queue. My understanding is that only action
> queues
> >> >> which are where msgs arrive when dispatched from main queue  can
> be
> >> >> set to be disk only. Therefor my assumption that only they can be
> >> made
> >> >> truly reliable.
> >> >>
> >> >> Any suggestions / feedback will be highly appreciated.
> >> >>
> >> >> -Sivan
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> http://www.rsyslog.com
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com



More information about the rsyslog mailing list