[rsyslog] Development of failsafe disk based queue

Rainer Gerhards rgerhards at hq.adiscon.com
Wed Oct 1 14:09:33 CEST 2008 

I am currently working on the queue engine in the devel branch. In that
light, it could make sense to start with that branch. Also, I won't make
any non-bug related changes in the stable version, so if you run into
anything I can quickly change, that will happen in devel.

Rainer

On Wed, 2008-10-01 at 14:05 +0200, David Ecker wrote:
> Hi,
> 
> should I use 3.18.4 (latest stable, I would preferr that one) or do I
> need the latest development version? I would actually alter queue.c
> directly changing the fctl flags in the disk based queue
> (O_DIRECT,O_SYNC,.O_NOATIME).
> 
> Performance is not really an issue. There will be only 1000 to 2000
> Messages per hour in peak times
> 
> bye
> David Ecker
> 
> Rainer Gerhards schrieb:
> > David,
> >
> > the file syncing mentioned in the compatibility doc applies to the
> > output action, only. 
> >
> > The queue does never do synchronous writes - I always assumed that a
> > critical system would have a UPS and could never think (so far) about a
> > valid reason for not having it. So the queue would need to have an extra
> > option to do sync writes. Obviously, that's not a big deal.
> >
> > Performance, of course, will be extremely terrible with such a setup...
> >
> > Rainer
> >
> > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote:
> >   
> >> On Wed, 1 Oct 2008, David Ecker wrote:
> >>
> >>     
> >>> Hi,
> >>>
> >>> I am looking for a failsafe solution to store syslog messages localy
> >>> until they could be send later. I already looked at the disk based
> >>> memory queue and the disk based queue. Both queue's don't work if you
> >>> just power down the system immediatly actually loosing the whole queue.
> >>>       
> >> are you sure about the disk based queue?
> >>
> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue 
> >> can be set to do a commit of the metadata after each message.
> >>
> >> Disk Queues
> >>
> >> Disk queues use disk drives for buffering. The important fact is that the 
> >> always use the disk and do not buffer anything in memory. Thus, the queue 
> >> is ultra-reliable, but by far the slowest mode. For regular use cases, 
> >> this queue mode is not recommended. It is useful if log data is so 
> >> important that it must not be lost, even in extreme cases.
> >>
> >> When a disk queue is written, it is done in chunks. Each chunk receives 
> >> its individual file. Files are named with a prefix (set via the 
> >> "$<object>QueueFilename" config directive) and followed by a 7-digit 
> >> number (starting at one and incremented for each file). Chunks are 10mb by 
> >> default, a different size can be set via the"$<object>QueueMaxFileSize" 
> >> config directive. Note that the size limit is not a sharp one: rsyslog 
> >> always writes one complete queue entry, even if it violates the size 
> >> limit. So chunks are actually a little but (usually less than 1k) larger 
> >> then the configured size. Each chunk also has a different size for the 
> >> same reason. If you observe different chunk sizes, you can relax: this is 
> >> not a problem.
> >>
> >> Writing in chunks is used so that processed data can quickly be deleted 
> >> and is free for other uses - while at the same time keeping no artificial 
> >> upper limit on disk space used. If a disk quota is set (instructions 
> >> further below), be sure that the quota/chunk size allows at least two 
> >> chunks to be written. Rsyslog currently does not check that and will fail 
> >> miserably if a single chunk is over the quota.
> >>
> >> Creating new chunks costs performance but provides quicker ability to free 
> >> disk space. The 10mb default is considered a good compromise between these 
> >> two. However, it may make sense to adapt these settings to local policies. 
> >> For example, if a disk queue is written on a dedicated 200gb disk, it may 
> >> make sense to use a 2gb (or even larger) chunk size.
> >>
> >> Please note, however, that the disk queue by default does not update its 
> >> housekeeping structures every time it writes to disk. This is for 
> >> performance reasons. In the event of failure, data will still be lost 
> >> (except when manually is mangled with the file structures). However, disk 
> >> queues can be set to write bookkeeping information on checkpoints (every n 
> >> records), so that this can be made ultra-reliable, too. If the checkpoint 
> >> interval is set to one, no data can be lost, but the queue is 
> >> exceptionally slow.
> >>
> >> Each queue can be placed on a different disk for best performance and/or 
> >> isolation. This is currently selected by specifying different 
> >> $WorkDirectory config directives before the queue creation statement.
> >>
> >> To create a disk queue, use the "$<object>QueueType Disk" config 
> >> directive. Checkpoint intervals can be specified via 
> >> "$<object>QueueCheckpointInterval", with 0 meaning no checkpoints.
> >>
> >>
> >>
> >>
> >>
> >> you also need to specificly enable syncing (from 
> >> http://www.rsyslog.com/doc-v3compatibility.html )
> >>
> >> Output File Syncing
> >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As 
> >> such, it retained stock syslogd's default of syncing every file write if 
> >> not specified otherwise (by placing a dash in front of the output file 
> >> name). While this was a useful feature in past days where hardware was 
> >> much less reliable and UPS seldom, this no longer is useful in today's 
> >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd 
> >> writes files around 50 *times* slower than without it. It also affects 
> >> overall system performance due to the high IO activity. In rsyslog v3, 
> >> syncing has been turned off by default. This is done via a specific 
> >> configuration directive "$ActionFileEnableSync on/off" which is off by 
> >> default. So even if rsyslogd finds sync selector lines, it ignores them by 
> >> default. In order to enable file syncing, the administrator must specify 
> >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that 
> >> syncing only happens in some installations where the administrator 
> >> actually wanted that (performance-intense) feature. In the fast majority 
> >> of cases (if not all), this dramatically increases rsyslogd performance 
> >> without any negative effects.
> >>
> >>
> >>
> >>     
> >>> I already looked at queue.c and it seemed to me that both queues were
> >>> not designed for that kind of failure, but I could be wrong there. Since
> >>> an immediate power down of the system is the major failure which will
> >>> occure pretty often I need to create a soltution there.
> >>>       
> >> with checkpoint interval set to 1 and syncing enabled the data should be 
> >> in on the disk safely (assuming you have hardware that supports this) and 
> >> a power-off won't affect it.
> >>
> >> David Lang
> >>
> >>
> >>
> >>     
> >>> Did you already start to develop something addressing that problem?
> >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new
> >>> queue myself? I would contribute the code to the rsyslog project if you
> >>> would like afterwards.
> >>>
> >>> bye
> >>> David Ecker
> >>>
> >>>       
> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
> >>     
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >   
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

More information about the rsyslog mailing list