[rsyslog] Development of failsafe disk based queue

[david at lang.hm]

On Wed, 1 Oct 2008, Rainer Gerhards wrote:

> One thing I forgot to mention: a pure disk queue (not a disk-assisted
> one) gets you as close to your goal as possible (well, mostly - we
> could, at a considerable performance expense, require synced writing).
> With that case, all data is immediately stored on disk. You can
> configure it to also write the meta data out immediately (and again with
> sync, not yet supported). However, you still have a window of exposure,
> for example if the power loss happens right in the middle of when the
> disk actually writes data to the disk sector.
>
> I still wonder why this scenario would be useful to address...

not all uses of rsyslog are for simple system logs. it's a good general 
purpose log tool, and there are some cases where you want to be as sure as 
you possibly can be that once a message has been acknowledged it has no 
chance of being lost.

useing some form of solid-state reliable storage (battery backed ram on a 
raid controller, a battery backed ram disk, a flash disk) it is possible 
(but not nessasarily cheap) to get the ability to do tens to hundreds of 
thousands of writes + syncs per second

David Lang

> Rainer
>
> On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote:
>> Hi,
>>
>> I am looking for a failsafe solution to store syslog messages localy
>> until they could be send later. I already looked at the disk based
>> memory queue and the disk based queue. Both queue's don't work if you
>> just power down the system immediatly actually loosing the whole queue.
>> I already looked at queue.c and it seemed to me that both queues were
>> not designed for that kind of failure, but I could be wrong there. Since
>> an immediate power down of the system is the major failure which will
>> occure pretty often I need to create a soltution there.
>>
>> Did you already start to develop something addressing that problem?
>> Could you help me extend rsyslog (3.18.4) so that I can develop a new
>> queue myself? I would contribute the code to the rsyslog project if you
>> would like afterwards.
>>
>> bye
>> David Ecker
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>