[rsyslog] Development of failsafe disk based queue

[david at lang.hm]

On Wed, 1 Oct 2008, Rainer Gerhards wrote:

> On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote:
>> this is already the second version of this systems we develop. Not being
>> able to do error analysis beacause of missing log data was one of the
>> big problems including filesystem crashes.
>
> mhhh... This sounds like you still may be in trouble if the file system
> crashes (as the queue files are stored inside that system).

pick the right filesystem and you should still be safe

ext3+data=journaled is safe

databases do the journaling themselves and like to store their journal 
files on ext2.

you can use a log-structured filesystem that never overwrites data. it 
writes the new sector to a new place on disk then modifies pointers to 
tell the filesystem that the data is in the new place instead of the old 
place (the results in horrible fragmentation in many cases, but it's 
_very_ safe)


some of this can be done without doing a sync for every write with 
barriers, which can guarentee that a write before the barrier takes place 
before any writes after the barrier. this lets you do things like modify 
the queue and then (after you know the data is safe) modify the metadata 
to indicate that the new data is there.

David Lang