[rsyslog] Development of failsafe disk based queue

[Rainer Gerhards]

On Wed, 2008-10-01 at 08:48 -0700, david at lang.hm wrote:
> On Wed, 1 Oct 2008, RB wrote:
> 
> >> I guess you've seen in the discussion (well, if you read all these
> >> mails, I just noticed how many they were...)
> >
> > ;-)  I was wondering why my hip was rattling during the entire commute today.
> >
> >> persisted (because that completes the "queue transaction"). I will
> >> verify with the code, but my current guess is that no more than 10 lines
> >> of code will be needed to support this functionality. If so, I think it
> >> is worth it.
> >
> > If it's something you can solve without worrying about block-level
> > writes and whether the underlying drive (if indeed there is even a
> > "drive") has a battery-backed cache or sufficient capacitor charge to
> > write your data, I'm all for it.  Even better if it's POSIX.
> >
> > Although a block driver and other filesystem-bypassing solutions may
> > be interesting in limited cases, I'd rather not see anyone stab their
> > wife over it.
> 
> I agree that trying to bypass the filesystem is highly questionable, and 
> not something for a core change (as always a contributed vrsion can be 
> tested to see if it makes a difference)

Just FYI: preliminary analysis indicates that is probably around the 10
lines of code that need to be added in stream.c, maybe a few more. The
idea is that I can set a flag similar to O_SYNC on stream creation but
then sync when the "atomic" writes are done. This may save some few
cycles over an O_SYNC open(). However, there need to be a few config
settings, which in turn need to be passed down to the queue and stream
class. That adds more code, maybe around 100 lines (the config interface
needs to be redone, thus the many LOC required, it's on the todo
list...). An alternative is to use a simple global $AllWritesSync on/off
option, which would probably be sufficient and cut down changes
required.

For a robust implementation, some more analysis is required (including
thinking about the implications of fsync()). So it is not totally
trivial, but well doable. I am just not sure if I'll do it immediately,
there are many things in the work queue. Testing effort is probably much
bigger than implementation effort, there are so many cases to check
out...

Rainer