[rsyslog] Development of failsafe disk based queue

[Rainer Gerhards]

On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote:
> On Wed, 1 Oct 2008, Rainer Gerhards wrote:
> 
> > One thing I forgot to mention: a pure disk queue (not a disk-assisted
> > one) gets you as close to your goal as possible (well, mostly - we
> > could, at a considerable performance expense, require synced writing).
> > With that case, all data is immediately stored on disk. You can
> > configure it to also write the meta data out immediately (and again with
> > sync, not yet supported). However, you still have a window of exposure,
> > for example if the power loss happens right in the middle of when the
> > disk actually writes data to the disk sector.
> >
> > I still wonder why this scenario would be useful to address...
> 
> not all uses of rsyslog are for simple system logs. it's a good general 
> purpose log tool, and there are some cases where you want to be as sure as 
> you possibly can be that once a message has been acknowledged it has no 
> chance of being lost.

I designed the engine for audit-class reliability. However, I assumed
that the rest of the system is also playing in that class. Doing
everything with a potential power failure in mind creates a lot of extra
demands. And I have never heard of anybody doing serious datacenter work
without a proper UPS. Is this *really* an issue?

Rainer
> 
> useing some form of solid-state reliable storage (battery backed ram on a 
> raid controller, a battery backed ram disk, a flash disk) it is possible 
> (but not nessasarily cheap) to get the ability to do tens to hundreds of 
> thousands of writes + syncs per second
> 
> David Lang
> 
> > Rainer
> >
> > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote:
> >> Hi,
> >>
> >> I am looking for a failsafe solution to store syslog messages localy
> >> until they could be send later. I already looked at the disk based
> >> memory queue and the disk based queue. Both queue's don't work if you
> >> just power down the system immediatly actually loosing the whole queue.
> >> I already looked at queue.c and it seemed to me that both queues were
> >> not designed for that kind of failure, but I could be wrong there. Since
> >> an immediate power down of the system is the major failure which will
> >> occure pretty often I need to create a soltution there.
> >>
> >> Did you already start to develop something addressing that problem?
> >> Could you help me extend rsyslog (3.18.4) so that I can develop a new
> >> queue myself? I would contribute the code to the rsyslog project if you
> >> would like afterwards.
> >>
> >> bye
> >> David Ecker
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com