[rsyslog] Development of failsafe disk based queue

[Rainer Gerhards]

On Wed, 2008-10-01 at 06:40 -0700, david at lang.hm wrote:
> On Wed, 1 Oct 2008, Rainer Gerhards wrote:
> 
> > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote:
> >
> >> one possible thing is tht if the write has not completed then the system
> >> sending you the logs has not received confirmation that you have the log
> >> yet, so they are the ones responsible for it.
> >
> > I concur. But as I understood the scenario here, the log messages are
> > emited from a process *inside* the failing machine. So that process
> > fails, too, and we do not have any interim that has a copy of the data.
> > So if it is not persisted to disk, it is lost. Anyhow, this requirement
> > has been relaxed in later posting.
> >
> > Also note that I was just thinking about the physical layer, considering
> > a single physical write - far away from rsyslog.
> >
> >> it's only after you acknowledge the message (via relp or equivalent) that
> >> you are required to not loose the log message.
> >
> > I concur and this is how RELP handles this. Well, actually I think there
> > currently is a very slight (but still existing) window of exposure. I
> > think RELP acks when the message is submited to the queue engine. That
> > does not necessarily mean the message is already present on disk.
> 
> right now it doesn't make a difference (since the queue doesn't sync the 
> data), but if/when this is added checking that the call to queue the data 
> doesn't return until after that point should be done.

I just checked the code. Actually, librelp already ensures this. It acks
only after the callback has successfully completed.

> 
> > Also,
> > I think some mild duplication of messages may happen with RELP in a
> > power fail scenario. It is not doing a two-phase commit, thought it
> > tries very hard to get a perfect understanding of what is written and
> > what not. I could check this with spec/code, but I think this is not
> > justified at this point in time ;)
> 
> no matter what checks you do, it's always possible for things to fail 
> after succeeding on doing the work and before the acknowlegement gets back 
> to the sender (with relp, if the receiving machine sends the 
> acknowledgement, but the sending machine crashes before fully processing 
> it, the receiving machine has no way of detecting this)

RELP has a "Window of uncertainty", in which peers negotiate where to
restart. But power failure was not on my spec list, so this only works
if the shutdown was relatively clean.

> 
> on the other hand, this is fairly easy to deal with by making messages 
> include a sequence number to guarentee that they are all unique, and then 
> have something filter out duplicates later.

well said - among others one reason why RELP currently is not receiving
more attention that just letting it mature...

Rainer
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com