[rsyslog] audit-grade queue / batching

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, May 15, 2009 7:53 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] audit-grade queue / batching
> >>>>>> one final thought, if a message gets a permanent 
> failure and backup
> >>>>>> processing kicks in, can this still be audit-grade?
> >>>>>
> >>>>> I doubt, but it probably depends on your legislation (or other
> > policies).
> >>>>
> >>>> what I was meaning with this questions is if shifting 
> from the primary
> >>>> delivery to the backup processing, would this still be the
> >>>> 'ultra-reliable' mode where if the system dies before 
> the log message
> > gets
> >>>> to the backup destination it will not be lost, but 
> instead retried?
> >>>
> >>> I misunderstood your question. Now thought a bit about 
> the scenarios. You
> >> are
> >>> right, this does not work with message-permanent 
> failures. This case
> >>> previously never existed and so - in theory - any message that was
> > enqueued
> >>> could also be processed, even if potentially much later. 
> Now we have
> >> messages
> >>> that are enqueued and thus taken off the previous queue AND action
> >> processing
> >>> but may still fail.
> >>>
> >>> I see two approaches at handling this:
> >>>
> >>> a) we enable an action to configure a backup file that 
> shall receive all
> >>> message permanent failures. This is simple (not only to 
> implement but to
> >>> configure and understand)
> >>>
> >>> b) we push the failed message back to the main queue, but with an
> > indication
> >>> that it failed in an action. This is harder to implement and most
> >> importantly
> >>> harder to understand/configure, but more flexible
> >>>
> >>> Do you see any other possibilities? Which one would you prefer?
> >>
> >> a definantly seems easier.
> >
> > I, too, think this is probably the best option. So I guess 
> we'll go for that.
> >
> > I have also updated the design doc a bit today (but not yet 
> applied all your
> > changes). The most important new content is on page 23ff, 
> including a rough
> > spec of the ultra-reliable disk store (I even being to feel 
> that implementing
> > it is less effort than changing the sequential queue store, 
> but that needs to
> > be changed in any case...).
> 
> I think your plan for the new disk queue is working a bit 
> harder than it 
> needs to, but I will resist commenting more until you decide 
> to focus on 
> it.

Sounds reasonable, I'll ping you when I approach it...

Rainer