[rsyslog] audit-grade queue / batching

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, May 15, 2009 5:35 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] audit-grade queue / batching
> 
> On Fri, 15 May 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Fri, 15 May 2009, Rainer Gerhards wrote:
> >>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>
> >>>> one final thought, if a message gets a permanent failure and backup
> >>>> processing kicks in, can this still be audit-grade?
> >>>
> >>> I doubt, but it probably depends on your legislation (or other
policies).
> >>
> >> what I was meaning with this questions is if shifting from the primary
> >> delivery to the backup processing, would this still be the
> >> 'ultra-reliable' mode where if the system dies before the log message
gets
> >> to the backup destination it will not be lost, but instead retried?
> >
> > I misunderstood your question. Now thought a bit about the scenarios. You
> are
> > right, this does not work with message-permanent failures. This case
> > previously never existed and so - in theory - any message that was
enqueued
> > could also be processed, even if potentially much later. Now we have
> messages
> > that are enqueued and thus taken off the previous queue AND action
> processing
> > but may still fail.
> >
> > I see two approaches at handling this:
> >
> > a) we enable an action to configure a backup file that shall receive all
> > message permanent failures. This is simple (not only to implement but to
> > configure and understand)
> >
> > b) we push the failed message back to the main queue, but with an
indication
> > that it failed in an action. This is harder to implement and most
> importantly
> > harder to understand/configure, but more flexible
> >
> > Do you see any other possibilities? Which one would you prefer?
> 
> a definantly seems easier.

I, too, think this is probably the best option. So I guess we'll go for that.

I have also updated the design doc a bit today (but not yet applied all your
changes). The most important new content is on page 23ff, including a rough
spec of the ultra-reliable disk store (I even being to feel that implementing
it is less effort than changing the sequential queue store, but that needs to
be changed in any case...).

http://www.rsyslog.com/download/design.pdf

I have now begun to go back to code, checking out subtle issues I have
overlooked. At first guess, the code will definitely become simpler at some
places, so the overall complexity should stay more or less constant.

I expect that until mid-next week I will now work on the code - except I see
something that requires design change. If everything goes well, we may than
even have a first rough implementation of the ultra-reliable capability (but
don't take that for guaranteed). At the detail layer, there are still a lot
of subtle issues. The worst thing is probably adapting the current disk queue
to the new queue layer design.

This just for info.

Rainer