[rsyslog] audit-grade queue / batching

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, May 15, 2009 1:53 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] audit-grade queue / batching
> 
> On Thu, 14 May 2009, Rainer Gerhards wrote:
> 
> > Hi all, and especially David,
> >
> > contrary to what I intended, I have gone back to the text processor to
> sketch
> > down what I intend to do. It turned out that looking at the code and
doing
> > small micro-modifications was counter-productive without a "big picture"
(a
> > sign of the magnitude of the change... ;)).
> >
> > So I have updated the design document, especially added section 4.6
(still,
> > it contains a lot of not-yet-updated information!).
> >
> > http://www.rsyslog.com/download/design.pdf
> >
> > One important design decision is documented at the end of 4.6.2 and I
would
> > appreciate feedback on it. Note that even 4.6.2 is not yet fully
completed,
> > but what is there is consistent. It will probably take me at least
another
> > day to get the full intended design for the ultra-reliable queue into the
> > document (and make my mind clear about it). But, again, what is in the
> > document is consistent, "just" some algorithms are missing. So please do
not
> > wonder that not everything is described.
> 
> I'm looking through it now, the definition of 'audit-grade' looks
> reasonable. when using the example of the disk system, I would probably
> have said that it should be redundant (usually implemented as a RAID
> array). then in the area where you emphisize that all components need to
> be audit-grade, you can mention having a RAID card that doesn't have
> battery-backed cache as an easily overlooked item that would undermine the
> entire system.

The disk was just intended as a sample. Even with a single RAID controller
you are not save IMHO - I have seen controllers die, and this sometimes with
very negative effects for the attached disks. So the question always remains
how far one most go (on my blog is a comment with another extreme sample, a
missile hitting a data center...).

However,  this section does not intend to convey how to build an audit-grade
system but rather that even if something is verified to be audit-grade, it
depends on the components used if the resulting system really is.

I think it would probably useful to add a whole chapter "how to build an
audit-grade system" to the paper, as this is something that is easy to do
wrong. So advise will probably be helpful even to experienced folks (digging
into the specifics of rsyslog).

> in section 3.1 what do you mean by "Further, no context switch will happen
> between calls to doAction() and endTransaction()"? the term 'context
> switch' has many meanings, what are you intending here?

You are right, I need to clarify that. I mean change in transaction context.
We cannot mix two transactions, else we will get into real trouble labeling
which messages are done and which not. 

> 
> 
> 
> 
> 4.1.1, if doAction() will implicitly call beginTransaction() if it hasn't
> been called yet, I still am not seeing the value in exposing
> beginTransaction() explicitly. what advantage is there to ever calling it
> rather than just letting the first doAction call it?
> 
> how much value is there in allowing the doAction() call to not commit the
> current message being submitted? it adds significantly to the
> complications in dealing with status. allowing this does let the output
> module run closer to the end of the buffer (without this ability
> doAction() would have to consider it's buffer full if there is not enough
> space for a max-size message after adding the current message), but is
> this worth the complication?
> 
> doing this would simplify the pseudocode for prepareAction() to be
> 
> def prepareAction():
>    if state == rtry:
>      try recovery (adjust state accordingly)
> 
> 
> 
> 
> I don't see how the return codes indicate an auto-commit _with_ m sub n
> (earlier you say the auto-commit may happen with or without the current
> message)
> 

That's the old, unedited text. It should probably not read doAction() but
processAction(). I'll also add a section on rsyslog's minidriver model, as
this is the real reason why we have the three entry points and also retry
logic as it is.

> 
> in section 4.3.2 you say (probable english translation issue)
> 
> "in spite of the two different failure cases, different handling is needed
> for them"
> 
> this doesn't make sense to me, if they are different failure cases, I
> would start off assuming that different handling would be needed. I would
> have said something like "because there are two different categories of
> failure causes, different handling is needed for them"


That was the intension, thanks for the text - will replace mine ;)

> 
> 
> 4.6.2, looking at the to-delete list, my reaction is OUCH!! (there is
> definantly a need for a non-sequential disk queue. but we don't want to go
> there now)

Yep - that's why I wrote about the history and design of the queue. For what
it is designed, it works well. For audit-grade... Something else would be
better.

I'll probably try to condense some thoughts on a potential new storage
system, as I begin to get a clear picture of it. It's probably worth the
extra effort to write it down, as this can be a big time safer once (when?)
we get to it. A bit of a problem is that all this takes time and thus pushes
away quick implementation of the urgent needs. I am prepared to do that (via
Adiscon sponsoring), but I am not sure when  we get into trouble with
deadlines. On the other hand, it makes perfect sense to do it as good as
possible and we have a very, very good discussion right now.

> 
> in the meantime, since it is allowable to duplicate messages in case of
> power failure, the to-delete list can be stored in ram (as opposed to
> needing to be stored on disk)

Yep

> 
> as an alturnative, instead of trying to store batch IDs (with all the
> problems that you accuratly describe), could you use queue slot numbers?
> instead of saying 'delete batch X' you just say 'delete slots 4-7'
> 
> or am I missing something here?

Yes - I don't have slot numbers inside the storage subsystem (!), especially
not with disk mode. Look at the storage subsystem comparison table. You'll
see that all I have is a file number and offset (and even this is bulky to
work with).

> 
> although since that can't work for sequential queues and you have to do a
> to-delete list for those, you are probably right that the right answer for
> now is to do the to-delete queue and allow future queue store definitions
> to replace that call.

Ok, I should have read this first ;)

> one final thought, if a message gets a permanent failure and backup
> processing kicks in, can this still be audit-grade?

I doubt, but it probably depends on your legislation (or other policies).

> 
> I'm thinking along the lines of inserting into a database, with a backup
> of writing the bad message to a local file.

Yes, I think this is needed. Note that rsyslog even currently support this!
You define an action unit with the second action being set to
$ActionExecOnlyWhenPrevousIsSuspended (or similar ;)) to "on" and it being a
write to file action. Than, it takes those messages that the first one could
not write. "IsSuspended" is a bit misleading, but that was the historic use
case. "HasFailed" would now probably be a better name. But the logic works.

> 
> 
> overall this looks very good to me.

Thanks for the review, very helpful.

Rainer
> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com