[rsyslog] output plugin calling interface

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, May 01, 2009 10:14 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] output plugin calling interface
> 
> On Fri, 1 May 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of 
> david at lang.hm
> >>>> 1. something wrong with a message.
> >>>>     retries won't help (with batches we need to narrow down
> >>>> to which message)
> >>>
> >>> But what could be wrong? For which outputs?
> >>>
> >>> Can you envison anything other than db outputs only,
> >> duplicate identity in
> >>> column errornously declared with a unique index (which I
> >> still consider an
> >>> admin error)? I really fail to find any other sample.
> >>
> >> things like quoting errors (which I know you defend against)
> >>
> >> unicode text causing grief
> >>
> >> dynafile hits a read-only file
> >>
> >> basicly data-driven things that trigger bugs in the 
> message delivery
> >> mechanism in some form.
> >>
> >> this is easiest to understand and provide examples with
> >> databases, but you
> >> could have problems with other methods.
> >>
> >> in an ideal world these would never happen, but for most
> >> output types I
> >> can think of some form of corrupt input that could cause that
> >> message to
> >> fail.
> >
> > OK, thanks a lot. It doesn't matter if I can defend a case 
> or two, but you
> > provided good examples. I think I now can be relieved that 
> we have sufficient
> > probability to seriously look at these cases.
> >
> > Resting sometimes helps, and so I also think I made a big 
> step forward (while
> > not actively working on it ;)) today. I think what makes 
> the problem so hard
> > to solve is the language we use. I thought about a purely 
> mathematical model,
> > and I have one on my mind. The state diagram was a first 
> step, but it went
> > not far enough. So I think I will try to write up that 
> model and then we can
> > discuss based on it and finally derive the actual code from 
> it. That's an
> > extra step, but I think it will be a useful one.
> >
> > As a side-note, I have also identified that we have 
> overlooked a subtle issue
> > so far: backup actions - they need to work on the subset of 
> the batch that
> > had message permanent failures. So the message state 
> actually needs to be
> > part of the message inside the batch. But now, I think, 
> things really begin
> > to come together and are far less complex than initially thought.
> >
> > One problem with the state chart - that was why I said it 
> is not 100% correct
> > - is that it does not properly abstract batches vs. single 
> messages. Both of
> > them entangled in a way that I thought [;)] to be very 
> complex. But if you
> > model that with processing states, then the batch 
> processing state is simply
> > a function of the individual message processing states.
> >
> > Please let me know if you also find a math model useful 
> (but I'll probably
> > need to do it in any case, because it helps me clean up my mind...).
> 
> I think it will help clarify things a lot. with a good model 
> we won't have 
> misunderstandings about what we are talking about.

Yes - and I also think that with the model some complexities disappear. I
think (hope I am right) the solution will become obvious. I know I am
investing a lot of time in a tiny portion of the code, but this is one of the
core elements involving many complexities.

> with my 'binary search' approach, handling permanently bad 
> messages could 
> be as simple as 'too many retries once we hit a batch size of 
> 1' (with a 
> possible option of the output module reporting back that it dectected 
> something that makes retries useless, but this is just an 
> optimization)

Yes, indeed. One quick thought: I see a batch as a set of (msg, state)
ordered pairs. Once we have procssed it in one action (all of them have
entered one permanent state), we can than build a subset that we use as the
new (remaining) batch in the backup actions. So the "bad record search" is
"just" one facet of many that we need to handle with little and hopefully
simple code (doing it with 2000 LoC would be rather easy ;)).

Rainer


> 
> David Lang
>