[rsyslog] output plugin calling interface

[Rainer Gerhards]

One quick question:

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, May 06, 2009 6:07 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] output plugin calling interface
> 
> > I agree with you on the performance. I disagree that this means the
output
> > transaction and the transaction from the upper layer must exactly match.
> > Think RELP. Would that mean that a relp window must always be as large as
> the
> > largest batch? OK, in this case I control the protocol and so could
change
> > it. But what with a SNMP trap? Or a rfc3195 conversation? They *have*
> > different notation of a transaction. So the best thing I can do IMHO is
> > permit the output plugin to tell the engine when its transaction was
> > finished. That's no problem for the upper layer, it just needs to mark
these
> > messages as committed. But it is simply impossible for all outputs to
have
> > the same idea of transaction than the upper layer may have.
> 
> I guess I see the upper layer having a larger definition of a batch than
> the lower level to be a configuration error.

The lower level limit is not a config setting but rather something that
"happens to be that way". RELP by design uses a sliding window, so the
"lower-level" batch is a moving target. Some for many other outputs.

> not one to refuse to boot with, but one that could result in wasted
> effort.
> 
> in the situations you describe another way of handling it would be to wait
> until the endTransaction() call and then return that it only suceeded with
> the first N messages (N being the number that fit it's limit)
> 
> that is less efficiant than what you are proposing, (in that the messages
> >N need to be resubmitted), but it simplifies things by avoiding the need
> for the handling the unexpected commit (and buffering errors for the next
> call, etc)

And now my core question: where do you think anything could be simplified? It
is very easy to mark messages as committed (inside the batch), but it is
comparatively complex to do the backup processing. Why do that? Let's say we
have a batch size of 1,000 and we run relp forwarding. So why always have the
full batch submitted just to come back and tell "I've done 100 an discarded
there rest" - and this nine times...

The complexity is *not* marking messages as commited inside the batch, the
complexity is keeping already dequeued messages in a "being processed" state
inside the queue. But both are two totally different topics.

Rainer