[rsyslog] output plugin calling interface

[david at lang.hm]

On Wed, 6 May 2009, Rainer Gerhards wrote:

> One quick question:
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Wednesday, May 06, 2009 6:07 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] output plugin calling interface
>>
>>> I agree with you on the performance. I disagree that this means the
> output
>>> transaction and the transaction from the upper layer must exactly match.
>>> Think RELP. Would that mean that a relp window must always be as large as
>> the
>>> largest batch? OK, in this case I control the protocol and so could
> change
>>> it. But what with a SNMP trap? Or a rfc3195 conversation? They *have*
>>> different notation of a transaction. So the best thing I can do IMHO is
>>> permit the output plugin to tell the engine when its transaction was
>>> finished. That's no problem for the upper layer, it just needs to mark
> these
>>> messages as committed. But it is simply impossible for all outputs to
> have
>>> the same idea of transaction than the upper layer may have.
>>
>> I guess I see the upper layer having a larger definition of a batch than
>> the lower level to be a configuration error.
>
> The lower level limit is not a config setting but rather something that
> "happens to be that way". RELP by design uses a sliding window, so the
> "lower-level" batch is a moving target. Some for many other outputs.

right, and if the admin sets the batch size higher than this limit it's a 
mistake.

>> not one to refuse to boot with, but one that could result in wasted
>> effort.
>>
>> in the situations you describe another way of handling it would be to wait
>> until the endTransaction() call and then return that it only suceeded with
>> the first N messages (N being the number that fit it's limit)
>>
>> that is less efficiant than what you are proposing, (in that the messages
>>> N need to be resubmitted), but it simplifies things by avoiding the need
>> for the handling the unexpected commit (and buffering errors for the next
>> call, etc)
>
> And now my core question: where do you think anything could be simplified? It
> is very easy to mark messages as committed (inside the batch), but it is
> comparatively complex to do the backup processing. Why do that? Let's say we
> have a batch size of 1,000 and we run relp forwarding. So why always have the
> full batch submitted just to come back and tell "I've done 100 an discarded
> there rest" - and this nine times...

it's overhead, it costs performance, but I don't see it as adding 
complexity.

I think this is because I see a need to have some partial retry 
processing no matter what, so if we have partial retry capability and it 
can handle this mode, then it is simpler to do so than it would be to have 
both the partial retry processing _and_ the output module issueing commits 
whenever it wants to.

David Lang

> The complexity is *not* marking messages as commited inside the batch, the
> complexity is keeping already dequeued messages in a "being processed" state
> inside the queue. But both are two totally different topics.