[rsyslog] output plugin calling interface

[david at lang.hm]

On Wed, 6 May 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com
>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Wednesday, May 06, 2009 9:54 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] output plugin calling interface
>>
>> On Wed, 6 May 2009, Rainer Gerhards wrote:
>>
>>> OK, I uploaded a new document. It is not really clean yet,
>> but much better
>>> than the version from around noon. I think I need at least
>> another hour to
>>> make sure that terminology is used consistently. Most
>> important, it may say
>>> "doAction()" where "processAction()" is more precise (but
>> not always). That's
>>> because my thinking evolved today ;)
>>
>> I thought we had decided that there was no need to have a
>> beginTransaction() call if doAction() would do it implicitly.
>
> OK, I think I'll stop posting unfinished work - I did not yet manage to edit
> that out. But it probably is a better idea to finish a consistent state, even
> if that takes a day or two.

Ok, I didn't know if you just accidently left it in, or had not redone 
that section.

>
>>
>> if something is commited early, I don't see any reason why you should
>> require that everything pending is commited. all you should
>> require is
>> that M1 -> Mn be commited (a contiguous set starting from the
>> first one)
>
> Yup, but that is what is described. It is not the batch that is commited, but
> everything that was uncommitted so far.

Ok.

>>
>>> Take care of the new state diagram and be sure to
>> understand that it models
>>> an own *action state*, not a batch transaction state
>> (that's different and
>>> for tomorrow ;)).
>>
>> I'm not sure that there can be an error from the inTx stage
>> that would be
>> worth retrying. errors there would not be related to outputting the
>> message, but simply to processing it and preparing it to be
>> output later.
>
> That's why I wrote *action state* and "in bold". This is not a message state.
> This is the state machine for the action logic. The message state is
> something different and not even yet described. This is a very important
> distinction. The action is a state machine and state transistions tell when
> various things need to be called.

Ok, just below the diagram you say "Note well that the state diagram 
describes the action state. It does not describe the transaction state"

if the action is not doing the retry, why would it have a retry state?

>>
>> in fact, I'm not sure that retry belongs in the message state
>> at all. I
>> could see it argued that the commit may result in a temporary
>> error that
>> could be retried, but is that really something that the action (i.e.
>> output module) should deal with? or should this be done at
>> the transaction
>> state?
>
> It's not the retry of the message - that's the upper layer. Here, for
> example, we reset the connection, try to establish a broken link and all
> that.

why should this state be visable at all to the caller? this should be 
handled transparently inside the endTransaction code (even if that code is 
called by the doAction() call) everything else can operate without the 
link being up, so there's no reason for the caller to have to check if it 
is before everything that it does.

>> in reading the page after the diagram, it appears that you
>> are thinking
>> the same thing, in which case the retry and suspend nodes should be
>> removed from the state diagram (or there may need to be a
>> suspend node if
>> you want the higher levels to be able to try again and the module to
>> reject it)
>>
>>
>> looking at your pseudocode, I started to re-write it, and I
>> think things
>> can be much simpler.
>>
>> if the retrys are done above this level, then the only thing
>> that we need
>> to do is to not hammer the destination.
>>
>> except for the fact that doAction() can trigger an EndTransaction()
>> internally, there is no reason why doAction() can't take place while
>> suspended (the output module can be preparing the stuff to
>> send out).
>
> How will you send e.g. a tcp message while the network link is down? How to
> talk to a mail server if it is down? How to write to the file system if it is
> full?
>
> doAction is not a "copy this to a buffer" kind of thing, but rather something
> that potentially does real work.

here is where we disagree.

doAction() is getting the message and putting it in the buffer to be sent 
when the endTransaction is called.

it may also decide to _do_ endTransaction, but it's the endTransaction 
logic that has the real work to do.

and part of that real work is to have logic like 'if socket is dead, 
re-open it'

if the filesystem is full, the socket cannot be opened, the socket goes 
away in the middle, etc the endTransaction() logic will need to return an 
batchFailed error.

> It looks like you prefer the matrix-like action interface we talked about a
> while ago, but this interface causes compatibility issues to existing
> modules, causes more code inside each module and causes far more complicated
> code inside the engine.

yes, I do think that the matrix-like interface would be better, but I 
don't think I'm arguing that at this point (I have slipped into that at 
points, where I argued to move the format_message into the doAction, I'll 
try to watch that)

at this point I understand the interface to be

doAction(message)
   takes the string $message and add it to the batch to be sent
   (initializing the batch if it isn't already initialized)

   optionally decides to call endTransaction()


endTransaction()
   takes the batch that was prepared by one or more doAction() calls, 
finalizes the batch if needed, and sends it to the destination.


David Lang