[rsyslog] output plugin calling interface

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, May 06, 2009 9:54 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] output plugin calling interface
> 
> On Wed, 6 May 2009, Rainer Gerhards wrote:
> 
> > OK, I uploaded a new document. It is not really clean yet, 
> but much better
> > than the version from around noon. I think I need at least 
> another hour to
> > make sure that terminology is used consistently. Most 
> important, it may say
> > "doAction()" where "processAction()" is more precise (but 
> not always). That's
> > because my thinking evolved today ;)
> 
> I thought we had decided that there was no need to have a 
> beginTransaction() call if doAction() would do it implicitly.

OK, I think I'll stop posting unfinished work - I did not yet manage to edit
that out. But it probably is a better idea to finish a consistent state, even
if that takes a day or two.


> 
> if something is commited early, I don't see any reason why you should 
> require that everything pending is commited. all you should 
> require is 
> that M1 -> Mn be commited (a contiguous set starting from the 
> first one)

Yup, but that is what is described. It is not the batch that is commited, but
everything that was uncommitted so far.
> 
> 
> > Take care of the new state diagram and be sure to 
> understand that it models
> > an own *action state*, not a batch transaction state 
> (that's different and
> > for tomorrow ;)).
> 
> I'm not sure that there can be an error from the inTx stage 
> that would be 
> worth retrying. errors there would not be related to outputting the 
> message, but simply to processing it and preparing it to be 
> output later.

That's why I wrote *action state* and "in bold". This is not a message state.
This is the state machine for the action logic. The message state is
something different and not even yet described. This is a very important
distinction. The action is a state machine and state transistions tell when
various things need to be called.

> 
> in fact, I'm not sure that retry belongs in the message state 
> at all. I 
> could see it argued that the commit may result in a temporary 
> error that 
> could be retried, but is that really something that the action (i.e. 
> output module) should deal with? or should this be done at 
> the transaction 
> state?

It's not the retry of the message - that's the upper layer. Here, for
example, we reset the connection, try to establish a broken link and all
that.

> 
> in reading the page after the diagram, it appears that you 
> are thinking 
> the same thing, in which case the retry and suspend nodes should be 
> removed from the state diagram (or there may need to be a 
> suspend node if 
> you want the higher levels to be able to try again and the module to 
> reject it)
> 
> 
> looking at your pseudocode, I started to re-write it, and I 
> think things 
> can be much simpler.
> 
> if the retrys are done above this level, then the only thing 
> that we need 
> to do is to not hammer the destination.
> 
> except for the fact that doAction() can trigger an EndTransaction() 
> internally, there is no reason why doAction() can't take place while 
> suspended (the output module can be preparing the stuff to 
> send out).

How will you send e.g. a tcp message while the network link is down? How to
talk to a mail server if it is down? How to write to the file system if it is
full?

doAction is not a "copy this to a buffer" kind of thing, but rather something
that potentially does real work.

It looks like you prefer the matrix-like action interface we talked about a
while ago, but this interface causes compatibility issues to existing
modules, causes more code inside each module and causes far more complicated
code inside the engine.

Rainer

> the 
> only place that needs to deal with the issue is that the 
> EndTransaction() 
> should sleep if the state is not itx
> 
> if doAction does beginTransaction() any time it's not in a 
> transaction, 
> there is no reason to have it as a seperate call.
> 
> so without retries or beginTransaction, is there any reason for 
> prepareAction() to exist?
> 
> you also will need to detect that the doAction() did 
> endTransaction() and 
> that you don't need to issue a endTransaction() for this 
> output module now 
> (until you do the next doAction() )
> 
> David Lang
> 
> 
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> >> Sent: Wednesday, May 06, 2009 6:33 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] output plugin calling interface
> >>
> >>> then it is simpler to do so than it would be to have
> >>> both the partial retry processing _and_ the output module issueing
> > commits
> >>> whenever it wants to.
> >>
> >>
> >> Why? If I think about code, it is very hard to beat auto-commits in
> >> simplicity...
> >>
> >> But I'd better finish the new state diagram and at least 
> part of the
> >> description, I think I finally found the right model for 
> the lower layer.
> >>
> >> Rainer
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>