[rsyslog] output plugin calling interface

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, May 06, 2009 11:24 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] output plugin calling interface
> 
> On Wed, 6 May 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com
> >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Wednesday, May 06, 2009 9:54 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] output plugin calling interface
> >>
> >> On Wed, 6 May 2009, Rainer Gerhards wrote:
> >>
> >>> OK, I uploaded a new document. It is not really clean yet,
> >> but much better
> >>> than the version from around noon. I think I need at least
> >> another hour to
> >>> make sure that terminology is used consistently. Most
> >> important, it may say
> >>> "doAction()" where "processAction()" is more precise (but
> >> not always). That's
> >>> because my thinking evolved today ;)
> >>
> >> I thought we had decided that there was no need to have a
> >> beginTransaction() call if doAction() would do it implicitly.
> >
> > OK, I think I'll stop posting unfinished work - I did not yet manage to
edit
> > that out. But it probably is a better idea to finish a consistent state,
> even
> > if that takes a day or two.
> 
> Ok, I didn't know if you just accidently left it in, or had not redone
> that section.

I have to admit that I did not think enough when I answered your mail last
evening. On closer look, the output module interface still needs this entry
point and this is why it is in the pseudocode. Please do NOT comment yet but
read on first...

> > How will you send e.g. a tcp message while the network link is down? How
to
> > talk to a mail server if it is down? How to write to the file system if
it
> is
> > full?
> >
> > doAction is not a "copy this to a buffer" kind of thing, but rather
> something
> > that potentially does real work.
> 
> here is where we disagree.
> 
> doAction() is getting the message and putting it in the buffer to be sent
> when the endTransaction is called.
> 
> it may also decide to _do_ endTransaction, but it's the endTransaction
> logic that has the real work to do.
> 
> and part of that real work is to have logic like 'if socket is dead,
> re-open it'
> 
> if the filesystem is full, the socket cannot be opened, the socket goes
> away in the middle, etc the endTransaction() logic will need to return an
> batchFailed error.

... and this disagreement we are actually circeling around in the whole
discussion. You have a quite different design philosophy on your mind than I
have.

Rsyslog's core philosophy is that outputs should be ultra-slim and very easy
to write. That does not outrule one can write a complex output, but it should
be possible to write a very simplistic one. This is more a mini-driver design
perspective, where a generic driver inside the core is just complemented by
some specifics.

Your design is that the core engine is slim and pushes off the complexity to
the outputs. This is more a traditional driver concept, where the core
requests rather broad functionality from its drivers.

Out of this different approach we see different ways of doing things. You
know that I value your opinion *very* much and it is always great fun and
educating to talk with you.

But I have to admit that it will require very strong argument to make me go
away from the traditional rsyslog design perspective. My goals (in addition
to things like performance and reliability, of course) are

a) identify as much common functionality as possible and put that into a
single place
b) do not break backward compatibility unless strictly necessary 

For b), I would even be prepared to accept some mild increased complexity
(which we do not have in this case).

I think almost all of our discussion topics circle around the design
philosophy. And this is also why we both think we are correct. Because we
*are both*, just depending from what design philosophy you look at.

I stick with the current rsyslog philosophy because it has served me very
well over all those years. I consider it a premier reasons why my software
projects are often much more successful than competing ones.

The beauty of trying hard to put functionality into a single place, and doing
that consistently, is that at some point in time your software will grow
exponentially with each feature being added (which is then automatically
inherited to the rest of the system, as everything is inside the right
layer).

> 
> > It looks like you prefer the matrix-like action interface we talked about
a
> > while ago, but this interface causes compatibility issues to existing
> > modules, causes more code inside each module and causes far more
complicated
> > code inside the engine.
> 
> yes, I do think that the matrix-like interface would be better, but I
> don't think I'm arguing that at this point (I have slipped into that at
> points, where I argued to move the format_message into the doAction, I'll
> try to watch that)

The core of the matrix-like interface is not how parameters are passed, but
rather that there is only one entry point that does the work. You are right
now modeling this design approach with two calls, but it basically remains
the matrix-type of interface. If we'd take that route, it would be cleaner to
use such a matrix interface. 

> 
> at this point I understand the interface to be
> 
> doAction(message)
>    takes the string $message and add it to the batch to be sent
>    (initializing the batch if it isn't already initialized)
> 
>    optionally decides to call endTransaction()
> 
> 
> endTransaction()
>    takes the batch that was prepared by one or more doAction() calls,
> finalizes the batch if needed, and sends it to the destination.
> 

Just to iterate the differences in our designs:

doAction(message)
   takes the string $message and process it. This may be adding
   it to a batch, immediately processing it, or buffering for some
   time and then process it
   (initializing the batch if it isn't already initialized)


endTransaction()
   finish whatever doAction() has left open. This may commit
   something or may not do anything at all (depending on
   the action's state)

It is up to the plugin to decide when to do what. Plugins must not even
support endTransation() if they commit every message.

Important Note: I am using doAction()/endTransaction() where actually other
entry points should be used. Because the former are mini-driver entry points
while we talk about the transaction handler (better be called
processMessage() and, maybe, endBatch()).

In any case, I'll verify some of my ideas in code today. If my design matches
rsyslog design, it should be fairly easy and straightforward to implement the
transaction handler inside the action layer. If that turns out to be major
pain, I may be thinking into the wrong direction. So that will probably be my
focus for today.

Rainer