[rsyslog] threading - was: output plugin calling interface

[Rainer Gerhards]

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, May 06, 2009 4:06 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] threading - was: output plugin calling interface
> 
> On Wed, 6 May 2009, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>
> >> On Wed, 6 May 2009, Rainer Gerhards wrote:
> >>
> >>>> as I understand it (this is part of the diagram I didn't get created
> >>>> yet )-:
> >>>
> >>> Actually, I don't know how you could envision that other than by
drawing
> > a
> >>> line where a queue exists in the diagrams that exists. Honestly. I have
> > no
> >>> clue how other I could show this graphically...
> >>>
> >>>>
> >>>> rsyslog basicly breaks down into the following pieces from a
> >>>> process/thread point of view
> >>>>
> >>>
> >>> But with words it goes better ;) Note that below is current
multi-dequeue
> >> git
> >>> branch, older versions (especially v3) are somewhat different:
> >>
> >> at this point I'm going to be lazy and just concentrate on the current
> >> version ;-)
> >>
> >>>
> >>> master thread (housekeeping, handling signals, etc)
> >>>
> >>> N input threads
> >>>   recieve messages
> >>>   put the message in the main queue
> >>>
> >>>
> >>> N main queue worker threads
> >>>   process messages in the main queue
> >>>   foreach message
> >>>     parse message
> >>>     foreach filters
> >>>       if filter applies
> >>>          put message into the action queue
> >>>          [if action queue mode is DIRECT call processAction()]
> >>>
> >>> N action queue worker threads
> >>>   process messages in the action queue
> >>>   foreach message
> >>>      processAction
> >>>
> >>> def processAction:
> >>>      format the message
> >>>      call doAction()
> >>>
> >>> That's the very rough picture. If you take it together with the message
> > flow
> >>> diagrams, you'll see that a threading boundary is wherever a queue is.
> >>
> >> mostly, but not completely (just enough mismatch to cause confusion for
> >> me)
> >>
> >>> Does that help?
> >>
> >> yes, it helps a _lot_.
> >>
> >> I think that in most cases where I have said doAction() in the last
week,
> >> a better fit for what I intended would have been processAction()
> >>
> >
> > We must be very careful here (and I'll check if I wasn't careful
enough...).
> > processAction() is generic core engine code. doAction() is provided by
the
> > plugin.
> 
> clarifying one point from earlier, this seems to say that the data stored
> in an action queue is exactly the same as the data stored in the main
> queue (specificly, it's the msg structure, not the formatted string)
> 
> when processAction() runs, it then does the output formatting and passed
> the formatted message to doAction()
> 
> is this correct?

Mostly ;) - it is a deep copy of the structure from the main message queue. A
deep copy is made because the rule engine may potentially change a message
object (not yet implemented). It copy also helps a lot with the subtle issues
that occur due to different queue types. Granted, it costs some memory, but
it pays back by greatly reduced complexity (we could do a copy-on-change, but
that's another larger thing that would needed to be looked at).

Rainer