[rsyslog] threading - was: output plugin calling interface

[david at lang.hm]

On Wed, 6 May 2009, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Wednesday, May 06, 2009 4:06 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] threading - was: output plugin calling interface
>>
>> On Wed, 6 May 2009, Rainer Gerhards wrote:
>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>
>>>> On Wed, 6 May 2009, Rainer Gerhards wrote:
>>>>
>>>>>> as I understand it (this is part of the diagram I didn't get created
>>>>>> yet )-:
>>>>>
>>>>> Actually, I don't know how you could envision that other than by
> drawing
>>> a
>>>>> line where a queue exists in the diagrams that exists. Honestly. I have
>>> no
>>>>> clue how other I could show this graphically...
>>>>>
>>>>>>
>>>>>> rsyslog basicly breaks down into the following pieces from a
>>>>>> process/thread point of view
>>>>>>
>>>>>
>>>>> But with words it goes better ;) Note that below is current
> multi-dequeue
>>>> git
>>>>> branch, older versions (especially v3) are somewhat different:
>>>>
>>>> at this point I'm going to be lazy and just concentrate on the current
>>>> version ;-)
>>>>
>>>>>
>>>>> master thread (housekeeping, handling signals, etc)
>>>>>
>>>>> N input threads
>>>>>   recieve messages
>>>>>   put the message in the main queue
>>>>>
>>>>>
>>>>> N main queue worker threads
>>>>>   process messages in the main queue
>>>>>   foreach message
>>>>>     parse message
>>>>>     foreach filters
>>>>>       if filter applies
>>>>>          put message into the action queue
>>>>>          [if action queue mode is DIRECT call processAction()]
>>>>>
>>>>> N action queue worker threads
>>>>>   process messages in the action queue
>>>>>   foreach message
>>>>>      processAction
>>>>>
>>>>> def processAction:
>>>>>      format the message
>>>>>      call doAction()
>>>>>
>>>>> That's the very rough picture. If you take it together with the message
>>> flow
>>>>> diagrams, you'll see that a threading boundary is wherever a queue is.
>>>>
>>>> mostly, but not completely (just enough mismatch to cause confusion for
>>>> me)
>>>>
>>>>> Does that help?
>>>>
>>>> yes, it helps a _lot_.
>>>>
>>>> I think that in most cases where I have said doAction() in the last
> week,
>>>> a better fit for what I intended would have been processAction()
>>>>
>>>
>>> We must be very careful here (and I'll check if I wasn't careful
> enough...).
>>> processAction() is generic core engine code. doAction() is provided by
> the
>>> plugin.
>>
>> clarifying one point from earlier, this seems to say that the data stored
>> in an action queue is exactly the same as the data stored in the main
>> queue (specificly, it's the msg structure, not the formatted string)
>>
>> when processAction() runs, it then does the output formatting and passed
>> the formatted message to doAction()
>>
>> is this correct?
>
> Mostly ;) - it is a deep copy of the structure from the main message queue. A
> deep copy is made because the rule engine may potentially change a message
> object (not yet implemented). It copy also helps a lot with the subtle issues
> that occur due to different queue types. Granted, it costs some memory, but
> it pays back by greatly reduced complexity (we could do a copy-on-change, but
> that's another larger thing that would needed to be looked at).

ok, this makes sense (and explains why you don't do single-instance-store, 
so you can ignore that portion of my prior comments)

one other question, how do dynafiles work?

as I see it (in my probably oversimplified view of things), the output 
module needs to be passed the filename to use for a particular message, or 
the output module needs to have access the the messae structure, and the 
dynafile template to do it's own processing.

David Lang