[rsyslog] does rsyslog supports data analytic

[david at lang.hm]

On Fri, 11 Sep 2009, Josh Zhao wrote:

> Is rsyslog no way to reslove problem, What about syslog-ng? What I think
> about,rsyslog's multi-thread archititure is better for my mulit-core
> hardware. The logs data is very high volume too. Could you give me any
> suggestion on this matter?

my experiance with syslog-ng was not good, so I'm not the right person to 
talk about doing this sort of thing with it.

but I am not aware of any syslog daemon that lets you insert your own 
logic in the middle of the processing. rsyslog has the concept, but it has 
not been implemented (fixing bugs and speeding it up has taken priority)

what sort of volume do you consider 'high'? (it's amazing the range that 
this can span, so I've learned to ask rather than assume ;-)

since you are needing to get your final data into a database, I think that 
you will find that rsyslog will (or will soon) suit your needs far better 
than alternate approaches. the ability to process multiple messages in one 
transaction that is being developed will be a huge improvement in terms of 
database interaction.

I would look at what rainer suggested for now.

have one copy of rsyslog that receives the messages, does whatever 
formatting/cleanup is needed on them, then passes the logs to one or more 
instances of your code to do additional processing, which can then feed 
the results into another instance of rsyslog to forward them on, insert 
them into a database, etc.

when rainerscript gains the capability to alter the fields (instead of 
just testing them), then there will be a lot more that can be done inside 
rsyslog.

David Lang

> Thank you!
>
> 2009/9/11 <david at lang.hm>
>
>> On Fri, 11 Sep 2009, Josh Zhao wrote:
>>
>>> You mean I have to rewrite the processing module in rainerscript.where
>> can i
>>> find  the detailed documents related to the scripting engine?
>>
>> right now rainerscript is as much an idea as an implementation. it can be
>> used for a few things, but mostly just for filter 'does this log match X'
>> type of things.
>>
>> David Lang
>>
>>> Thank you!
>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao
>>>>> Sent: Thursday, September 10, 2009 3:25 PM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
>>>>>
>>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain my
>>>>> question clearly.I m new to rsyslog and want to add a processing module
>>>>> in
>>>>> rsyslog.The rsyslog has input plugins(front-end) and output
>>>>> plugins(back-end).My processing module receives data from input plugins
>>>>> and
>>>>> output the processed data and raw data both into output plugins.So how
>>>>> I add
>>>>> it?
>>>>
>>>> What you are looking for is a library plugin. Unfortunaley, library
>> plugins
>>>> will work together with the scripting engine. In other words: there
>>>> currently
>>>> is no in-proc method available.
>>>>
>>>> What you can do, however, is chain two rsyslog instances, pipe data to
>> your
>>>> plugin and send that data to the other instance. Far from perfect and
>> easy
>>>> to
>>>> do, but maybe a workable work-around...
>>>>
>>>> Rainer
>>>>
>>>>>
>>>>>
>>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rsyslog-bounces at lists.adiscon.com
>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
>>>>> david at lang.hm
>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM
>>>>>>> To: rsyslog-users
>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
>>>>>>
>>>>>>>>  PS: i browse the git source code, but i can't understand why the
>>>>>>>>
>>>>>>> Experimental-lockfree<http://git.adiscon.com/?p=rsyslog.git;a=
>>>>>> shortlog;h=refs/heads/Experimental-lockfree>
>>>>>>>> is
>>>>>>>> not adopted?
>>>>>>>
>>>>>>> I believe that it boils down to complications in being sure
>>>>>>> that there are
>>>>>>> no bugs, and the fact that even without that there has been a
>>>>>>> LOT of room
>>>>>>> for improvement from the early 3.x timeframe to the current
>>>>>>> 5.x version.
>>>>>>>
>>>>>>> I expect that after the current round of improvements are
>>>>>>> settled that
>>>>>>> aspect of things will get reexamined.
>>>>>>
>>>>>> That branch is mostly there for historical reasons. I keep that
>>>>> branch as a
>>>>>> think-tank, but it is is obsoleted. Also, in less polite words than
>>>>> David
>>>>>> used, it simply doesn't work. Getting this code with multiple
>>>>> producers and
>>>>>> consumers correct is far from being trivial and the literature I
>>>>> browsed
>>>>>> indicates that it is probably not possible given the other predicates
>>>>> the
>>>>>> code must obey to. Still, optimization is high up on the todo list.
>>>>>>
>>>>>> Rainer
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>