[rsyslog] does rsyslog supports data analytic

david at lang.hm david at lang.hm
Fri Sep 11 17:24:04 CEST 2009 

On Fri, 11 Sep 2009, Josh Zhao wrote:

> Our raw data is "high" volume that means to prcocess data about 100M/min.

is this 100M log records, or 100MB of log data (if the latter, 
approximatly how large are the recors, of how many log records/min)

I'm currently processing ~300K messages averaging ~256 bytes/message for a 
total of ~75MB of logs/min.

in my testing v4 will support up to about 6x this volume before it runs 
into problems (it can receive them faster, up to gig-E wire speed, the 
limit is in the output, which is ~80K records a sec if doing trivial work 
like writing them to disk or ~30K records/sec if doing more complex things 
like forwarding them elsewhere)

improvements in V5 include a batch mode that lets an output module process 
up to N records at a time. I expect this to provide close to a Nx speedup 
to the output capabilities (with single log per action much of the 
overhead is in the queue locking, so multiple output workers doesn't help 
much, with batches not only is much more getting done per pass, but you 
have the possibility of each output thread taking long enough to get it's 
work done that it's effective to run more of them without locking 
contention being the bottleneck)

this batch mode will be especially useful for database work as it will let 
you insert multiple messages in the database in a single transaction.

what transport are you using to deliver the logs to your server?

> Yes, I want to improve the system performance as soon as possibe.

what is the bottleneck you are running into today (what syslog system are 
you using, etc)?

>  As you
> said,rsyslog has a concept that inserts my logic module into it ,but it was
> not been implemented. Could you point out in detail? The rainerscript seems
> not that strong,otherwise, it is a good idea for user interface.

if you are looking at the source look for imtemplate and omtemplate, 
basicly he is suggesting creating a custom output module that rsyslog 
thinks is delivering the messages somewhere, have it be given the log, do 
it's processing, then acting like an input module and delivering the 
result to rsyslog as if it was a new message that just arrived.

you will need to put some filters in rsyslog to keep your output module 
from seeing the logs that it creates, and either use discard or filters to 
keep the other output modules from seeing the raw input that your module 
is looking for.

David Lang

>
> 2009/9/11 <david at lang.hm>
>
>> On Fri, 11 Sep 2009, Josh Zhao wrote:
>>
>>> Is rsyslog no way to reslove problem, What about syslog-ng? What I think
>>> about,rsyslog's multi-thread archititure is better for my mulit-core
>>> hardware. The logs data is very high volume too. Could you give me any
>>> suggestion on this matter?
>>
>> my experiance with syslog-ng was not good, so I'm not the right person to
>> talk about doing this sort of thing with it.
>>
>> but I am not aware of any syslog daemon that lets you insert your own
>> logic in the middle of the processing. rsyslog has the concept, but it has
>> not been implemented (fixing bugs and speeding it up has taken priority)
>>
>> what sort of volume do you consider 'high'? (it's amazing the range that
>> this can span, so I've learned to ask rather than assume ;-)
>>
>> since you are needing to get your final data into a database, I think that
>> you will find that rsyslog will (or will soon) suit your needs far better
>> than alternate approaches. the ability to process multiple messages in one
>> transaction that is being developed will be a huge improvement in terms of
>> database interaction.
>>
>> I would look at what rainer suggested for now.
>>
>> have one copy of rsyslog that receives the messages, does whatever
>> formatting/cleanup is needed on them, then passes the logs to one or more
>> instances of your code to do additional processing, which can then feed
>> the results into another instance of rsyslog to forward them on, insert
>> them into a database, etc.
>>
>> when rainerscript gains the capability to alter the fields (instead of
>> just testing them), then there will be a lot more that can be done inside
>> rsyslog.
>>
>> David Lang
>>
>>> Thank you!
>>>
>>> 2009/9/11 <david at lang.hm>
>>>
>>>> On Fri, 11 Sep 2009, Josh Zhao wrote:
>>>>
>>>>> You mean I have to rewrite the processing module in rainerscript.where
>>>> can i
>>>>> find  the detailed documents related to the scripting engine?
>>>>
>>>> right now rainerscript is as much an idea as an implementation. it can
>> be
>>>> used for a few things, but mostly just for filter 'does this log match
>> X'
>>>> type of things.
>>>>
>>>> David Lang
>>>>
>>>>> Thank you!
>>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao
>>>>>>> Sent: Thursday, September 10, 2009 3:25 PM
>>>>>>> To: rsyslog-users
>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
>>>>>>>
>>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain
>> my
>>>>>>> question clearly.I m new to rsyslog and want to add a processing
>> module
>>>>>>> in
>>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output
>>>>>>> plugins(back-end).My processing module receives data from input
>> plugins
>>>>>>> and
>>>>>>> output the processed data and raw data both into output plugins.So
>> how
>>>>>>> I add
>>>>>>> it?
>>>>>>
>>>>>> What you are looking for is a library plugin. Unfortunaley, library
>>>> plugins
>>>>>> will work together with the scripting engine. In other words: there
>>>>>> currently
>>>>>> is no in-proc method available.
>>>>>>
>>>>>> What you can do, however, is chain two rsyslog instances, pipe data to
>>>> your
>>>>>> plugin and send that data to the other instance. Far from perfect and
>>>> easy
>>>>>> to
>>>>>> do, but maybe a workable work-around...
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com
>>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
>>>>>>> david at lang.hm
>>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM
>>>>>>>>> To: rsyslog-users
>>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
>>>>>>>>
>>>>>>>>>>  PS: i browse the git source code, but i can't understand why the
>>>>>>>>>>
>>>>>>>>> Experimental-lockfree<http://git.adiscon.com/?p=rsyslog.git;a=
>>>>>>>> shortlog;h=refs/heads/Experimental-lockfree>
>>>>>>>>>> is
>>>>>>>>>> not adopted?
>>>>>>>>>
>>>>>>>>> I believe that it boils down to complications in being sure
>>>>>>>>> that there are
>>>>>>>>> no bugs, and the fact that even without that there has been a
>>>>>>>>> LOT of room
>>>>>>>>> for improvement from the early 3.x timeframe to the current
>>>>>>>>> 5.x version.
>>>>>>>>>
>>>>>>>>> I expect that after the current round of improvements are
>>>>>>>>> settled that
>>>>>>>>> aspect of things will get reexamined.
>>>>>>>>
>>>>>>>> That branch is mostly there for historical reasons. I keep that
>>>>>>> branch as a
>>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words than
>>>>>>> David
>>>>>>>> used, it simply doesn't work. Getting this code with multiple
>>>>>>> producers and
>>>>>>>> consumers correct is far from being trivial and the literature I
>>>>>>> browsed
>>>>>>>> indicates that it is probably not possible given the other
>> predicates
>>>>>>> the
>>>>>>>> code must obey to. Still, optimization is high up on the todo list.
>>>>>>>>
>>>>>>>> Rainer
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

More information about the rsyslog mailing list