[rsyslog] does rsyslog supports data analytic

Rainer Gerhards rgerhards at hq.adiscon.com
Fri Sep 11 08:16:43 CEST 2009 

Hi Davd,

I think you hit it right on the nail.

But I have also thought a bit more about the idea. Actually, I think, one can
implement processing modules right now. Especially the configuration is a bit
tricky, but it should really work.

The rough outline is to use an output module for that. Output modules may do
whatever they want as long as they use the provided interfaces. As such, they
can also inject messages. So the idea is to define an output module, that
accepts the message, does any processing necessary, indicated RS_RET_DISCARD
to the rule engine (to prevent the message from being further processed) and
inject the "newly generated" message back into the main message queue.

That would also be much faster than whatever RainerScript will have to offer,
because RainerScript relies on VM execution.

I just don't have time to elaborately talk someone through this approach...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com 
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, September 11, 2009 6:29 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] does rsyslog supports data analytic
> 
> On Fri, 11 Sep 2009, Josh Zhao wrote:
> 
> > Is rsyslog no way to reslove problem, What about syslog-ng? 
> What I think
> > about,rsyslog's multi-thread archititure is better for my mulit-core
> > hardware. The logs data is very high volume too. Could you 
> give me any
> > suggestion on this matter?
> 
> my experiance with syslog-ng was not good, so I'm not the 
> right person to 
> talk about doing this sort of thing with it.
> 
> but I am not aware of any syslog daemon that lets you insert your own 
> logic in the middle of the processing. rsyslog has the 
> concept, but it has 
> not been implemented (fixing bugs and speeding it up has 
> taken priority)
> 
> what sort of volume do you consider 'high'? (it's amazing the 
> range that 
> this can span, so I've learned to ask rather than assume ;-)
> 
> since you are needing to get your final data into a database, 
> I think that 
> you will find that rsyslog will (or will soon) suit your 
> needs far better 
> than alternate approaches. the ability to process multiple 
> messages in one 
> transaction that is being developed will be a huge 
> improvement in terms of 
> database interaction.
> 
> I would look at what rainer suggested for now.
> 
> have one copy of rsyslog that receives the messages, does whatever 
> formatting/cleanup is needed on them, then passes the logs to 
> one or more 
> instances of your code to do additional processing, which can 
> then feed 
> the results into another instance of rsyslog to forward them 
> on, insert 
> them into a database, etc.
> 
> when rainerscript gains the capability to alter the fields 
> (instead of 
> just testing them), then there will be a lot more that can be 
> done inside 
> rsyslog.
> 
> David Lang
> 
> > Thank you!
> >
> > 2009/9/11 <david at lang.hm>
> >
> >> On Fri, 11 Sep 2009, Josh Zhao wrote:
> >>
> >>> You mean I have to rewrite the processing module in 
> rainerscript.where
> >> can i
> >>> find  the detailed documents related to the scripting engine?
> >>
> >> right now rainerscript is as much an idea as an 
> implementation. it can be
> >> used for a few things, but mostly just for filter 'does 
> this log match X'
> >> type of things.
> >>
> >> David Lang
> >>
> >>> Thank you!
> >>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao
> >>>>> Sent: Thursday, September 10, 2009 3:25 PM
> >>>>> To: rsyslog-users
> >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>
> >>>>> Thanks for David and Rainer's reply.I m sorry that I 
> did not explain my
> >>>>> question clearly.I m new to rsyslog and want to add a 
> processing module
> >>>>> in
> >>>>> rsyslog.The rsyslog has input plugins(front-end) and output
> >>>>> plugins(back-end).My processing module receives data 
> from input plugins
> >>>>> and
> >>>>> output the processed data and raw data both into output 
> plugins.So how
> >>>>> I add
> >>>>> it?
> >>>>
> >>>> What you are looking for is a library plugin. 
> Unfortunaley, library
> >> plugins
> >>>> will work together with the scripting engine. In other 
> words: there
> >>>> currently
> >>>> is no in-proc method available.
> >>>>
> >>>> What you can do, however, is chain two rsyslog 
> instances, pipe data to
> >> your
> >>>> plugin and send that data to the other instance. Far 
> from perfect and
> >> easy
> >>>> to
> >>>> do, but maybe a workable work-around...
> >>>>
> >>>> Rainer
> >>>>
> >>>>>
> >>>>>
> >>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: rsyslog-bounces at lists.adiscon.com
> >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> >>>>> david at lang.hm
> >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM
> >>>>>>> To: rsyslog-users
> >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>>
> >>>>>>>>  PS: i browse the git source code, but i can't 
> understand why the
> >>>>>>>>
> >>>>>>> Experimental-lockfree<http://git.adiscon.com/?p=rsyslog.git;a=
> >>>>>> shortlog;h=refs/heads/Experimental-lockfree>
> >>>>>>>> is
> >>>>>>>> not adopted?
> >>>>>>>
> >>>>>>> I believe that it boils down to complications in being sure
> >>>>>>> that there are
> >>>>>>> no bugs, and the fact that even without that there has been a
> >>>>>>> LOT of room
> >>>>>>> for improvement from the early 3.x timeframe to the current
> >>>>>>> 5.x version.
> >>>>>>>
> >>>>>>> I expect that after the current round of improvements are
> >>>>>>> settled that
> >>>>>>> aspect of things will get reexamined.
> >>>>>>
> >>>>>> That branch is mostly there for historical reasons. I keep that
> >>>>> branch as a
> >>>>>> think-tank, but it is is obsoleted. Also, in less 
> polite words than
> >>>>> David
> >>>>>> used, it simply doesn't work. Getting this code with multiple
> >>>>> producers and
> >>>>>> consumers correct is far from being trivial and the 
> literature I
> >>>>> browsed
> >>>>>> indicates that it is probably not possible given the 
> other predicates
> >>>>> the
> >>>>>> code must obey to. Still, optimization is high up on 
> the todo list.
> >>>>>>
> >>>>>> Rainer
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com
> >>>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

More information about the rsyslog mailing list