[rsyslog] does rsyslog supports data analytic

[Josh Zhao]

Our raw data is "high" volume that means to prcocess data about 100M/min.
Yes, I want to improve the system performance as soon as possibe.  As you
said,rsyslog has a concept that inserts my logic module into it ,but it was
not been implemented. Could you point out in detail? The rainerscript seems
not that strong,otherwise, it is a good idea for user interface.


2009/9/11 <david at lang.hm>

> On Fri, 11 Sep 2009, Josh Zhao wrote:
>
> > Is rsyslog no way to reslove problem, What about syslog-ng? What I think
> > about,rsyslog's multi-thread archititure is better for my mulit-core
> > hardware. The logs data is very high volume too. Could you give me any
> > suggestion on this matter?
>
> my experiance with syslog-ng was not good, so I'm not the right person to
> talk about doing this sort of thing with it.
>
> but I am not aware of any syslog daemon that lets you insert your own
> logic in the middle of the processing. rsyslog has the concept, but it has
> not been implemented (fixing bugs and speeding it up has taken priority)
>
> what sort of volume do you consider 'high'? (it's amazing the range that
> this can span, so I've learned to ask rather than assume ;-)
>
> since you are needing to get your final data into a database, I think that
> you will find that rsyslog will (or will soon) suit your needs far better
> than alternate approaches. the ability to process multiple messages in one
> transaction that is being developed will be a huge improvement in terms of
> database interaction.
>
> I would look at what rainer suggested for now.
>
> have one copy of rsyslog that receives the messages, does whatever
> formatting/cleanup is needed on them, then passes the logs to one or more
> instances of your code to do additional processing, which can then feed
> the results into another instance of rsyslog to forward them on, insert
> them into a database, etc.
>
> when rainerscript gains the capability to alter the fields (instead of
> just testing them), then there will be a lot more that can be done inside
> rsyslog.
>
> David Lang
>
> > Thank you!
> >
> > 2009/9/11 <david at lang.hm>
> >
> >> On Fri, 11 Sep 2009, Josh Zhao wrote:
> >>
> >>> You mean I have to rewrite the processing module in rainerscript.where
> >> can i
> >>> find  the detailed documents related to the scripting engine?
> >>
> >> right now rainerscript is as much an idea as an implementation. it can
> be
> >> used for a few things, but mostly just for filter 'does this log match
> X'
> >> type of things.
> >>
> >> David Lang
> >>
> >>> Thank you!
> >>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>
> >>>>> -----Original Message-----
> >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao
> >>>>> Sent: Thursday, September 10, 2009 3:25 PM
> >>>>> To: rsyslog-users
> >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>
> >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain
> my
> >>>>> question clearly.I m new to rsyslog and want to add a processing
> module
> >>>>> in
> >>>>> rsyslog.The rsyslog has input plugins(front-end) and output
> >>>>> plugins(back-end).My processing module receives data from input
> plugins
> >>>>> and
> >>>>> output the processed data and raw data both into output plugins.So
> how
> >>>>> I add
> >>>>> it?
> >>>>
> >>>> What you are looking for is a library plugin. Unfortunaley, library
> >> plugins
> >>>> will work together with the scripting engine. In other words: there
> >>>> currently
> >>>> is no in-proc method available.
> >>>>
> >>>> What you can do, however, is chain two rsyslog instances, pipe data to
> >> your
> >>>> plugin and send that data to the other instance. Far from perfect and
> >> easy
> >>>> to
> >>>> do, but maybe a workable work-around...
> >>>>
> >>>> Rainer
> >>>>
> >>>>>
> >>>>>
> >>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: rsyslog-bounces at lists.adiscon.com
> >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> >>>>> david at lang.hm
> >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM
> >>>>>>> To: rsyslog-users
> >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>>
> >>>>>>>>  PS: i browse the git source code, but i can't understand why the
> >>>>>>>>
> >>>>>>> Experimental-lockfree<http://git.adiscon.com/?p=rsyslog.git;a=
> >>>>>> shortlog;h=refs/heads/Experimental-lockfree>
> >>>>>>>> is
> >>>>>>>> not adopted?
> >>>>>>>
> >>>>>>> I believe that it boils down to complications in being sure
> >>>>>>> that there are
> >>>>>>> no bugs, and the fact that even without that there has been a
> >>>>>>> LOT of room
> >>>>>>> for improvement from the early 3.x timeframe to the current
> >>>>>>> 5.x version.
> >>>>>>>
> >>>>>>> I expect that after the current round of improvements are
> >>>>>>> settled that
> >>>>>>> aspect of things will get reexamined.
> >>>>>>
> >>>>>> That branch is mostly there for historical reasons. I keep that
> >>>>> branch as a
> >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than
> >>>>> David
> >>>>>> used, it simply doesn't work. Getting this code with multiple
> >>>>> producers and
> >>>>>> consumers correct is far from being trivial and the literature I
> >>>>> browsed
> >>>>>> indicates that it is probably not possible given the other
> predicates
> >>>>> the
> >>>>>> code must obey to. Still, optimization is high up on the todo list.
> >>>>>>
> >>>>>> Rainer
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com
> >>>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>