[rsyslog] does rsyslog supports data analytic

Fri Sep 11 17:30:25 CEST 2009

> if you are looking at the source look for imtemplate and omtemplate,
> basicly he is suggesting creating a custom output module that rsyslog
> thinks is delivering the messages somewhere, have it be given the log,
> do
> it's processing, then acting like an input module and delivering the
> result to rsyslog as if it was a new message that just arrived.

I think I did not state one important fact: this is not a dirty trick, but
something that the engine was designed for. This mechanism was originally
designed and is (somewhat) actually used to report back error conditions.
It's used sparsely, because of the circular loop potential. But it is
something the engine can handle and is designed to - so no abuse. Actually, I
have begun to think if for some feature requests (string replacements before
finally writing to an output) this may be good alternative approach. But it
seems to involve more overhead than necessary for the job.

> you will need to put some filters in rsyslog to keep your output module
> from seeing the logs that it creates, and either use discard or filters
> to
> keep the other output modules from seeing the raw input that your
> module
> is looking for.

Returning RS_RET_DISCARD would solve this, as it stops processing. You just
need to make sure that the newly injected messages don't go back into the
same rule. With multiple rulesets we now have, this is trivial.

But while all this is interesting, I unfortunately have more pressing things
to do ;)

Rainer

> 
> David Lang
> 
> >
> > 2009/9/11 <david at lang.hm>
> >
> >> On Fri, 11 Sep 2009, Josh Zhao wrote:
> >>
> >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I
> think
> >>> about,rsyslog's multi-thread archititure is better for my mulit-
> core
> >>> hardware. The logs data is very high volume too. Could you give me
> any
> >>> suggestion on this matter?
> >>
> >> my experiance with syslog-ng was not good, so I'm not the right
> person to
> >> talk about doing this sort of thing with it.
> >>
> >> but I am not aware of any syslog daemon that lets you insert your
> own
> >> logic in the middle of the processing. rsyslog has the concept, but
> it has
> >> not been implemented (fixing bugs and speeding it up has taken
> priority)
> >>
> >> what sort of volume do you consider 'high'? (it's amazing the range
> that
> >> this can span, so I've learned to ask rather than assume ;-)
> >>
> >> since you are needing to get your final data into a database, I
> think that
> >> you will find that rsyslog will (or will soon) suit your needs far
> better
> >> than alternate approaches. the ability to process multiple messages
> in one
> >> transaction that is being developed will be a huge improvement in
> terms of
> >> database interaction.
> >>
> >> I would look at what rainer suggested for now.
> >>
> >> have one copy of rsyslog that receives the messages, does whatever
> >> formatting/cleanup is needed on them, then passes the logs to one or
> more
> >> instances of your code to do additional processing, which can then
> feed
> >> the results into another instance of rsyslog to forward them on,
> insert
> >> them into a database, etc.
> >>
> >> when rainerscript gains the capability to alter the fields (instead
> of
> >> just testing them), then there will be a lot more that can be done
> inside
> >> rsyslog.
> >>
> >> David Lang
> >>
> >>> Thank you!
> >>>
> >>> 2009/9/11 <david at lang.hm>
> >>>
> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote:
> >>>>
> >>>>> You mean I have to rewrite the processing module in
> rainerscript.where
> >>>> can i
> >>>>> find  the detailed documents related to the scripting engine?
> >>>>
> >>>> right now rainerscript is as much an idea as an implementation. it
> can
> >> be
> >>>> used for a few things, but mostly just for filter 'does this log
> match
> >> X'
> >>>> type of things.
> >>>>
> >>>> David Lang
> >>>>
> >>>>> Thank you!
> >>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao
> >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM
> >>>>>>> To: rsyslog-users
> >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>>>
> >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not
> explain
> >> my
> >>>>>>> question clearly.I m new to rsyslog and want to add a
> processing
> >> module
> >>>>>>> in
> >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output
> >>>>>>> plugins(back-end).My processing module receives data from input
> >> plugins
> >>>>>>> and
> >>>>>>> output the processed data and raw data both into output
> plugins.So
> >> how
> >>>>>>> I add
> >>>>>>> it?
> >>>>>>
> >>>>>> What you are looking for is a library plugin. Unfortunaley,
> library
> >>>> plugins
> >>>>>> will work together with the scripting engine. In other words:
> there
> >>>>>> currently
> >>>>>> is no in-proc method available.
> >>>>>>
> >>>>>> What you can do, however, is chain two rsyslog instances, pipe
> data to
> >>>> your
> >>>>>> plugin and send that data to the other instance. Far from
> perfect and
> >>>> easy
> >>>>>> to
> >>>>>> do, but maybe a workable work-around...
> >>>>>>
> >>>>>> Rainer
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> 2009/9/10 Rainer Gerhards <rgerhards at hq.adiscon.com>
> >>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com
> >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of
> >>>>>>> david at lang.hm
> >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM
> >>>>>>>>> To: rsyslog-users
> >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic
> >>>>>>>>
> >>>>>>>>>>  PS: i browse the git source code, but i can't understand
> why the
> >>>>>>>>>>
> >>>>>>>>> Experimental-
> lockfree<http://git.adiscon.com/?p=rsyslog.git;a=
> >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree>
> >>>>>>>>>> is
> >>>>>>>>>> not adopted?
> >>>>>>>>>
> >>>>>>>>> I believe that it boils down to complications in being sure
> >>>>>>>>> that there are
> >>>>>>>>> no bugs, and the fact that even without that there has been a
> >>>>>>>>> LOT of room
> >>>>>>>>> for improvement from the early 3.x timeframe to the current
> >>>>>>>>> 5.x version.
> >>>>>>>>>
> >>>>>>>>> I expect that after the current round of improvements are
> >>>>>>>>> settled that
> >>>>>>>>> aspect of things will get reexamined.
> >>>>>>>>
> >>>>>>>> That branch is mostly there for historical reasons. I keep
> that
> >>>>>>> branch as a
> >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words
> than
> >>>>>>> David
> >>>>>>>> used, it simply doesn't work. Getting this code with multiple
> >>>>>>> producers and
> >>>>>>>> consumers correct is far from being trivial and the literature
> I
> >>>>>>> browsed
> >>>>>>>> indicates that it is probably not possible given the other
> >> predicates
> >>>>>>> the
> >>>>>>>> code must obey to. Still, optimization is high up on the todo
> list.
> >>>>>>>>
> >>>>>>>> Rainer
> >>>>>>>> _______________________________________________
> >>>>>>>> rsyslog mailing list
> >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>> http://www.rsyslog.com
> >>>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>> http://www.rsyslog.com
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com
> >>>>>>
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com