[rsyslog] message parser info - was: rsyslog 5.3.4 (devel) released

[Rainer Gerhards]

David, all,

I hope I have answered all questions in this new document:

http://www.rsyslog.com/doc-messageparser.html

I would also appreciate feedback on that part:

======
Note that it is currently under evaluation if rsyslog will support binding
parser chains to specific inputs directly, without depending on the ruleset.
There are some concerns that this may not be necessary but adds considerable
complexity to the configuration. So this may or may not be possible in the
future. In any case, if we decide to add it, input modules need to support
it, so this functionality would require some time to implement.
======

If I implement this, a listener would probably have a parser chain assigned,
it not, the parser chain from the ruleset is used, if not, the parser chain
from the default ruleset is used.

I it (fairly...) trivial to add this capability, but I am really concerned
that the config options get more and more complex...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Thursday, November 05, 2009 8:53 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog 5.3.4 (devel) released
> 
> yOn Thu, 5 Nov 2009, Tom Bergfeld wrote:
> 
> > We have just released rsyslog 5.3.4, a member of the v5-devel branch.
> > Today's release offers a number of important improvements. I would
> like to
> > highlight the ability to nest rulesets [1] (I already announced that)
> and the
> > ability to define custom parsers [2].
> >
> > Let me elaborate a bit on the later. In the syslog world exists a
> myriad of
> > incompatible and invalidly formatted messages. We have had numerous
> support
> > requests on how to parse this or that malformed message format. With
> custom
> > parsers, we now have a vehicel to integrate all these invalid formats
> in an
> > elegant way that is also offering high performance ([2] has a
> concrete
> > sample). The core idea now is that parsers are plugins just like
> input and
> > output modules. It is relatively easy to write such a parser (roughly
> a day,
> > I expect) and custom parsers can be integrated into rsyslogd in a way
> that
> > they only affect messages received on specific port. So far, I have
> not
> > actually created a custom parser, but I hope that over time many of
> them will
> > become available. My hope here is that we actually can build a
> directory,
> > which other folks can browse so that they are able to find a solution
> to the
> > mess their vendor has provided ;)
> 
> this sounds very interesting. however, reading the link I'm a little
> confused.
> 
> on the one hand it talks a lot about the priority of parsers, but then
> it
> talks about binding different parsers to different ports. It's not
> clear
> if these are two different ways to use parsers, or how these two would
> interact.
> 
> where can these parsers be used?
> 
> obviously the imudp module can use them. can all input modules use
> them?
> 
> what are the limits of the parser?
> 
> a couple of extreme examples:
> 
> could you implement relp as a parser for imtcp, or since relp sends
> replies that can't be done (limiting it to different ways of processing
> a
> message that's already been received)
> 
> could a parser detect that it had a piece of a multi-line messsage and
> stash the piece until it receives the rest of the message so that it
> could
> submit the complete message?
> 
> 
> a coouple questions from trying to look through the code at almost 3am
> local time
> 
> if it can easily be done, may I suggest changing the santization flag
> from
> a yes/no boolean to an enum so that there can be more than one
> sanitation
> routine
> 
> do you have the default parser broken out as a seperate file tha canbe
> used as an example?
> 
> David Lang
> 
> > This release also introduces the capability to run rulesets off their
> own
> > queue (also already mentioned on the mailing list). Plus, it contains
> the
> > usual set of bug fixes.
> >
> > We plan to make this version the basis for the next v5-stable.
> >
> >
> > [1] http://www.rsyslog.com/doc-omruleset.html
> > [2] http://www.rsyslog.com/doc-rsconf1_rulesetparser.html
> >
> >
> >
> > ChangeLog:
> >
> > http://www.rsyslog.com/Article423.phtml
> >
> > Download:
> >
> > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-
> 185.phtml
> >
> > As always, feedback is appreciated.
> >
> > Best regards,
> > Tom Bergfeld
> > --
> > Support
> > =======
> >
> > Improving rsyslog is costly, but you can help!  We are looking for
> > organizations that find rsyslog useful and wish to contribute back.
> You can
> > contribute by reporting bugs, improve the software, or donate money
> or
> > equipment.
> >
> > Commercial support contracts for rsyslog are available, and they help
> finance
> > continued maintenance.  Adiscon GmbH, a privately held German
> company, is
> > currently funding rsyslog development. We are always looking for
> interesting
> > development projects. For details on how to help, please see
> > http://www.rsyslog.com/doc-how2help.html .
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com