rgerhards at hq.adiscon.com
Wed Oct 13 15:29:02 CEST 2010
I have put a copy of the liblognorm API doc (temporarily) up on:
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, October 13, 2010 2:45 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] liblognorm
> Hi David,
> as usual, many thanks for your great thoughts. I had a day of heavy
> yesterday, thus the late response. See below...
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> > Sent: Monday, October 11, 2010 10:33 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] liblognorm
> > On Mon, 11 Oct 2010, Rainer Gerhards wrote:
> > > I have just written another post on the normalization library. It
> > looks like
> > > the design tends to favor a split into two libraries:
> > >
> > > http://blog.gerhards.net/2010/10/splitting-up-normalization-
> > library.html
> > this seems like a good idea.
> > there is a definate need for a good, efficient parsing tool that can
> > used for high volume sites. There are a lot of tools that heavily use
> > regex matching, but those tend to collapse at high volumes.
> > you can create your own parser with lex, yacc, bison, or flex, but th
> > work
> > needed to create the input config file for these (with their specific
> > syntax) is daunting.
> > a tool that could take it's configuration in something that looks
> > similar to log lines (with some sort of syntax to show the variable
> > part),
> > that would then compile into something very effient like the tools
> > above
> > would be very useful for a lot of different tools.
> That's the basic idea, instead that I do not intend to create e.g. lex
> but rather have the engine do that part itself. The main advantage is
> this could be done dynamically. I think this will be possible in almost
> constant time, as long as all fields can be parsed via primitive types
> do not require too much effort to back off).
> I will be working on the parse tree the next days, so you'll hopefully
> able to get an idea of it by looking at the code. At it's heart, it
> simply is
> a radix tree, with constants and field syntaxes definig how the tree is
> > this may just need to be a configuration generator for the tools
> > above that can take the list of annotated lines and create the
> > appropriate
> > config file to build the parser. If this can accept regex lines and
> > then compile them down to a parser tree it would be wonderful.
> Regex is a different beast, because for it you need to create a full-
> DFA, which also explains the slowness of regexes. I'll not tackle that
> For some fields, I will support regex matches, but when they are used,
> performance is affected. The overall idea is that you usually do not
> need any
> regex/DFA at all.
> > so once there is a high performance parser to pull the data apart,
> > the question is what to do with it.
> > some people will want to write it to various places, others will want
> > to
> > make decisions based on what is matched.
> > for those who are wanting to write the normalized output to various
> > places, a plugin structure like rsyslog has (with the ability to
> > the messages based on the various properties that are discovered) is
> > very
> > appealing, and it may make a lot of sense to see what can be done to
> > re-use that work. If so, there will need to be a 'format string' that
> > creates the output with all the properties that are known tagged, but
> > without including ones that didn't have any matches in this log
> > message.
> One thing I definitely intend to do is utilize the library in rsyslog.
> envision a parser module that works based on the library. That also
> rsyslog's core engine must be extended to support the additional
> fields, but
> something that can definitely be done. With that approach, no complex
> engine is needed - one can just use the rsyslog plugin. And with a near
> algorithm, we can probably expect that this happens in real-time even
> very large traffic loads (but probably not for the largest ones).
> It is important to know here that the current parsers also have some
> backout needs, for example for the date and tag/hostname fields. So
> this can
> be done quickly.
> > for those who are wanting to then implement logic based on what it
> > gets,
> > thing get much more interesting. I suspect that the thing to do here
> > will
> > be to make the event normalization engine be something that can be a
> > library included in other programs (in various languages), something
> > that you can have the config file be something along the lines of
> that's actually the idea. An initial sketch of the API is already in
> git and
> I hope to get some better-readbly doxgen-generated interface spec up
> > documentation (hopefully including a sample raw line)
> > line-to-match
> > function to call when matched
> > there are a log of programs out there written to do good and
> > interesting
> > stuff with lines that it receives, if there was an ability to replace
> > their sequential 'does it match rule 1, does it match rule 2' logic
> > with a
> > more efficient parser it would be a huge win.
> > I don't think you are wanting to tackle that portion of the task.
> The lib part yes, but that obviously requires a lot of changes for
> applications using it.
> > David Lang
> > > Rainer
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > >> Sent: Monday, October 11, 2010 9:01 AM
> > >> To: rsyslog-users
> > >> Subject: Re: [rsyslog] liblognorm vs. libeventnorm
> > >>
> > >> I would like to add as an argument pro liblognorm, that many
> > >> probably
> > >> better understand what "log normalization" is whereas "event
> > >> normalization"
> > >> may sound strange. In that sense, liblognorm may be a better name.
> > >> Feedback
> > >> is appreciated.
> > >>
> > >> Rainer
> > >>
> > >>> -----Original Message-----
> > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > >>> Sent: Sunday, October 10, 2010 11:53 AM
> > >>> To: rsyslog-users
> > >>> Subject: [rsyslog] liblognorm vs. libeventnorm
> > >>>
> > >>> Hi all,
> > >>>
> > >>> I think I'll start with the libeventnorm name for the normalizing
> > >>> library
> > >>> instead of liblognorm. Reason here:
> > >>>
> > >>> http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html
> > >>>
> > >>> Further name suggestions or arguments are very welcome!
> > >>>
> > >>> Rainer
> > >>> _______________________________________________
> > >>> rsyslog mailing list
> > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> http://www.rsyslog.com
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> rsyslog mailing list
More information about the rsyslog