[Lognorm] getting started document

[Rainer Gerhards]

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, January 18, 2011 9:39 AM
> To: lognorm
> Subject: Re: [Lognorm] getting started document
> 
> thanks.
> 
> a couple things that jump out at me as being useful
> 
> for numbers, be able to specify interger or decimal (sometimes numbers
> are
> ##.## where they are two independant fields with the decimal as the
> separator
> 
> besides the obvious IPv6 there is a need for predefined options
> 
> such as anyIP = (IPv4 || IPv6)
> 
> or host = (hostname || anyIP)
> 
> there are also times when you want to specify multiple words as a
> single
> field (i.e. next 2 words as field named X)
> 
> wildcard to end of line
> 
> with word matches, define allowed characters (hostname can be 0-9A-Za-
> z-_.
> but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
> 
> some way for the parser to report that the ruleset is ambiguous (two
> rules
> match the same line)

That's an error, in this case the rulebase does not load.

> 
> can you change the prefix during the ruleset?

yes

> 
> 
> these things will matter more as useage grows (it will become harder
> and
> harder to specify the difference between things)
> 
> 
> 
> I see two forks to this project
> 
> fork 1, the parser library (keeping it fast while letting it specify
> all
> the nuances needed to match everything without ambiguity.
> 
> 
> fork 2. the rulesets for common log messages, defining them with common
> termenology.
> 
> 
> I can see other projects using the parser, without using the rulesets
> (the rsyslog parser module for example, it's mostly having to define
> all
> the variations of possible message formats to just figure out how to
> reformat it to a 'standard' syslog format for the other rules to
> handle)

actually, liblognorm is what you describe as fork 1. The "fork 2" project is
what I try to achieve via a website. My goal here is to have the fork 2
things available for everyone who wants. For example, in rsyslog all I expect
you need is a different prefix, but otherwise the rulebases should be usable.

Rainer
> 
> 
> David Lang
> 
> 
> On Tue, 18 Jan 2011, Rainer Gerhards wrote:
> 
> > Date: Tue, 18 Jan 2011 08:04:26 +0100
> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> > Reply-To: lognorm <lognorm at lists.adiscon.com>
> > To: lognorm <lognorm at lists.adiscon.com>
> > Subject: Re: [Lognorm] getting started document
> >
> > The web site is currently being built (any help on content, design,
> etc is
> > deeply appreciated). It is available at
> >
> > http://www.liblognorm.com
> >
> > In the menu there is a link to the current *very sparse*
> documentation.
> >
> > On data types: there is not yet really such a thing as a "data type"
> -- all
> > are strings. Liblognorm uses a different concept right now, that is
> > "parsers". A parser actually describes a syntax that a string must
> fullfil in
> > order to be treated as "correct". There are a number of parsers,
> which should
> > be in the doc (but I see Florian did still not include the new ones,
> hope he
> > will do soon...). It is these types that the rules are build around.
> >
> > I have recently asked which additional "types" (parsers) are
> considered
> > useful and I am ready to add new ones (I am sure there must be more
> than we
> > currently have).
> >
> > We also need standard field names. I have started an effort on what
> to use
> > and Florian started a sample directory. We are looking for feedback
> here.
> >
> > Finally, but very importantly, I'd like to add a section with rule
> bases to
> > the web site, where device-specifc rules can be found ... and
> contributed.
> > Suggestion on how to do this best would be very appreciated.
> >
> > I'll also see if I can write a small getting started document. I
> guess it
> > should focus on the technical issues. Right now, I am doing a lot of
> writing,
> > so this should be possible as a side-activity (though I would prefer
> to code
> > a little me, but... ;)).
> >
> > Hope that helps at least a little bit.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Tuesday, January 18, 2011 12:26 AM
> >> To: lognorm
> >> Subject: Re: [Lognorm] getting started document
> >>
> >> From: "Champ Clark III [Softwink]" <champ at softwink.com>
> >>
> >>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> >>>> the format of the rulebase files seems pretty straightforward, but
> >>>> where can I find the list of what datatypes are supported?
> >>>
> >>>        Keep in mind that liblognorm is pretty young at this point.
> >>> So more documentation probably needs to be done.
> >>
> >> that's what I figured, and why I posted what i found needed to be
> done
> >> rather than just griping about it :-)
> >>
> >>> Also, there's not much in the way of rulebase files yet, but that
> will
> >>> likely change in the future.  Right now, you're probably going to
> have
> >> to
> >>> write your own rulebase files.
> >>
> >> I expected to have to write my own rules, but it's hard to write
> rules
> >> without knowing what datatypes are available.
> >>
> >>> When you do that, keep a copy so that you
> >>> can send them in later.  :)
> >>
> >> I expect to send a bundch in as I work on things, although I also
> >> expect
> >> to be using the parsing library in ways that don't directly tie in
> to
> >> the 'big normalization' project.
> >>
> >> For example, Rainer wondered out loud about the possibility of using
> >> this
> >> library for a rsyslog log parser instead of writing the parsers
> >> directly
> >> in C. in that case, most of what you want in terms of normalization
> >> won't
> >> matter, as that parser is just focused on figuring out what part of
> the
> >> blob that arrives is the message, and what metadata in included, not
> in
> >> parsing the message itself into different pieces.
> >>
> >>>> also, where can I find an example of how to compile the rulebase,
> >> and
> >>>> an example program that uses the rulebase?
> >>>
> >>>        Check out "normalizer.c" with the liblognorm git.  It's a
> >> basic
> >>> program that does normalization.
> >>
> >> will do.
> >>
> >> can you point me at what file (or files) the datatypes are defined
> in?
> >>
> >> David Lang
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm