[Lognorm] getting started document

[david at lang.hm]

thanks.

a couple things that jump out at me as being useful

for numbers, be able to specify interger or decimal (sometimes numbers are 
##.## where they are two independant fields with the decimal as the 
separator

besides the obvious IPv6 there is a need for predefined options

such as anyIP = (IPv4 || IPv6)

or host = (hostname || anyIP)

there are also times when you want to specify multiple words as a single 
field (i.e. next 2 words as field named X)

wildcard to end of line

with word matches, define allowed characters (hostname can be 0-9A-Za-z-_. 
but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)

some way for the parser to report that the ruleset is ambiguous (two rules 
match the same line)

can you change the prefix during the ruleset?


these things will matter more as useage grows (it will become harder and 
harder to specify the difference between things)



I see two forks to this project

fork 1, the parser library (keeping it fast while letting it specify all 
the nuances needed to match everything without ambiguity.


fork 2. the rulesets for common log messages, defining them with common 
termenology.


I can see other projects using the parser, without using the rulesets 
(the rsyslog parser module for example, it's mostly having to define all 
the variations of possible message formats to just figure out how to 
reformat it to a 'standard' syslog format for the other rules to handle)


David Lang


On Tue, 18 Jan 2011, Rainer Gerhards wrote:

> Date: Tue, 18 Jan 2011 08:04:26 +0100
> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Reply-To: lognorm <lognorm at lists.adiscon.com>
> To: lognorm <lognorm at lists.adiscon.com>
> Subject: Re: [Lognorm] getting started document
> 
> The web site is currently being built (any help on content, design, etc is
> deeply appreciated). It is available at
>
> http://www.liblognorm.com
>
> In the menu there is a link to the current *very sparse* documentation.
>
> On data types: there is not yet really such a thing as a "data type" -- all
> are strings. Liblognorm uses a different concept right now, that is
> "parsers". A parser actually describes a syntax that a string must fullfil in
> order to be treated as "correct". There are a number of parsers, which should
> be in the doc (but I see Florian did still not include the new ones, hope he
> will do soon...). It is these types that the rules are build around.
>
> I have recently asked which additional "types" (parsers) are considered
> useful and I am ready to add new ones (I am sure there must be more than we
> currently have).
>
> We also need standard field names. I have started an effort on what to use
> and Florian started a sample directory. We are looking for feedback here.
>
> Finally, but very importantly, I'd like to add a section with rule bases to
> the web site, where device-specifc rules can be found ... and contributed.
> Suggestion on how to do this best would be very appreciated.
>
> I'll also see if I can write a small getting started document. I guess it
> should focus on the technical issues. Right now, I am doing a lot of writing,
> so this should be possible as a side-activity (though I would prefer to code
> a little me, but... ;)).
>
> Hope that helps at least a little bit.
>
> Rainer
>
>> -----Original Message-----
>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, January 18, 2011 12:26 AM
>> To: lognorm
>> Subject: Re: [Lognorm] getting started document
>>
>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
>>
>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>>>> the format of the rulebase files seems pretty straightforward, but
>>>> where can I find the list of what datatypes are supported?
>>>
>>>        Keep in mind that liblognorm is pretty young at this point.
>>> So more documentation probably needs to be done.
>>
>> that's what I figured, and why I posted what i found needed to be done
>> rather than just griping about it :-)
>>
>>> Also, there's not much in the way of rulebase files yet, but that will
>>> likely change in the future.  Right now, you're probably going to have
>> to
>>> write your own rulebase files.
>>
>> I expected to have to write my own rules, but it's hard to write rules
>> without knowing what datatypes are available.
>>
>>> When you do that, keep a copy so that you
>>> can send them in later.  :)
>>
>> I expect to send a bundch in as I work on things, although I also
>> expect
>> to be using the parsing library in ways that don't directly tie in to
>> the 'big normalization' project.
>>
>> For example, Rainer wondered out loud about the possibility of using
>> this
>> library for a rsyslog log parser instead of writing the parsers
>> directly
>> in C. in that case, most of what you want in terms of normalization
>> won't
>> matter, as that parser is just focused on figuring out what part of the
>> blob that arrives is the message, and what metadata in included, not in
>> parsing the message itself into different pieces.
>>
>>>> also, where can I find an example of how to compile the rulebase,
>> and
>>>> an example program that uses the rulebase?
>>>
>>>        Check out "normalizer.c" with the liblognorm git.  It's a
>> basic
>>> program that does normalization.
>>
>> will do.
>>
>> can you point me at what file (or files) the datatypes are defined in?
>>
>> David Lang
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>