[Lognorm] getting started document

[david at lang.hm]

based on another thread, some additional things that are needed (or at 
least need to be documented)

how to escape a '%'

it would be good to have some regex features available for the rules.

ones that I think are very useful

alternate words (this|that|other)

in word matches, being able to specify that this must match a pattern as 
well (eth* or [sh]da[0-9]* for example)


I don't know how far it makes sense to go, but if the regex can be 
compiled down to the parse tree with the other rules it shouldn't hurt 
performance, and if we can use regex rules that people have created for 
other tools, it should jumpstart the ruleset development.

David Lang

On Tue, 18 Jan 2011, david at lang.hm wrote:

> a couple things that jump out at me as being useful
>
> for numbers, be able to specify interger or decimal (sometimes numbers are 
> ##.## where they are two independant fields with the decimal as the separator
>
> besides the obvious IPv6 there is a need for predefined options
>
> such as anyIP = (IPv4 || IPv6)
>
> or host = (hostname || anyIP)
>
> there are also times when you want to specify multiple words as a single 
> field (i.e. next 2 words as field named X)
>
> wildcard to end of line
>
> with word matches, define allowed characters (hostname can be 0-9A-Za-z-_. 
> but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
>
> some way for the parser to report that the ruleset is ambiguous (two rules 
> match the same line)
>
> can you change the prefix during the ruleset?
>
>
> these things will matter more as useage grows (it will become harder and 
> harder to specify the difference between things)
>
>
>
> I see two forks to this project
>
> fork 1, the parser library (keeping it fast while letting it specify all the 
> nuances needed to match everything without ambiguity.
>
>
> fork 2. the rulesets for common log messages, defining them with common 
> termenology.
>
>
> I can see other projects using the parser, without using the rulesets (the 
> rsyslog parser module for example, it's mostly having to define all the 
> variations of possible message formats to just figure out how to reformat it 
> to a 'standard' syslog format for the other rules to handle)
>
>
> David Lang
>
>
> On Tue, 18 Jan 2011, Rainer Gerhards wrote:
>
>> Date: Tue, 18 Jan 2011 08:04:26 +0100
>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> Reply-To: lognorm <lognorm at lists.adiscon.com>
>> To: lognorm <lognorm at lists.adiscon.com>
>> Subject: Re: [Lognorm] getting started document
>> 
>> The web site is currently being built (any help on content, design, etc is
>> deeply appreciated). It is available at
>> 
>> http://www.liblognorm.com
>> 
>> In the menu there is a link to the current *very sparse* documentation.
>> 
>> On data types: there is not yet really such a thing as a "data type" -- all
>> are strings. Liblognorm uses a different concept right now, that is
>> "parsers". A parser actually describes a syntax that a string must fullfil 
>> in
>> order to be treated as "correct". There are a number of parsers, which 
>> should
>> be in the doc (but I see Florian did still not include the new ones, hope 
>> he
>> will do soon...). It is these types that the rules are build around.
>> 
>> I have recently asked which additional "types" (parsers) are considered
>> useful and I am ready to add new ones (I am sure there must be more than we
>> currently have).
>> 
>> We also need standard field names. I have started an effort on what to use
>> and Florian started a sample directory. We are looking for feedback here.
>> 
>> Finally, but very importantly, I'd like to add a section with rule bases to
>> the web site, where device-specifc rules can be found ... and contributed.
>> Suggestion on how to do this best would be very appreciated.
>> 
>> I'll also see if I can write a small getting started document. I guess it
>> should focus on the technical issues. Right now, I am doing a lot of 
>> writing,
>> so this should be possible as a side-activity (though I would prefer to 
>> code
>> a little me, but... ;)).
>> 
>> Hope that helps at least a little bit.
>> 
>> Rainer
>> 
>>> -----Original Message-----
>>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>> Sent: Tuesday, January 18, 2011 12:26 AM
>>> To: lognorm
>>> Subject: Re: [Lognorm] getting started document
>>> 
>>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
>>> 
>>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>>>>> the format of the rulebase files seems pretty straightforward, but
>>>>> where can I find the list of what datatypes are supported?
>>>>
>>>>        Keep in mind that liblognorm is pretty young at this point.
>>>> So more documentation probably needs to be done.
>>> 
>>> that's what I figured, and why I posted what i found needed to be done
>>> rather than just griping about it :-)
>>> 
>>>> Also, there's not much in the way of rulebase files yet, but that will
>>>> likely change in the future.  Right now, you're probably going to have
>>> to
>>>> write your own rulebase files.
>>> 
>>> I expected to have to write my own rules, but it's hard to write rules
>>> without knowing what datatypes are available.
>>> 
>>>> When you do that, keep a copy so that you
>>>> can send them in later.  :)
>>> 
>>> I expect to send a bundch in as I work on things, although I also
>>> expect
>>> to be using the parsing library in ways that don't directly tie in to
>>> the 'big normalization' project.
>>> 
>>> For example, Rainer wondered out loud about the possibility of using
>>> this
>>> library for a rsyslog log parser instead of writing the parsers
>>> directly
>>> in C. in that case, most of what you want in terms of normalization
>>> won't
>>> matter, as that parser is just focused on figuring out what part of the
>>> blob that arrives is the message, and what metadata in included, not in
>>> parsing the message itself into different pieces.
>>> 
>>>>> also, where can I find an example of how to compile the rulebase,
>>> and
>>>>> an example program that uses the rulebase?
>>>>
>>>>        Check out "normalizer.c" with the liblognorm git.  It's a
>>> basic
>>>> program that does normalization.
>>> 
>>> will do.
>>> 
>>> can you point me at what file (or files) the datatypes are defined in?
>>> 
>>> David Lang
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>> 
>