From champ at softwink.com  Thu Jan 13 16:34:09 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Thu, 13 Jan 2011 10:34:09 -0500
Subject: [Lognorm] Sonicwall Normalization tests & other....
Message-ID: <20110113153409.GA15921@bundy.vistech.net>


Yay!  The new liblognorm mailing list!  

FP!  Woo.  Okay,  silliness aside. 

I started doing some "serious" log normalization with liblognorm while working
a new sonicwall.rules for Sagan (and a sonicwall.rulebase for liblognorm). 
The good news,  after about 20 hours of running,  it's been normalizing 
wonderfully and hasn't "blown up".   Here's an example of what it's currently
"normalizing" (sanitized):

id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1 pri=1 c=32 m=608  msg="IPS Detection Alert: ICMP Echo Reply" sid=316 ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0

I have a pretty generic Sagan rule to be triggered: 

alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-event; normalize: sonicwall; reference: url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)

(Note the "normalize: sonicwall" flag).

The liblognorm rule looks like thus (from my sonicwall.rulebase). 

--<snip>----

prefix=id=%firewall:word% sn=%serial:word% time="%date:word% %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4% pri=%pri:number% c=%c:number% m=%m:number%

rule=:  msg="IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number% n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word% dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%

--<snip>----

	Actually,  for the "IPS Detection Alert",  I had to write multiple liblognorm
rules.  This it might be "IPS Detection: ICMP PING" or "IPS Detection: ICMP PING 
reply" (ie - multiple fields).  Over all,  I'm okay with this.   However,  I did 
run into some interesting issues: 

1. Note the "time="%date:word% %hour:number%:%minute:number%:%seconds:number%"
   in the prefix.  There probably needs to be a parser for date (2011-01-13)
   format.  There probably also needs to be a parser for the 24 clock 
   (10:05:14).   Doing something like "%hour:word% doesn't work,  so I'm
   breaking up the fields manually.  I'm assuming this is caused by the
   :'s. 

2. Check out the "msg=" portion.  If you look closely/compare the "real"
   output with the lognorm rule,  it's missing a ".   For example,  the
   log line is actually: 

   msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....

   But the rule is: 

   IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%

   The last %t3:word% captures not only the item in the string but the
   " as well.   Not sure if that's a "issue",  but thought I'd mention
   it. 

   Lastly,  on a side note,  I've had to alter my rsyslog.conf to the
following: 

$template sagan, "%fromhost-ip%|%syslogfacility-text%|%syslogpriority-text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-rfc3339%|%timegenerated:12:19:date-rfc3339%|%programname%|%syslogtag%%msg%\n"

	Note the end of the line.  For some reason,  syslog-ng likes to 
prepend the 'program[pid]' as part of the message.  I researched this,  and
I don't see any way to remove it via syslog-ng,  so my "fix" was to 
have rsyslog do the same.  It's annoying,  but I have a feeling that bringing
it up with the syslog-ng team will just lead to bickering about what "should"
be part of the %msg% (or $MSG) and what shouldn't be.  

	Also,  without the CEE definitions,  I'm obviously "making up" field
names as I go.   If you have any pointers to "real" CEE definitions of fields,
that would help.  If I recall,   they haven't been finalized/released yet (?)


	Anyways,  that's all.. Let me know what you thing. 


-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110113/b6779009/attachment.pgp>

From rgerhards at hq.adiscon.com  Thu Jan 13 17:23:55 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 13 Jan 2011 17:23:55 +0100
Subject: [Lognorm] Sonicwall Normalization tests & other....
References: <20110113153409.GA15921@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Thursday, January 13, 2011 4:34 PM
> To: lognorm at lists.adiscon.com
> Subject: [Lognorm] Sonicwall Normalization tests & other....
> 
> 
> Yay!  The new liblognorm mailing list!

Welcome, you managed the first-ever post ;)
 
> 
> FP!  Woo.  Okay,  silliness aside.
> 
> I started doing some "serious" log normalization with liblognorm while
> working
> a new sonicwall.rules for Sagan (and a sonicwall.rulebase for
> liblognorm).
> The good news,  after about 20 hours of running,  it's been normalizing
> wonderfully and hasn't "blown up".   Here's an example of what it's
> currently
> "normalizing" (sanitized):
> 
> id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1
> pri=1 c=32 m=608  msg="IPS Detection Alert: ICMP Echo Reply" sid=316
> ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0
> 
> I have a pretty generic Sagan rule to be triggered:
> 
> alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS
> Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-
> event; normalize: sonicwall; reference:
> url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)
> 
> (Note the "normalize: sonicwall" flag).
> 
> The liblognorm rule looks like thus (from my sonicwall.rulebase).
> 
> --<snip>----
> 
> prefix=id=%firewall:word% sn=%serial:word% time="%date:word%
> %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4%
> pri=%pri:number% c=%c:number% m=%m:number%
> 
> rule=:  msg="IPS Detection Alert: %t1:word% %t2:word %t3:word%
> sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number%
> n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word%
> dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%
> 
> --<snip>----
> 
> 	Actually,  for the "IPS Detection Alert",  I had to write
> multiple liblognorm
> rules.

I came across this with iptables as welll. The root cause is that the data is
(at least) semi-structured: it already has name-value pairs, but the order is
not necessarily normalized. One idea I have is to create a special parser
that pulls all name-value pairs and assigns the appropriate fields. However,
there are many subtle issues with that (the most important one probably is
the need to rename fields so that they have a canonical form -- I have not
yet thought enough about that [in terms of actual config syntax]).

>  This it might be "IPS Detection: ICMP PING" or "IPS Detection:
> ICMP PING
> reply" (ie - multiple fields).  Over all,  I'm okay with this.

Oh yeah, that's another one, but I think can not be solved by keeping it
efficient.

> However,  I did
> run into some interesting issues:
> 
> 1. Note the "time="%date:word%
> %hour:number%:%minute:number%:%seconds:number%"
>    in the prefix.  There probably needs to be a parser for date (2011-
> 01-13)
>    format.  There probably also needs to be a parser for the 24 clock
>    (10:05:14).   Doing something like "%hour:word% doesn't work,  so
> I'm
>    breaking up the fields manually.  I'm assuming this is caused by the
>    :'s.

OK, parsers needed:
ISO-Date
Timestamps (12/24 hour clocks)
integer (don't we have this already)

Word is terminated by a space, the integer would be terminated by the first
non-digit. Hex numbers would probably be useful as well.

I'll see that I implement these.

> 
> 2. Check out the "msg=" portion.  If you look closely/compare the
> "real"
>    output with the lognorm rule,  it's missing a ".   For example,  the
>    log line is actually:
> 
>    msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....
> 
>    But the rule is:
> 
>    IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%

OT: there it is "number" is an integer (oh, the joys of vacation and
forgetting things... ;)

> 
>    The last %t3:word% captures not only the item in the string but the
>    " as well.   Not sure if that's a "issue",  but thought I'd mention
>    it.

Good point! Next parser:
quoted String! Very important one.

> 
>    Lastly,  on a side note,  I've had to alter my rsyslog.conf to the
> following:
> 
> $template sagan, "%fromhost-ip%|%syslogfacility-text%|%syslogpriority-
> text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-
> rfc3339%|%timegenerated:12:19:date-
> rfc3339%|%programname%|%syslogtag%%msg%\n"
> 
> 	Note the end of the line.  For some reason,  syslog-ng likes to
> prepend the 'program[pid]' as part of the message.  I researched this,
> and
> I don't see any way to remove it via syslog-ng,  so my "fix" was to
> have rsyslog do the same.  It's annoying,  but I have a feeling that
> bringing
> it up with the syslog-ng team will just lead to bickering about what
> "should"
> be part of the %msg% (or $MSG) and what shouldn't be.

probably, but I can't do much against this ;) (RFC3164 says it is not...)
> 
> 	Also,  without the CEE definitions,  I'm obviously "making up"
> field
> names as I go.   If you have any pointers to "real" CEE definitions of
> fields,
> that would help.  If I recall,   they haven't been finalized/released
> yet (?)

still the same status. I think it takes another few weeks until this changes.
I suggest that in the interim we create a dictionary of field names/semantics
on liblognorm.org as an interim solution. The good news is that I could add
them as an alias to the then-finalized CEE dictionary and make libee replace
aliases with "the real thing". That means you do not need to change things
manually, at least at this level.

I'll see that we setup an area for the dictionary on the website very soon. I
hope for your feedback on getting this right ;)

Rainer
> 
> 
> 	Anyways,  that's all.. Let me know what you thing.
> 
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From rgerhards at hq.adiscon.com  Thu Jan 13 18:48:22 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 13 Jan 2011 18:48:22 +0100
Subject: [Lognorm] Sonicwall Normalization tests & other....
References: <20110113153409.GA15921@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>
	<20110113173334.GA19911@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>

> 	Yes,  quotestring would be a powerful one..  Here's what I mean
> about the "word"....  Assume you have this message:
> 
> msg="this is a test"
> 
> 	If my rule is:
> 
> msg="%w1:word %w2:word %w3:word% %w4:word%
> 
> 	The values will turn out like:
> 
> w1 = this
> w2 = is
> w3 = a
> w4 = test"
> 
> 	Note the "

It just occured to me that you can use char-to! Minimal doc:

http://www.liblognorm.com/files/manual/sampledatabase.htm

Rainer

> 
> 	The quotestring is great,  because then you can deal with multi-
> values within a string..  The question is,  do you do it:
> 
> msg="%string:quotestring%"
> 
> 	or
> 
> msg=%string:quotestring%
> 
> 	Probably doesn't matter.  Just need to make sure it's documented
> on "how" quotestring works.
> 
> 
> > probably, but I can't do much against this ;) (RFC3164 says it is
> not...)
> 
> 	My thought is syslog-ng is doing this wrong.  I might bring it
> up on the syslog-ng mailing list,  as I see no other way around this
> issue.  My other thought is that it'll lead to arguments that I really
> don't want to be a part of... We'll see.
> 
> > > 	Also,  without the CEE definitions,  I'm obviously "making up"
> > > field
> > > names as I go.   If you have any pointers to "real" CEE definitions
> of
> > > fields,
> > > that would help.  If I recall,   they haven't been
> finalized/released
> > > yet (?)
> >
> > still the same status. I think it takes another few weeks until this
> changes.
> > I suggest that in the interim we create a dictionary of field
> names/semantics
> > on liblognorm.org as an interim solution. The good news is that I
> could add
> > them as an alias to the then-finalized CEE dictionary and make libee
> replace
> > aliases with "the real thing". That means you do not need to change
> things
> > manually, at least at this level.
> >
> > I'll see that we setup an area for the dictionary on the website very
> soon. I
> > hope for your feedback on getting this right ;)
> 
> 	That sounds like a good idea.  I know nothing is finalized yet,
> but there enough work to be done that it wouldn't be good to wait
> around.
> 
> 	Oh,  you'll be getting feedback from me :)  My tests with the
> sonicwall rules has gone very well.  On a side note,  I'll be working
> with a new client soon (about 3 weeks) in an environment that pushes
> about 60-80 million log lines a day.   I'll be able to push
> Sagan/liblognorm in there for testing,  which should be really
> interesting.   Those numbers will likely go down a good bit,  as I
> suspect a lot of "crap" noise being injected.
> 
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Thu Jan 13 19:13:29 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Thu, 13 Jan 2011 13:13:29 -0500
Subject: [Lognorm] Sonicwall Normalization tests & other....
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
References: <20110113153409.GA15921@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>
	<20110113173334.GA19911@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
Message-ID: <20110113181329.GB21448@bundy.vistech.net>

> It just occured to me that you can use char-to! Minimal doc:
> http://www.liblognorm.com/files/manual/sampledatabase.htm

	Ah! :) 

rule=:  msg="%alert:char-to:\x22%" sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number% n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word% dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%

	The above worked perfect.  One thought,   It might be beneficial 
to make a "%field:quotestring" (or something similar).  Basically,  it'd
do exactly what char-to does,  but with a pre-set field (x22 in my
case).   My thinking is that it'll make rules "easier" to write.  Ie -

msg="%alert:quotestring%" "looks" nicer.  I could be wrong on this.  Of
course then you'd probably want to have several "preset" values.. ie 

	" ' : ( )

	Hmmph.. Might be easier to just stay with char-to.  This is
interesting because I thought char-to did something different.  Very
nice Rainer.  

	On a side note,  syslog-ng supports a $MSGONLY so my old problem
it moot.  ($MSG by defaults includes the program... hrmph).

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110113/0d6a7ddd/attachment.pgp>

From champ at softwink.com  Thu Jan 13 19:34:13 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Thu, 13 Jan 2011 13:34:13 -0500
Subject: [Lognorm] Sonicwall Normalization tests & other....
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
References: <20110113153409.GA15921@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>
	<20110113173334.GA19911@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
Message-ID: <20110113183413.GA22721@bundy.vistech.net>


	Wow.. This is nice..  I was also able to do this (for UDP/TCP
port scans on the Sonicwall): 

rule=: msg="Possible port scan detected" n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word% dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word% note="%ports-scanned:char-to:\x22%" 

	Went from about 10 (or so) liblognorm rules to 2.  At the end of
the line (note=) would display multiple values (for example: 

note="UDP scanned port list, 58797, 21923, 21405, 40539, 22739"

	Of course, "note=" could have various values.  I don't really 
"need" this information (for input into the database),  but it makes the
parsing extremely more flexable. 

	Oh,  side note.  When I reply to the list,  I reply to the
sender directly.  Do you want replies to go to the list?


-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110113/6ff38cea/attachment.pgp>

From rgerhards at hq.adiscon.com  Fri Jan 14 09:01:35 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Jan 2011 09:01:35 +0100
Subject: [Lognorm] Sonicwall Normalization tests & other....
References: <20110113153409.GA15921@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA03@GRFEXC.intern.adiscon.com>

I just committed a couple of new parsers (only the hex number is still
missing), see these commits:

http://git.adiscon.com/?p=libee.git;a=commitdiff;h=429915802b1a37bd6e0283888f
59978e7258f3a3
http://git.adiscon.com/?p=liblognorm.git;a=commitdiff;h=f63f04cfd08ab1611b761
93cf0a80b036c3cd6e2

I have done basic testing, but it would be nice if you could play a little
bit with them. Doc is upcoming, but their use should be straightforward from
the source comments. QuotedString pulls the quotes, but does not make them
part of the extracted field. The parse pointer is placed right after the
closing quote.

I intend to do new releases early next week with the new features and bug
fixes.

As always, feedback is appreciated.

Rainer

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, January 13, 2011 5:24 PM
> To: Champ Clark III [Softwink]
> Cc: lognorm at lists.adiscon.com
> Subject: Re: [Lognorm] Sonicwall Normalization tests & other....
> 
> > -----Original Message-----
> > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > Sent: Thursday, January 13, 2011 4:34 PM
> > To: lognorm at lists.adiscon.com
> > Subject: [Lognorm] Sonicwall Normalization tests & other....
> >
> >
> > Yay!  The new liblognorm mailing list!
> 
> Welcome, you managed the first-ever post ;)
> 
> >
> > FP!  Woo.  Okay,  silliness aside.
> >
> > I started doing some "serious" log normalization with liblognorm
> while
> > working
> > a new sonicwall.rules for Sagan (and a sonicwall.rulebase for
> > liblognorm).
> > The good news,  after about 20 hours of running,  it's been
> normalizing
> > wonderfully and hasn't "blown up".   Here's an example of what it's
> > currently
> > "normalizing" (sanitized):
> >
> > id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1
> > pri=1 c=32 m=608  msg="IPS Detection Alert: ICMP Echo Reply" sid=316
> > ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0
> >
> > I have a pretty generic Sagan rule to be triggered:
> >
> > alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS
> > Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-
> > event; normalize: sonicwall; reference:
> > url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)
> >
> > (Note the "normalize: sonicwall" flag).
> >
> > The liblognorm rule looks like thus (from my sonicwall.rulebase).
> >
> > --<snip>----
> >
> > prefix=id=%firewall:word% sn=%serial:word% time="%date:word%
> > %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4%
> > pri=%pri:number% c=%c:number% m=%m:number%
> >
> > rule=:  msg="IPS Detection Alert: %t1:word% %t2:word %t3:word%
> > sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number%
> > n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word%
> > dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%
> >
> > --<snip>----
> >
> > 	Actually,  for the "IPS Detection Alert",  I had to write
> > multiple liblognorm
> > rules.
> 
> I came across this with iptables as welll. The root cause is that the
> data is
> (at least) semi-structured: it already has name-value pairs, but the
> order is
> not necessarily normalized. One idea I have is to create a special
> parser
> that pulls all name-value pairs and assigns the appropriate fields.
> However,
> there are many subtle issues with that (the most important one probably
> is
> the need to rename fields so that they have a canonical form -- I have
> not
> yet thought enough about that [in terms of actual config syntax]).
> 
> >  This it might be "IPS Detection: ICMP PING" or "IPS Detection:
> > ICMP PING
> > reply" (ie - multiple fields).  Over all,  I'm okay with this.
> 
> Oh yeah, that's another one, but I think can not be solved by keeping
> it
> efficient.
> 
> > However,  I did
> > run into some interesting issues:
> >
> > 1. Note the "time="%date:word%
> > %hour:number%:%minute:number%:%seconds:number%"
> >    in the prefix.  There probably needs to be a parser for date
> (2011-
> > 01-13)
> >    format.  There probably also needs to be a parser for the 24 clock
> >    (10:05:14).   Doing something like "%hour:word% doesn't work,  so
> > I'm
> >    breaking up the fields manually.  I'm assuming this is caused by
> the
> >    :'s.
> 
> OK, parsers needed:
> ISO-Date
> Timestamps (12/24 hour clocks)
> integer (don't we have this already)
> 
> Word is terminated by a space, the integer would be terminated by the
> first
> non-digit. Hex numbers would probably be useful as well.
> 
> I'll see that I implement these.
> 
> >
> > 2. Check out the "msg=" portion.  If you look closely/compare the
> > "real"
> >    output with the lognorm rule,  it's missing a ".   For example,
> the
> >    log line is actually:
> >
> >    msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....
> >
> >    But the rule is:
> >
> >    IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%
> 
> OT: there it is "number" is an integer (oh, the joys of vacation and
> forgetting things... ;)
> 
> >
> >    The last %t3:word% captures not only the item in the string but
> the
> >    " as well.   Not sure if that's a "issue",  but thought I'd
> mention
> >    it.
> 
> Good point! Next parser:
> quoted String! Very important one.
> 
> >
> >    Lastly,  on a side note,  I've had to alter my rsyslog.conf to the
> > following:
> >
> > $template sagan, "%fromhost-ip%|%syslogfacility-
> text%|%syslogpriority-
> > text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-
> > rfc3339%|%timegenerated:12:19:date-
> > rfc3339%|%programname%|%syslogtag%%msg%\n"
> >
> > 	Note the end of the line.  For some reason,  syslog-ng likes to
> > prepend the 'program[pid]' as part of the message.  I researched
> this,
> > and
> > I don't see any way to remove it via syslog-ng,  so my "fix" was to
> > have rsyslog do the same.  It's annoying,  but I have a feeling that
> > bringing
> > it up with the syslog-ng team will just lead to bickering about what
> > "should"
> > be part of the %msg% (or $MSG) and what shouldn't be.
> 
> probably, but I can't do much against this ;) (RFC3164 says it is
> not...)
> >
> > 	Also,  without the CEE definitions,  I'm obviously "making up"
> > field
> > names as I go.   If you have any pointers to "real" CEE definitions
> of
> > fields,
> > that would help.  If I recall,   they haven't been finalized/released
> > yet (?)
> 
> still the same status. I think it takes another few weeks until this
> changes.
> I suggest that in the interim we create a dictionary of field
> names/semantics
> on liblognorm.org as an interim solution. The good news is that I could
> add
> them as an alias to the then-finalized CEE dictionary and make libee
> replace
> aliases with "the real thing". That means you do not need to change
> things
> manually, at least at this level.
> 
> I'll see that we setup an area for the dictionary on the website very
> soon. I
> hope for your feedback on getting this right ;)
> 
> Rainer
> >
> >
> > 	Anyways,  that's all.. Let me know what you thing.
> >
> >
> > --
> >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> >                      http://www.softwink.com
> >
> > GPG Key ID: 58A2A58F
> > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

From rgerhards at hq.adiscon.com  Fri Jan 14 09:05:29 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Jan 2011 09:05:29 +0100
Subject: [Lognorm] replies to list
References: <20110113153409.GA15921@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com><20110113173334.GA19911@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
	<20110113183413.GA22721@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA04@GRFEXC.intern.adiscon.com>

> 	Oh,  side note.  When I reply to the list,  I reply to the
> sender directly.  Do you want replies to go to the list?

mmhhh... I just re-checked, and I have set the mailman config to set the
reply-to header to the list. So far, this always worked, but obviously not
this time. Will try to dig out why, in the meantime please reply to all.

Thanks,
Rainer

From rgerhards at hq.adiscon.com  Fri Jan 14 09:07:28 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 14 Jan 2011 09:07:28 +0100
Subject: [Lognorm] replies to list
References: <20110113153409.GA15921@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com><20110113173334.GA19911@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com><20110113183413.GA22721@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA04@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA05@GRFEXC.intern.adiscon.com>

gosh... it works. I guess I used an older reply mail for my reply. Actually,
I initially forgot the change that mailman setting and did it after a few
messages were exchanged. Looks like I trapped myself ;)

Rainer 

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Friday, January 14, 2011 9:05 AM
> To: lognorm
> Subject: [Lognorm] replies to list
> 
> > 	Oh,  side note.  When I reply to the list,  I reply to the
> > sender directly.  Do you want replies to go to the list?
> 
> mmhhh... I just re-checked, and I have set the mailman config to set
> the
> reply-to header to the list. So far, this always worked, but obviously
> not
> this time. Will try to dig out why, in the meantime please reply to
> all.
> 
> Thanks,
> Rainer
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

From friedl at hq.adiscon.com  Fri Jan 14 16:33:51 2011
From: friedl at hq.adiscon.com (Florian Riedl)
Date: Fri, 14 Jan 2011 16:33:51 +0100
Subject: [Lognorm] liblognorm default field names
References: <20110113153409.GA15921@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com><20110113173334.GA19911@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com><20110113183413.GA22721@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA04@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA0E@GRFEXC.intern.adiscon.com>

Dear liblognorm fans,

we are currently trying to compile some sort of list for field names that
should be a standard in liblognorm. 
	
We already thought about a few. Like the following:
Src-ip = Source IP -> IPv4
Dst-ip = Destination IP -> IPv4

So this is not yet completely thought through. What we definitely need, is a
list of field names, that is most generic and definite. Like the examples
above. With the field name you already have some kind of description, like
the Source IP. It is directly clear what content it has and works for any
kind of message that contains this sort of information.

That is, why I want to ask everyone on the list to provide suggestions for
default fieldnames. We have compiled a list already and put it on the
website. Accepted suggestions will be put there. You can find the list here:
http://www.liblognorm.com/field-names/

I want to ask one more thing: If someone has a striking and brilliant idea of
how to have this list sorted or best displayed and organized, feel free to
tell us. This will not only be good for us to have this list more accessible,
but this is a good contribution for the whole community.

Looking forward to all suggestions.

Florian


From champ at softwink.com  Mon Jan 17 21:11:08 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Mon, 17 Jan 2011 15:11:08 -0500
Subject: [Lognorm] liblognorm default field names
Message-ID: <20110117201108.GA13629@bundy.vistech.net>


	I've incorporated the 'quoted-string',  'date-iso', and 'time-24hr' 
into my Sonicwall test rules.  So far,  looks great.  

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110117/11d7be6b/attachment.pgp>

From david at lang.hm  Mon Jan 17 22:33:00 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 17 Jan 2011 13:33:00 -0800 (PST)
Subject: [Lognorm] getting started document
In-Reply-To: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>
Message-ID: <alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm>

the format of the rulebase files seems pretty straightforward, but where 
can I find the list of what datatypes are supported?

also, where can I find an example of how to compile the rulebase, and an 
example program that uses the rulebase?

David Lang

On Mon, 17 Jan 2011, david at lang.hm wrote:

> documentation of how to get started is not easy to find (if it exists yet)
>
> I found how to download via git
>
> git clone git://git.adiscon.com/git/liblognorm liblognorm
>
> and then did autoreconf -fvi followed by ./configure but configure died 
> because it needs libestr
>
> this seems to be available via git
>
> git clone git://git.adiscon.com/git/libestr libestr
>
> then did autoreconf -fvi followed by ./configure ; make ; make install
>
> I then found that libee is needed
>
>  git clone git://git.adiscon.com/git/libee libee
>
> autoreconf -fvi; ./configure ; make ; make install
>
> this finally let the ./configure; make ; make install work for liblognorm
>
>
> now to hunt for how to use it ;-)
>
> David Lang
>

From david at lang.hm  Mon Jan 17 22:22:44 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 17 Jan 2011 13:22:44 -0800 (PST)
Subject: [Lognorm] getting started document
Message-ID: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>

documentation of how to get started is not easy to find (if it exists yet)

I found how to download via git

  git clone git://git.adiscon.com/git/liblognorm liblognorm

and then did autoreconf -fvi followed by ./configure but configure died 
because it needs libestr

this seems to be available via git

  git clone git://git.adiscon.com/git/libestr libestr

then did autoreconf -fvi followed by ./configure ; make ; make install

I then found that libee is needed

   git clone git://git.adiscon.com/git/libee libee

autoreconf -fvi; ./configure ; make ; make install

this finally let the ./configure; make ; make install work for liblognorm


now to hunt for how to use it ;-)

David Lang

From champ at softwink.com  Mon Jan 17 23:53:13 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Mon, 17 Jan 2011 17:53:13 -0500
Subject: [Lognorm] getting started document
In-Reply-To: <alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>
	<alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm>
Message-ID: <20110117225313.GA16101@bundy.vistech.net>

On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> the format of the rulebase files seems pretty straightforward, but where 
> can I find the list of what datatypes are supported?

	Keep in mind that liblognorm is pretty young at this point. 
So more documentation probably needs to be done.  Also,  there's not
much in the way of rulebase files yet,  but that will likely change
in the future.  Right now,  you're probably going to have to write your
own rulebase files.  When you do that,  keep a copy so that you can
send them in later.  :)

> also, where can I find an example of how to compile the rulebase, and an 
> example program that uses the rulebase?

	Check out "normalizer.c" with the liblognorm git.  It's a basic
program that does normalization. 

> 
> David Lang
> 
> On Mon, 17 Jan 2011, david at lang.hm wrote:
> 
> > documentation of how to get started is not easy to find (if it exists yet)
> >
> > I found how to download via git
> >
> > git clone git://git.adiscon.com/git/liblognorm liblognorm
> >
> > and then did autoreconf -fvi followed by ./configure but configure died 
> > because it needs libestr
> >
> > this seems to be available via git
> >
> > git clone git://git.adiscon.com/git/libestr libestr
> >
> > then did autoreconf -fvi followed by ./configure ; make ; make install
> >
> > I then found that libee is needed
> >
> >  git clone git://git.adiscon.com/git/libee libee
> >
> > autoreconf -fvi; ./configure ; make ; make install
> >
> > this finally let the ./configure; make ; make install work for liblognorm
> >
> >
> > now to hunt for how to use it ;-)
> >
> > David Lang
> >
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110117/cbdc74ca/attachment.pgp>

From david at lang.hm  Tue Jan 18 00:26:29 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 17 Jan 2011 15:26:29 -0800 (PST)
Subject: [Lognorm] getting started document
In-Reply-To: <20110117225313.GA16101@bundy.vistech.net>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>
	<alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm>
	<20110117225313.GA16101@bundy.vistech.net>
Message-ID: <alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm>

From: "Champ Clark III [Softwink]" <champ at softwink.com>

>On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>> the format of the rulebase files seems pretty straightforward, but 
>> where can I find the list of what datatypes are supported?
>
>        Keep in mind that liblognorm is pretty young at this point.
>So more documentation probably needs to be done.

that's what I figured, and why I posted what i found needed to be done 
rather than just griping about it :-)

>Also, there's not much in the way of rulebase files yet, but that will 
>likely change in the future.  Right now, you're probably going to have to 
>write your own rulebase files.

I expected to have to write my own rules, but it's hard to write rules 
without knowing what datatypes are available.

>When you do that, keep a copy so that you 
>can send them in later.  :)

I expect to send a bundch in as I work on things, although I also expect 
to be using the parsing library in ways that don't directly tie in to 
the 'big normalization' project.

For example, Rainer wondered out loud about the possibility of using this 
library for a rsyslog log parser instead of writing the parsers directly 
in C. in that case, most of what you want in terms of normalization won't 
matter, as that parser is just focused on figuring out what part of the 
blob that arrives is the message, and what metadata in included, not in 
parsing the message itself into different pieces.

>> also, where can I find an example of how to compile the rulebase, and 
>> an example program that uses the rulebase?
>
>        Check out "normalizer.c" with the liblognorm git.  It's a basic 
> program that does normalization.

will do.

can you point me at what file (or files) the datatypes are defined in?

David Lang
-------------- next part --------------
On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> the format of the rulebase files seems pretty straightforward, but where 
> can I find the list of what datatypes are supported?

	Keep in mind that liblognorm is pretty young at this point. 
So more documentation probably needs to be done.  Also,  there's not
much in the way of rulebase files yet,  but that will likely change
in the future.  Right now,  you're probably going to have to write your
own rulebase files.  When you do that,  keep a copy so that you can
send them in later.  :)

> also, where can I find an example of how to compile the rulebase, and an 
> example program that uses the rulebase?

	Check out "normalizer.c" with the liblognorm git.  It's a basic
program that does normalization. 

> 
> David Lang
> 
> On Mon, 17 Jan 2011, david at lang.hm wrote:
> 
> > documentation of how to get started is not easy to find (if it exists yet)
> >
> > I found how to download via git
> >
> > git clone git://git.adiscon.com/git/liblognorm liblognorm
> >
> > and then did autoreconf -fvi followed by ./configure but configure died 
> > because it needs libestr
> >
> > this seems to be available via git
> >
> > git clone git://git.adiscon.com/git/libestr libestr
> >
> > then did autoreconf -fvi followed by ./configure ; make ; make install
> >
> > I then found that libee is needed
> >
> >  git clone git://git.adiscon.com/git/libee libee
> >
> > autoreconf -fvi; ./configure ; make ; make install
> >
> > this finally let the ./configure; make ; make install work for liblognorm
> >
> >
> > now to hunt for how to use it ;-)
> >
> > David Lang
> >
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110117/d04c0641/attachment-0001.pgp>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From rgerhards at hq.adiscon.com  Tue Jan 18 08:04:26 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Jan 2011 08:04:26 +0100
Subject: [Lognorm] getting started document
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net>
	<alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>

The web site is currently being built (any help on content, design, etc is
deeply appreciated). It is available at

http://www.liblognorm.com

In the menu there is a link to the current *very sparse* documentation.

On data types: there is not yet really such a thing as a "data type" -- all
are strings. Liblognorm uses a different concept right now, that is
"parsers". A parser actually describes a syntax that a string must fullfil in
order to be treated as "correct". There are a number of parsers, which should
be in the doc (but I see Florian did still not include the new ones, hope he
will do soon...). It is these types that the rules are build around.

I have recently asked which additional "types" (parsers) are considered
useful and I am ready to add new ones (I am sure there must be more than we
currently have).

We also need standard field names. I have started an effort on what to use
and Florian started a sample directory. We are looking for feedback here.

Finally, but very importantly, I'd like to add a section with rule bases to
the web site, where device-specifc rules can be found ... and contributed.
Suggestion on how to do this best would be very appreciated.

I'll also see if I can write a small getting started document. I guess it
should focus on the technical issues. Right now, I am doing a lot of writing,
so this should be possible as a side-activity (though I would prefer to code
a little me, but... ;)).

Hope that helps at least a little bit.

Rainer

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, January 18, 2011 12:26 AM
> To: lognorm
> Subject: Re: [Lognorm] getting started document
> 
> From: "Champ Clark III [Softwink]" <champ at softwink.com>
> 
> >On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> >> the format of the rulebase files seems pretty straightforward, but
> >> where can I find the list of what datatypes are supported?
> >
> >        Keep in mind that liblognorm is pretty young at this point.
> >So more documentation probably needs to be done.
> 
> that's what I figured, and why I posted what i found needed to be done
> rather than just griping about it :-)
> 
> >Also, there's not much in the way of rulebase files yet, but that will
> >likely change in the future.  Right now, you're probably going to have
> to
> >write your own rulebase files.
> 
> I expected to have to write my own rules, but it's hard to write rules
> without knowing what datatypes are available.
> 
> >When you do that, keep a copy so that you
> >can send them in later.  :)
> 
> I expect to send a bundch in as I work on things, although I also
> expect
> to be using the parsing library in ways that don't directly tie in to
> the 'big normalization' project.
> 
> For example, Rainer wondered out loud about the possibility of using
> this
> library for a rsyslog log parser instead of writing the parsers
> directly
> in C. in that case, most of what you want in terms of normalization
> won't
> matter, as that parser is just focused on figuring out what part of the
> blob that arrives is the message, and what metadata in included, not in
> parsing the message itself into different pieces.
> 
> >> also, where can I find an example of how to compile the rulebase,
> and
> >> an example program that uses the rulebase?
> >
> >        Check out "normalizer.c" with the liblognorm git.  It's a
> basic
> > program that does normalization.
> 
> will do.
> 
> can you point me at what file (or files) the datatypes are defined in?
> 
> David Lang

From david at lang.hm  Tue Jan 18 09:39:08 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 18 Jan 2011 00:39:08 -0800 (PST)
Subject: [Lognorm] getting started document
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net>
	<alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>

thanks.

a couple things that jump out at me as being useful

for numbers, be able to specify interger or decimal (sometimes numbers are 
##.## where they are two independant fields with the decimal as the 
separator

besides the obvious IPv6 there is a need for predefined options

such as anyIP = (IPv4 || IPv6)

or host = (hostname || anyIP)

there are also times when you want to specify multiple words as a single 
field (i.e. next 2 words as field named X)

wildcard to end of line

with word matches, define allowed characters (hostname can be 0-9A-Za-z-_. 
but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)

some way for the parser to report that the ruleset is ambiguous (two rules 
match the same line)

can you change the prefix during the ruleset?


these things will matter more as useage grows (it will become harder and 
harder to specify the difference between things)



I see two forks to this project

fork 1, the parser library (keeping it fast while letting it specify all 
the nuances needed to match everything without ambiguity.


fork 2. the rulesets for common log messages, defining them with common 
termenology.


I can see other projects using the parser, without using the rulesets 
(the rsyslog parser module for example, it's mostly having to define all 
the variations of possible message formats to just figure out how to 
reformat it to a 'standard' syslog format for the other rules to handle)


David Lang


On Tue, 18 Jan 2011, Rainer Gerhards wrote:

> Date: Tue, 18 Jan 2011 08:04:26 +0100
> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> Reply-To: lognorm <lognorm at lists.adiscon.com>
> To: lognorm <lognorm at lists.adiscon.com>
> Subject: Re: [Lognorm] getting started document
> 
> The web site is currently being built (any help on content, design, etc is
> deeply appreciated). It is available at
>
> http://www.liblognorm.com
>
> In the menu there is a link to the current *very sparse* documentation.
>
> On data types: there is not yet really such a thing as a "data type" -- all
> are strings. Liblognorm uses a different concept right now, that is
> "parsers". A parser actually describes a syntax that a string must fullfil in
> order to be treated as "correct". There are a number of parsers, which should
> be in the doc (but I see Florian did still not include the new ones, hope he
> will do soon...). It is these types that the rules are build around.
>
> I have recently asked which additional "types" (parsers) are considered
> useful and I am ready to add new ones (I am sure there must be more than we
> currently have).
>
> We also need standard field names. I have started an effort on what to use
> and Florian started a sample directory. We are looking for feedback here.
>
> Finally, but very importantly, I'd like to add a section with rule bases to
> the web site, where device-specifc rules can be found ... and contributed.
> Suggestion on how to do this best would be very appreciated.
>
> I'll also see if I can write a small getting started document. I guess it
> should focus on the technical issues. Right now, I am doing a lot of writing,
> so this should be possible as a side-activity (though I would prefer to code
> a little me, but... ;)).
>
> Hope that helps at least a little bit.
>
> Rainer
>
>> -----Original Message-----
>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, January 18, 2011 12:26 AM
>> To: lognorm
>> Subject: Re: [Lognorm] getting started document
>>
>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
>>
>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>>>> the format of the rulebase files seems pretty straightforward, but
>>>> where can I find the list of what datatypes are supported?
>>>
>>>        Keep in mind that liblognorm is pretty young at this point.
>>> So more documentation probably needs to be done.
>>
>> that's what I figured, and why I posted what i found needed to be done
>> rather than just griping about it :-)
>>
>>> Also, there's not much in the way of rulebase files yet, but that will
>>> likely change in the future.  Right now, you're probably going to have
>> to
>>> write your own rulebase files.
>>
>> I expected to have to write my own rules, but it's hard to write rules
>> without knowing what datatypes are available.
>>
>>> When you do that, keep a copy so that you
>>> can send them in later.  :)
>>
>> I expect to send a bundch in as I work on things, although I also
>> expect
>> to be using the parsing library in ways that don't directly tie in to
>> the 'big normalization' project.
>>
>> For example, Rainer wondered out loud about the possibility of using
>> this
>> library for a rsyslog log parser instead of writing the parsers
>> directly
>> in C. in that case, most of what you want in terms of normalization
>> won't
>> matter, as that parser is just focused on figuring out what part of the
>> blob that arrives is the message, and what metadata in included, not in
>> parsing the message itself into different pieces.
>>
>>>> also, where can I find an example of how to compile the rulebase,
>> and
>>>> an example program that uses the rulebase?
>>>
>>>        Check out "normalizer.c" with the liblognorm git.  It's a
>> basic
>>> program that does normalization.
>>
>> will do.
>>
>> can you point me at what file (or files) the datatypes are defined in?
>>
>> David Lang
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>

From rgerhards at hq.adiscon.com  Tue Jan 18 09:43:14 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Jan 2011 09:43:14 +0100
Subject: [Lognorm] getting started document
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net><alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA1F@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, January 18, 2011 9:39 AM
> To: lognorm
> Subject: Re: [Lognorm] getting started document
> 
> thanks.
> 
> a couple things that jump out at me as being useful
> 
> for numbers, be able to specify interger or decimal (sometimes numbers
> are
> ##.## where they are two independant fields with the decimal as the
> separator
> 
> besides the obvious IPv6 there is a need for predefined options
> 
> such as anyIP = (IPv4 || IPv6)
> 
> or host = (hostname || anyIP)
> 
> there are also times when you want to specify multiple words as a
> single
> field (i.e. next 2 words as field named X)
> 
> wildcard to end of line
> 
> with word matches, define allowed characters (hostname can be 0-9A-Za-
> z-_.
> but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
> 
> some way for the parser to report that the ruleset is ambiguous (two
> rules
> match the same line)

That's an error, in this case the rulebase does not load.

> 
> can you change the prefix during the ruleset?

yes

> 
> 
> these things will matter more as useage grows (it will become harder
> and
> harder to specify the difference between things)
> 
> 
> 
> I see two forks to this project
> 
> fork 1, the parser library (keeping it fast while letting it specify
> all
> the nuances needed to match everything without ambiguity.
> 
> 
> fork 2. the rulesets for common log messages, defining them with common
> termenology.
> 
> 
> I can see other projects using the parser, without using the rulesets
> (the rsyslog parser module for example, it's mostly having to define
> all
> the variations of possible message formats to just figure out how to
> reformat it to a 'standard' syslog format for the other rules to
> handle)

actually, liblognorm is what you describe as fork 1. The "fork 2" project is
what I try to achieve via a website. My goal here is to have the fork 2
things available for everyone who wants. For example, in rsyslog all I expect
you need is a different prefix, but otherwise the rulebases should be usable.

Rainer
> 
> 
> David Lang
> 
> 
> On Tue, 18 Jan 2011, Rainer Gerhards wrote:
> 
> > Date: Tue, 18 Jan 2011 08:04:26 +0100
> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> > Reply-To: lognorm <lognorm at lists.adiscon.com>
> > To: lognorm <lognorm at lists.adiscon.com>
> > Subject: Re: [Lognorm] getting started document
> >
> > The web site is currently being built (any help on content, design,
> etc is
> > deeply appreciated). It is available at
> >
> > http://www.liblognorm.com
> >
> > In the menu there is a link to the current *very sparse*
> documentation.
> >
> > On data types: there is not yet really such a thing as a "data type"
> -- all
> > are strings. Liblognorm uses a different concept right now, that is
> > "parsers". A parser actually describes a syntax that a string must
> fullfil in
> > order to be treated as "correct". There are a number of parsers,
> which should
> > be in the doc (but I see Florian did still not include the new ones,
> hope he
> > will do soon...). It is these types that the rules are build around.
> >
> > I have recently asked which additional "types" (parsers) are
> considered
> > useful and I am ready to add new ones (I am sure there must be more
> than we
> > currently have).
> >
> > We also need standard field names. I have started an effort on what
> to use
> > and Florian started a sample directory. We are looking for feedback
> here.
> >
> > Finally, but very importantly, I'd like to add a section with rule
> bases to
> > the web site, where device-specifc rules can be found ... and
> contributed.
> > Suggestion on how to do this best would be very appreciated.
> >
> > I'll also see if I can write a small getting started document. I
> guess it
> > should focus on the technical issues. Right now, I am doing a lot of
> writing,
> > so this should be possible as a side-activity (though I would prefer
> to code
> > a little me, but... ;)).
> >
> > Hope that helps at least a little bit.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Tuesday, January 18, 2011 12:26 AM
> >> To: lognorm
> >> Subject: Re: [Lognorm] getting started document
> >>
> >> From: "Champ Clark III [Softwink]" <champ at softwink.com>
> >>
> >>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> >>>> the format of the rulebase files seems pretty straightforward, but
> >>>> where can I find the list of what datatypes are supported?
> >>>
> >>>        Keep in mind that liblognorm is pretty young at this point.
> >>> So more documentation probably needs to be done.
> >>
> >> that's what I figured, and why I posted what i found needed to be
> done
> >> rather than just griping about it :-)
> >>
> >>> Also, there's not much in the way of rulebase files yet, but that
> will
> >>> likely change in the future.  Right now, you're probably going to
> have
> >> to
> >>> write your own rulebase files.
> >>
> >> I expected to have to write my own rules, but it's hard to write
> rules
> >> without knowing what datatypes are available.
> >>
> >>> When you do that, keep a copy so that you
> >>> can send them in later.  :)
> >>
> >> I expect to send a bundch in as I work on things, although I also
> >> expect
> >> to be using the parsing library in ways that don't directly tie in
> to
> >> the 'big normalization' project.
> >>
> >> For example, Rainer wondered out loud about the possibility of using
> >> this
> >> library for a rsyslog log parser instead of writing the parsers
> >> directly
> >> in C. in that case, most of what you want in terms of normalization
> >> won't
> >> matter, as that parser is just focused on figuring out what part of
> the
> >> blob that arrives is the message, and what metadata in included, not
> in
> >> parsing the message itself into different pieces.
> >>
> >>>> also, where can I find an example of how to compile the rulebase,
> >> and
> >>>> an example program that uses the rulebase?
> >>>
> >>>        Check out "normalizer.c" with the liblognorm git.  It's a
> >> basic
> >>> program that does normalization.
> >>
> >> will do.
> >>
> >> can you point me at what file (or files) the datatypes are defined
> in?
> >>
> >> David Lang
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

From champ at softwink.com  Tue Jan 18 18:34:14 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 12:34:14 -0500
Subject: [Lognorm] type:char-to: issues (?)
Message-ID: <20110118173414.GA10582@bundy.vistech.net>


	I might have two issues coming up,  but let me explain
them one at a time:  First (this is for DNS),  here's my input from
the syslog message:

 unexpected RCODE (REFUSED) resolving '1.0.168.192.in-addr.arpa/PTR/IN': 10.20.1.1#53

Here's my rule: 

rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-to:\x27%': %src-ip%ipv4#%src-port:number%

Here's my normalized output:

Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL) resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-data=".197.215.181#53"]

Not the "unparsed-data" (of course).  I'd expect "type" to become
"example.com/MX/IN".  However, it's 'eating' through the first 
octet of the src-ip.   

While I was pretty sure it wouldn't work, I tried the new 
quoted-string,  but as I expected that didn't work.  I'm assuming
quoted-string only handles "'s and not 's ?  That might be a 'feature'
to add to quoted-string as well.

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/08469266/attachment.pgp>

From rgerhards at hq.adiscon.com  Tue Jan 18 18:37:41 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Jan 2011 18:37:41 +0100
Subject: [Lognorm] type:char-to: issues (?)
References: <20110118173414.GA10582@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>

It looks like you have mixed up you sample :) Please check. Will try to look
at it later today (but it is already close to 7p over here), else tomorrow
morning.

Rainer

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Tuesday, January 18, 2011 6:34 PM
> To: lognorm at lists.adiscon.com
> Subject: [Lognorm] type:char-to: issues (?)
> 
> 
> 	I might have two issues coming up,  but let me explain
> them one at a time:  First (this is for DNS),  here's my input from
> the syslog message:
> 
>  unexpected RCODE (REFUSED) resolving '1.0.168.192.in-
> addr.arpa/PTR/IN': 10.20.1.1#53
> 
> Here's my rule:
> 
> rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-to:\x27%':
> %src-ip%ipv4#%src-port:number%
> 
> Here's my normalized output:
> 
> Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL)
> resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-
> data=".197.215.181#53"]
> 
> Not the "unparsed-data" (of course).  I'd expect "type" to become
> "example.com/MX/IN".  However, it's 'eating' through the first
> octet of the src-ip.
> 
> While I was pretty sure it wouldn't work, I tried the new
> quoted-string,  but as I expected that didn't work.  I'm assuming
> quoted-string only handles "'s and not 's ?  That might be a 'feature'
> to add to quoted-string as well.
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Tue Jan 18 18:51:22 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 12:51:22 -0500
Subject: [Lognorm] type:char-to: issues (?)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
References: <20110118173414.GA10582@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
Message-ID: <20110118175122.GA11661@bundy.vistech.net>

On Tue, Jan 18, 2011 at 06:37:41PM +0100, Rainer Gerhards wrote:
> It looks like you have mixed up you sample :) Please check. Will try to look
> at it later today (but it is already close to 7p over here), else tomorrow
> morning.

	Yes,  I did.  It was actually two rules (one for REFUSED and other
for SERVFAIL).  Anyways,  I've revised it down to one rule.  Here's
the more accurate information:

rule=: unexpected RCODE %failcode:word% resolving '%type:char-to:\x27%': %src-ip%ipv4#%src-port:number%

Syslog input:

 unexpected RCODE (SERVFAIL) resolving '161.41.46.64.in-addr.arpa/PTR/IN': 209.97.206.2#53

Normalized Output:

Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL) resolving '161.41.46.64.in-addr.arpa/PTR/IN': 192.197.212.68#53" unparsed-data=".197.212.68#53"]

	This might be user error somewhere.  On thing I have to tinker with
are things like "(%failcode:word%)" verses "%failcode:word%",  as spaces
seem to be the delimiter in the end.... Correct?


> 
> Rainer
> 
> > -----Original Message-----
> > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > Sent: Tuesday, January 18, 2011 6:34 PM
> > To: lognorm at lists.adiscon.com
> > Subject: [Lognorm] type:char-to: issues (?)
> > 
> > 
> > 	I might have two issues coming up,  but let me explain
> > them one at a time:  First (this is for DNS),  here's my input from
> > the syslog message:
> > 
> >  unexpected RCODE (REFUSED) resolving '1.0.168.192.in-
> > addr.arpa/PTR/IN': 10.20.1.1#53
> > 
> > Here's my rule:
> > 
> > rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-to:\x27%':
> > %src-ip%ipv4#%src-port:number%
> > 
> > Here's my normalized output:
> > 
> > Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL)
> > resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-
> > data=".197.215.181#53"]
> > 
> > Not the "unparsed-data" (of course).  I'd expect "type" to become
> > "example.com/MX/IN".  However, it's 'eating' through the first
> > octet of the src-ip.
> > 
> > While I was pretty sure it wouldn't work, I tried the new
> > quoted-string,  but as I expected that didn't work.  I'm assuming
> > quoted-string only handles "'s and not 's ?  That might be a 'feature'
> > to add to quoted-string as well.
> > 
> > --
> >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> >                      http://www.softwink.com
> > 
> > GPG Key ID: 58A2A58F
> > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/ebe7f356/attachment.pgp>

From rgerhards at hq.adiscon.com  Tue Jan 18 18:54:09 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Jan 2011 18:54:09 +0100
Subject: [Lognorm] type:char-to: issues (?)
References: <20110118173414.GA10582@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
	<20110118175122.GA11661@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>

I guess this is interference from some other rule. Can you try to use this
rule alone? 

I think I should add the information which rule was meet, at least in debug
mode...

Rainer

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Tuesday, January 18, 2011 6:51 PM
> To: lognorm
> Subject: Re: [Lognorm] type:char-to: issues (?)
> 
> On Tue, Jan 18, 2011 at 06:37:41PM +0100, Rainer Gerhards wrote:
> > It looks like you have mixed up you sample :) Please check. Will try
> to look
> > at it later today (but it is already close to 7p over here), else
> tomorrow
> > morning.
> 
> 	Yes,  I did.  It was actually two rules (one for REFUSED and
> other
> for SERVFAIL).  Anyways,  I've revised it down to one rule.  Here's
> the more accurate information:
> 
> rule=: unexpected RCODE %failcode:word% resolving '%type:char-
> to:\x27%': %src-ip%ipv4#%src-port:number%
> 
> Syslog input:
> 
>  unexpected RCODE (SERVFAIL) resolving '161.41.46.64.in-
> addr.arpa/PTR/IN': 209.97.206.2#53
> 
> Normalized Output:
> 
> Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL)
> resolving '161.41.46.64.in-addr.arpa/PTR/IN': 192.197.212.68#53"
> unparsed-data=".197.212.68#53"]
> 
> 	This might be user error somewhere.  On thing I have to tinker
> with
> are things like "(%failcode:word%)" verses "%failcode:word%",  as
> spaces
> seem to be the delimiter in the end.... Correct?
> 
> 
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > > Sent: Tuesday, January 18, 2011 6:34 PM
> > > To: lognorm at lists.adiscon.com
> > > Subject: [Lognorm] type:char-to: issues (?)
> > >
> > >
> > > 	I might have two issues coming up,  but let me explain
> > > them one at a time:  First (this is for DNS),  here's my input from
> > > the syslog message:
> > >
> > >  unexpected RCODE (REFUSED) resolving '1.0.168.192.in-
> > > addr.arpa/PTR/IN': 10.20.1.1#53
> > >
> > > Here's my rule:
> > >
> > > rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-to:\x27%':
> > > %src-ip%ipv4#%src-port:number%
> > >
> > > Here's my normalized output:
> > >
> > > Normalize output: [cee at 115 originalmsg=" unexpected RCODE
> (SERVFAIL)
> > > resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-
> > > data=".197.215.181#53"]
> > >
> > > Not the "unparsed-data" (of course).  I'd expect "type" to become
> > > "example.com/MX/IN".  However, it's 'eating' through the first
> > > octet of the src-ip.
> > >
> > > While I was pretty sure it wouldn't work, I tried the new
> > > quoted-string,  but as I expected that didn't work.  I'm assuming
> > > quoted-string only handles "'s and not 's ?  That might be a
> 'feature'
> > > to add to quoted-string as well.
> > >
> > > --
> > >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> > >                      http://www.softwink.com
> > >
> > > GPG Key ID: 58A2A58F
> > > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2
> A58F
> > > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Tue Jan 18 18:57:49 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 12:57:49 -0500
Subject: [Lognorm] type:char-to: issues (?)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
References: <20110118173414.GA10582@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
	<20110118175122.GA11661@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
Message-ID: <20110118175749.GA12362@bundy.vistech.net>

On Tue, Jan 18, 2011 at 06:54:09PM +0100, Rainer Gerhards wrote:
> I guess this is interference from some other rule. Can you try to use this
> rule alone? 

	Yep.. 

	Doing that now...

> 
> I think I should add the information which rule was meet, at least in debug
> mode...
> 
> Rainer
> 
> > -----Original Message-----
> > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > Sent: Tuesday, January 18, 2011 6:51 PM
> > To: lognorm
> > Subject: Re: [Lognorm] type:char-to: issues (?)
> > 
> > On Tue, Jan 18, 2011 at 06:37:41PM +0100, Rainer Gerhards wrote:
> > > It looks like you have mixed up you sample :) Please check. Will try
> > to look
> > > at it later today (but it is already close to 7p over here), else
> > tomorrow
> > > morning.
> > 
> > 	Yes,  I did.  It was actually two rules (one for REFUSED and
> > other
> > for SERVFAIL).  Anyways,  I've revised it down to one rule.  Here's
> > the more accurate information:
> > 
> > rule=: unexpected RCODE %failcode:word% resolving '%type:char-
> > to:\x27%': %src-ip%ipv4#%src-port:number%
> > 
> > Syslog input:
> > 
> >  unexpected RCODE (SERVFAIL) resolving '161.41.46.64.in-
> > addr.arpa/PTR/IN': 209.97.206.2#53
> > 
> > Normalized Output:
> > 
> > Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL)
> > resolving '161.41.46.64.in-addr.arpa/PTR/IN': 192.197.212.68#53"
> > unparsed-data=".197.212.68#53"]
> > 
> > 	This might be user error somewhere.  On thing I have to tinker
> > with
> > are things like "(%failcode:word%)" verses "%failcode:word%",  as
> > spaces
> > seem to be the delimiter in the end.... Correct?
> > 
> > 
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > > > Sent: Tuesday, January 18, 2011 6:34 PM
> > > > To: lognorm at lists.adiscon.com
> > > > Subject: [Lognorm] type:char-to: issues (?)
> > > >
> > > >
> > > > 	I might have two issues coming up,  but let me explain
> > > > them one at a time:  First (this is for DNS),  here's my input from
> > > > the syslog message:
> > > >
> > > >  unexpected RCODE (REFUSED) resolving '1.0.168.192.in-
> > > > addr.arpa/PTR/IN': 10.20.1.1#53
> > > >
> > > > Here's my rule:
> > > >
> > > > rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-to:\x27%':
> > > > %src-ip%ipv4#%src-port:number%
> > > >
> > > > Here's my normalized output:
> > > >
> > > > Normalize output: [cee at 115 originalmsg=" unexpected RCODE
> > (SERVFAIL)
> > > > resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-
> > > > data=".197.215.181#53"]
> > > >
> > > > Not the "unparsed-data" (of course).  I'd expect "type" to become
> > > > "example.com/MX/IN".  However, it's 'eating' through the first
> > > > octet of the src-ip.
> > > >
> > > > While I was pretty sure it wouldn't work, I tried the new
> > > > quoted-string,  but as I expected that didn't work.  I'm assuming
> > > > quoted-string only handles "'s and not 's ?  That might be a
> > 'feature'
> > > > to add to quoted-string as well.
> > > >
> > > > --
> > > >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> > > >                      http://www.softwink.com
> > > >
> > > > GPG Key ID: 58A2A58F
> > > > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2
> > A58F
> > > > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> > > _______________________________________________
> > > Lognorm mailing list
> > > Lognorm at lists.adiscon.com
> > > http://lists.adiscon.net/mailman/listinfo/lognorm
> > 
> > --
> >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> >                      http://www.softwink.com
> > 
> > GPG Key ID: 58A2A58F
> > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/ef6ac946/attachment.pgp>

From rgerhards at hq.adiscon.com  Tue Jan 18 18:59:28 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 18 Jan 2011 18:59:28 +0100
Subject: [Lognorm] type:char-to: issues (?)
References: <20110118173414.GA10582@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com><20110118175122.GA11661@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
	<20110118175749.GA12362@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA3C@GRFEXC.intern.adiscon.com>

Mhhhh... on the other hand it is "unparsed" -- that can not be interference.
I Think I'll run it through the normalizer myself tomorrow morning...

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Tuesday, January 18, 2011 6:58 PM
> To: lognorm
> Subject: Re: [Lognorm] type:char-to: issues (?)
> 
> On Tue, Jan 18, 2011 at 06:54:09PM +0100, Rainer Gerhards wrote:
> > I guess this is interference from some other rule. Can you try to use
> this
> > rule alone?
> 
> 	Yep..
> 
> 	Doing that now...
> 
> >
> > I think I should add the information which rule was meet, at least in
> debug
> > mode...
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > > Sent: Tuesday, January 18, 2011 6:51 PM
> > > To: lognorm
> > > Subject: Re: [Lognorm] type:char-to: issues (?)
> > >
> > > On Tue, Jan 18, 2011 at 06:37:41PM +0100, Rainer Gerhards wrote:
> > > > It looks like you have mixed up you sample :) Please check. Will
> try
> > > to look
> > > > at it later today (but it is already close to 7p over here), else
> > > tomorrow
> > > > morning.
> > >
> > > 	Yes,  I did.  It was actually two rules (one for REFUSED and
> > > other
> > > for SERVFAIL).  Anyways,  I've revised it down to one rule.  Here's
> > > the more accurate information:
> > >
> > > rule=: unexpected RCODE %failcode:word% resolving '%type:char-
> > > to:\x27%': %src-ip%ipv4#%src-port:number%
> > >
> > > Syslog input:
> > >
> > >  unexpected RCODE (SERVFAIL) resolving '161.41.46.64.in-
> > > addr.arpa/PTR/IN': 209.97.206.2#53
> > >
> > > Normalized Output:
> > >
> > > Normalize output: [cee at 115 originalmsg=" unexpected RCODE
> (SERVFAIL)
> > > resolving '161.41.46.64.in-addr.arpa/PTR/IN': 192.197.212.68#53"
> > > unparsed-data=".197.212.68#53"]
> > >
> > > 	This might be user error somewhere.  On thing I have to tinker
> > > with
> > > are things like "(%failcode:word%)" verses "%failcode:word%",  as
> > > spaces
> > > seem to be the delimiter in the end.... Correct?
> > >
> > >
> > > >
> > > > Rainer
> > > >
> > > > > -----Original Message-----
> > > > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > > > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III
> [Softwink]
> > > > > Sent: Tuesday, January 18, 2011 6:34 PM
> > > > > To: lognorm at lists.adiscon.com
> > > > > Subject: [Lognorm] type:char-to: issues (?)
> > > > >
> > > > >
> > > > > 	I might have two issues coming up,  but let me explain
> > > > > them one at a time:  First (this is for DNS),  here's my input
> from
> > > > > the syslog message:
> > > > >
> > > > >  unexpected RCODE (REFUSED) resolving '1.0.168.192.in-
> > > > > addr.arpa/PTR/IN': 10.20.1.1#53
> > > > >
> > > > > Here's my rule:
> > > > >
> > > > > rule=: unexpected RCODE (SERVFAIL) resolving '%type:char-
> to:\x27%':
> > > > > %src-ip%ipv4#%src-port:number%
> > > > >
> > > > > Here's my normalized output:
> > > > >
> > > > > Normalize output: [cee at 115 originalmsg=" unexpected RCODE
> > > (SERVFAIL)
> > > > > resolving 'example.com/MX/IN': 66.197.215.181#53" unparsed-
> > > > > data=".197.215.181#53"]
> > > > >
> > > > > Not the "unparsed-data" (of course).  I'd expect "type" to
> become
> > > > > "example.com/MX/IN".  However, it's 'eating' through the first
> > > > > octet of the src-ip.
> > > > >
> > > > > While I was pretty sure it wouldn't work, I tried the new
> > > > > quoted-string,  but as I expected that didn't work.  I'm
> assuming
> > > > > quoted-string only handles "'s and not 's ?  That might be a
> > > 'feature'
> > > > > to add to quoted-string as well.
> > > > >
> > > > > --
> > > > >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> > > > >                      http://www.softwink.com
> > > > >
> > > > > GPG Key ID: 58A2A58F
> > > > > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2
> > > A58F
> > > > > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> > > > _______________________________________________
> > > > Lognorm mailing list
> > > > Lognorm at lists.adiscon.com
> > > > http://lists.adiscon.net/mailman/listinfo/lognorm
> > >
> > > --
> > >         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> > >                      http://www.softwink.com
> > >
> > > GPG Key ID: 58A2A58F
> > > Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2
> A58F
> > > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Tue Jan 18 19:02:32 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 13:02:32 -0500
Subject: [Lognorm] type:char-to: issues (?)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA3C@GRFEXC.intern.adiscon.com>
References: <20110118173414.GA10582@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
	<20110118175122.GA11661@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
	<20110118175749.GA12362@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3C@GRFEXC.intern.adiscon.com>
Message-ID: <20110118180232.GB12362@bundy.vistech.net>

On Tue, Jan 18, 2011 at 06:59:28PM +0100, Rainer Gerhards wrote:
> Mhhhh... on the other hand it is "unparsed" -- that can not be interference.
> I Think I'll run it through the normalizer myself tomorrow morning...

	Yep.... 

rule=: rule=: unexpected RCODE %failcode:word% resolving '%type:char-to:\x27%': %src-ip%ipv4#%src-port:number%

Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL) resolving 'rbldns2.sorbs.net/A/IN': 174.36.235.174#53" unparsed-data=".36.235.174#53"]

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/6f33ac0b/attachment-0001.pgp>

From champ at softwink.com  Tue Jan 18 20:14:21 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 14:14:21 -0500
Subject: [Lognorm] type:char-to: issues (?)
In-Reply-To: <20110118180232.GB12362@bundy.vistech.net>
References: <20110118173414.GA10582@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
	<20110118175122.GA11661@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
	<20110118175749.GA12362@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3C@GRFEXC.intern.adiscon.com>
	<20110118180232.GB12362@bundy.vistech.net>
Message-ID: <20110118191421.GA17061@bundy.vistech.net>

On Tue, Jan 18, 2011 at 01:02:32PM -0500, Champ Clark III [Softwink] wrote:
> On Tue, Jan 18, 2011 at 06:59:28PM +0100, Rainer Gerhards wrote:
> > Mhhhh... on the other hand it is "unparsed" -- that can not be interference.
> > I Think I'll run it through the normalizer myself tomorrow morning...
> 
> 	Yep.... 
> 
> rule=: rule=: unexpected RCODE %failcode:word% resolving '%type:char-to:\x27%': %src-ip%ipv4#%src-port:number%
> 
> Normalize output: [cee at 115 originalmsg=" unexpected RCODE (SERVFAIL) resolving 'rbldns2.sorbs.net/A/IN': 174.36.235.174#53" unparsed-data=".36.235.174#53"]

	Sorry,  typo: It's really.

rule=: unexpected RCODE %failcode:word% resolving '%type:char-to:\x27%': %src-ip%ipv4#%src-port:number%



-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/73505264/attachment.pgp>

From champ at softwink.com  Tue Jan 18 22:22:05 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 16:22:05 -0500
Subject: [Lognorm] type:char-to: issues (?)
In-Reply-To: <20110118191421.GA17061@bundy.vistech.net>
References: <20110118173414.GA10582@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA39@GRFEXC.intern.adiscon.com>
	<20110118175122.GA11661@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3A@GRFEXC.intern.adiscon.com>
	<20110118175749.GA12362@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3C@GRFEXC.intern.adiscon.com>
	<20110118180232.GB12362@bundy.vistech.net>
	<20110118191421.GA17061@bundy.vistech.net>
Message-ID: <20110118212205.GA24507@bundy.vistech.net>


	Argh.. Strike that.  It helps if the rule is properly formated: 

rule=: unexpected RCODE %failcode:word% resolving '%type:char-to:\x27%': %src-ip:ipv4%#%src-port:number%

backup src # cat input  | ./normalizer -r ./rule
[cee at 115 src-port="53" src-ip="192.197.212.68" type="161.41.46.64.in-addr.arpa/PTR/IN" failcode="(SERVFAIL)"]

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/d07268e8/attachment.pgp>

From champ at softwink.com  Wed Jan 19 00:29:08 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 18:29:08 -0500
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
Message-ID: <20110118232908.GA30902@bundy.vistech.net>


	How does liblognorm deal with something like this:

 1w3d: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 192.168.0.1

 	I'm wondering if the %SNMP-3-AUTHFAIL will cause problems.  For
example,  this rule:

rule=: %uptime:word% %authfail:word% Authentication failure for SNMP req from host %src-ip:ipv4%

	Works fine.  This one:

rule=: %uptime:word% %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host %src-ip:ipv4%

	Does not.  

Normalize output: [cee at 115 originalmsg=" 1w3d: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 66.177.167.194" unparsed-data="Authentication failure for SNMP req from host 66.177.167.194"]

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/cdd1e73e/attachment.pgp>

From david at lang.hm  Wed Jan 19 00:47:00 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 18 Jan 2011 15:47:00 -0800 (PST)
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
In-Reply-To: <20110118232908.GA30902@bundy.vistech.net>
References: <20110118232908.GA30902@bundy.vistech.net>
Message-ID: <alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>

you would need to escape the % in front of %SNMP, that's probably throwing 
off the parser.

I don't know how you would actually _do_ that though.

David Lang
-------------- next part --------------

	How does liblognorm deal with something like this:

 1w3d: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 192.168.0.1

 	I'm wondering if the %SNMP-3-AUTHFAIL will cause problems.  For
example,  this rule:

rule=: %uptime:word% %authfail:word% Authentication failure for SNMP req from host %src-ip:ipv4%

	Works fine.  This one:

rule=: %uptime:word% %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host %src-ip:ipv4%

	Does not.  

Normalize output: [cee at 115 originalmsg=" 1w3d: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 66.177.167.194" unparsed-data="Authentication failure for SNMP req from host 66.177.167.194"]

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/db75ef2e/attachment.pgp>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From champ at softwink.com  Wed Jan 19 01:22:22 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Tue, 18 Jan 2011 19:22:22 -0500
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
In-Reply-To: <alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>
References: <20110118232908.GA30902@bundy.vistech.net>
	<alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>
Message-ID: <20110119002222.GA31632@bundy.vistech.net>

On Tue, Jan 18, 2011 at 03:47:00PM -0800, david at lang.hm wrote:
> you would need to escape the % in front of %SNMP, that's probably throwing 
> off the parser.
> 
> I don't know how you would actually _do_ that though.

	Yep. Me either.  :) Tried a couple of things, but it failed.
No biggie.  I'm sure Rainer will have an answer tomorrow. 

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110118/4f163a75/attachment.pgp>

From david at lang.hm  Wed Jan 19 03:41:18 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 18 Jan 2011 18:41:18 -0800 (PST)
Subject: [Lognorm] getting started document
In-Reply-To: <alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net>
	<alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
Message-ID: <alpine.DEB.2.00.1101181835070.5754@asgard.lang.hm>

based on another thread, some additional things that are needed (or at 
least need to be documented)

how to escape a '%'

it would be good to have some regex features available for the rules.

ones that I think are very useful

alternate words (this|that|other)

in word matches, being able to specify that this must match a pattern as 
well (eth* or [sh]da[0-9]* for example)


I don't know how far it makes sense to go, but if the regex can be 
compiled down to the parse tree with the other rules it shouldn't hurt 
performance, and if we can use regex rules that people have created for 
other tools, it should jumpstart the ruleset development.

David Lang

On Tue, 18 Jan 2011, david at lang.hm wrote:

> a couple things that jump out at me as being useful
>
> for numbers, be able to specify interger or decimal (sometimes numbers are 
> ##.## where they are two independant fields with the decimal as the separator
>
> besides the obvious IPv6 there is a need for predefined options
>
> such as anyIP = (IPv4 || IPv6)
>
> or host = (hostname || anyIP)
>
> there are also times when you want to specify multiple words as a single 
> field (i.e. next 2 words as field named X)
>
> wildcard to end of line
>
> with word matches, define allowed characters (hostname can be 0-9A-Za-z-_. 
> but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
>
> some way for the parser to report that the ruleset is ambiguous (two rules 
> match the same line)
>
> can you change the prefix during the ruleset?
>
>
> these things will matter more as useage grows (it will become harder and 
> harder to specify the difference between things)
>
>
>
> I see two forks to this project
>
> fork 1, the parser library (keeping it fast while letting it specify all the 
> nuances needed to match everything without ambiguity.
>
>
> fork 2. the rulesets for common log messages, defining them with common 
> termenology.
>
>
> I can see other projects using the parser, without using the rulesets (the 
> rsyslog parser module for example, it's mostly having to define all the 
> variations of possible message formats to just figure out how to reformat it 
> to a 'standard' syslog format for the other rules to handle)
>
>
> David Lang
>
>
> On Tue, 18 Jan 2011, Rainer Gerhards wrote:
>
>> Date: Tue, 18 Jan 2011 08:04:26 +0100
>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> Reply-To: lognorm <lognorm at lists.adiscon.com>
>> To: lognorm <lognorm at lists.adiscon.com>
>> Subject: Re: [Lognorm] getting started document
>> 
>> The web site is currently being built (any help on content, design, etc is
>> deeply appreciated). It is available at
>> 
>> http://www.liblognorm.com
>> 
>> In the menu there is a link to the current *very sparse* documentation.
>> 
>> On data types: there is not yet really such a thing as a "data type" -- all
>> are strings. Liblognorm uses a different concept right now, that is
>> "parsers". A parser actually describes a syntax that a string must fullfil 
>> in
>> order to be treated as "correct". There are a number of parsers, which 
>> should
>> be in the doc (but I see Florian did still not include the new ones, hope 
>> he
>> will do soon...). It is these types that the rules are build around.
>> 
>> I have recently asked which additional "types" (parsers) are considered
>> useful and I am ready to add new ones (I am sure there must be more than we
>> currently have).
>> 
>> We also need standard field names. I have started an effort on what to use
>> and Florian started a sample directory. We are looking for feedback here.
>> 
>> Finally, but very importantly, I'd like to add a section with rule bases to
>> the web site, where device-specifc rules can be found ... and contributed.
>> Suggestion on how to do this best would be very appreciated.
>> 
>> I'll also see if I can write a small getting started document. I guess it
>> should focus on the technical issues. Right now, I am doing a lot of 
>> writing,
>> so this should be possible as a side-activity (though I would prefer to 
>> code
>> a little me, but... ;)).
>> 
>> Hope that helps at least a little bit.
>> 
>> Rainer
>> 
>>> -----Original Message-----
>>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>> Sent: Tuesday, January 18, 2011 12:26 AM
>>> To: lognorm
>>> Subject: Re: [Lognorm] getting started document
>>> 
>>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
>>> 
>>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>>>>> the format of the rulebase files seems pretty straightforward, but
>>>>> where can I find the list of what datatypes are supported?
>>>>
>>>>        Keep in mind that liblognorm is pretty young at this point.
>>>> So more documentation probably needs to be done.
>>> 
>>> that's what I figured, and why I posted what i found needed to be done
>>> rather than just griping about it :-)
>>> 
>>>> Also, there's not much in the way of rulebase files yet, but that will
>>>> likely change in the future.  Right now, you're probably going to have
>>> to
>>>> write your own rulebase files.
>>> 
>>> I expected to have to write my own rules, but it's hard to write rules
>>> without knowing what datatypes are available.
>>> 
>>>> When you do that, keep a copy so that you
>>>> can send them in later.  :)
>>> 
>>> I expect to send a bundch in as I work on things, although I also
>>> expect
>>> to be using the parsing library in ways that don't directly tie in to
>>> the 'big normalization' project.
>>> 
>>> For example, Rainer wondered out loud about the possibility of using
>>> this
>>> library for a rsyslog log parser instead of writing the parsers
>>> directly
>>> in C. in that case, most of what you want in terms of normalization
>>> won't
>>> matter, as that parser is just focused on figuring out what part of the
>>> blob that arrives is the message, and what metadata in included, not in
>>> parsing the message itself into different pieces.
>>> 
>>>>> also, where can I find an example of how to compile the rulebase,
>>> and
>>>>> an example program that uses the rulebase?
>>>>
>>>>        Check out "normalizer.c" with the liblognorm git.  It's a
>>> basic
>>>> program that does normalization.
>>> 
>>> will do.
>>> 
>>> can you point me at what file (or files) the datatypes are defined in?
>>> 
>>> David Lang
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>> 
>

From rgerhards at hq.adiscon.com  Wed Jan 19 07:25:22 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Jan 2011 07:25:22 +0100
Subject: [Lognorm] getting started document
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net><alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
	<alpine.DEB.2.00.1101181835070.5754@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA3E@GRFEXC.intern.adiscon.com>

Just some quick points...

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, January 19, 2011 3:41 AM
> To: lognorm
> Subject: Re: [Lognorm] getting started document
> 
> based on another thread, some additional things that are needed (or at
> least need to be documented)
> 
> how to escape a '%'
> 
> it would be good to have some regex features available for the rules.

I know this seems desirable, but it is very dangerous. The speed of
liblognorm depends on very fast processing and limited backtracking. The more
regexpes are used, the more you get into the "normal" slow processing. I
agree there is a need for this, but I really intend to make it hard to use
regexpes so that they are only used if there is no way around. Otherwise, I
fear people turn back the familiar regexpes, because they can be used to do
what all other parses do. But they do this at an immense performance cost - a
cost so high that it can actually ruin the base concept. Having said that, I
am not even sure if I really intend to support them at all...

> 
> ones that I think are very useful
> 
> alternate words (this|that|other)

*Very* evil -- you can do this with separate rules.

> 
> in word matches, being able to specify that this must match a pattern
> as
> well (eth* or [sh]da[0-9]* for example)

Yeah, that's a possibility.

> 
> 
> I don't know how far it makes sense to go, but if the regex can be
> compiled down to the parse tree with the other rules it shouldn't hurt
> performance, and if we can use regex rules that people have created for
> other tools, it should jumpstart the ruleset development.

I am with you on the jumpstart, but if we base on regexpes, we can use the
other tools in the first place (because they are probably not bad, just
comparatively slow).

But I think I now got your full message: what I said is true iff a parser is
build using the regular regexp libraries. If liblognorm compiles the regexp
itself into the parse tree, the problem does not exist. This sounds very
good, but obviously is quite some work to do. But I keep it on my mind and
will think how I could best implement it. Something like the multi-word
functionality could probably be done via a preprocessing stage with
relatively little effort. Thanks for the useful idea!

Rainer

> 
> David Lang
> 
> On Tue, 18 Jan 2011, david at lang.hm wrote:
> 
> > a couple things that jump out at me as being useful
> >
> > for numbers, be able to specify interger or decimal (sometimes
> numbers are
> > ##.## where they are two independant fields with the decimal as the
> separator
> >
> > besides the obvious IPv6 there is a need for predefined options
> >
> > such as anyIP = (IPv4 || IPv6)
> >
> > or host = (hostname || anyIP)
> >
> > there are also times when you want to specify multiple words as a
> single
> > field (i.e. next 2 words as field named X)
> >
> > wildcard to end of line
> >
> > with word matches, define allowed characters (hostname can be 0-9A-
> Za-z-_.
> > but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
> >
> > some way for the parser to report that the ruleset is ambiguous (two
> rules
> > match the same line)
> >
> > can you change the prefix during the ruleset?
> >
> >
> > these things will matter more as useage grows (it will become harder
> and
> > harder to specify the difference between things)
> >
> >
> >
> > I see two forks to this project
> >
> > fork 1, the parser library (keeping it fast while letting it specify
> all the
> > nuances needed to match everything without ambiguity.
> >
> >
> > fork 2. the rulesets for common log messages, defining them with
> common
> > termenology.
> >
> >
> > I can see other projects using the parser, without using the rulesets
> (the
> > rsyslog parser module for example, it's mostly having to define all
> the
> > variations of possible message formats to just figure out how to
> reformat it
> > to a 'standard' syslog format for the other rules to handle)
> >
> >
> > David Lang
> >
> >
> > On Tue, 18 Jan 2011, Rainer Gerhards wrote:
> >
> >> Date: Tue, 18 Jan 2011 08:04:26 +0100
> >> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> >> Reply-To: lognorm <lognorm at lists.adiscon.com>
> >> To: lognorm <lognorm at lists.adiscon.com>
> >> Subject: Re: [Lognorm] getting started document
> >>
> >> The web site is currently being built (any help on content, design,
> etc is
> >> deeply appreciated). It is available at
> >>
> >> http://www.liblognorm.com
> >>
> >> In the menu there is a link to the current *very sparse*
> documentation.
> >>
> >> On data types: there is not yet really such a thing as a "data type"
> -- all
> >> are strings. Liblognorm uses a different concept right now, that is
> >> "parsers". A parser actually describes a syntax that a string must
> fullfil
> >> in
> >> order to be treated as "correct". There are a number of parsers,
> which
> >> should
> >> be in the doc (but I see Florian did still not include the new ones,
> hope
> >> he
> >> will do soon...). It is these types that the rules are build around.
> >>
> >> I have recently asked which additional "types" (parsers) are
> considered
> >> useful and I am ready to add new ones (I am sure there must be more
> than we
> >> currently have).
> >>
> >> We also need standard field names. I have started an effort on what
> to use
> >> and Florian started a sample directory. We are looking for feedback
> here.
> >>
> >> Finally, but very importantly, I'd like to add a section with rule
> bases to
> >> the web site, where device-specifc rules can be found ... and
> contributed.
> >> Suggestion on how to do this best would be very appreciated.
> >>
> >> I'll also see if I can write a small getting started document. I
> guess it
> >> should focus on the technical issues. Right now, I am doing a lot of
> >> writing,
> >> so this should be possible as a side-activity (though I would prefer
> to
> >> code
> >> a little me, but... ;)).
> >>
> >> Hope that helps at least a little bit.
> >>
> >> Rainer
> >>
> >>> -----Original Message-----
> >>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>> Sent: Tuesday, January 18, 2011 12:26 AM
> >>> To: lognorm
> >>> Subject: Re: [Lognorm] getting started document
> >>>
> >>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
> >>>
> >>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
> >>>>> the format of the rulebase files seems pretty straightforward,
> but
> >>>>> where can I find the list of what datatypes are supported?
> >>>>
> >>>>        Keep in mind that liblognorm is pretty young at this point.
> >>>> So more documentation probably needs to be done.
> >>>
> >>> that's what I figured, and why I posted what i found needed to be
> done
> >>> rather than just griping about it :-)
> >>>
> >>>> Also, there's not much in the way of rulebase files yet, but that
> will
> >>>> likely change in the future.  Right now, you're probably going to
> have
> >>> to
> >>>> write your own rulebase files.
> >>>
> >>> I expected to have to write my own rules, but it's hard to write
> rules
> >>> without knowing what datatypes are available.
> >>>
> >>>> When you do that, keep a copy so that you
> >>>> can send them in later.  :)
> >>>
> >>> I expect to send a bundch in as I work on things, although I also
> >>> expect
> >>> to be using the parsing library in ways that don't directly tie in
> to
> >>> the 'big normalization' project.
> >>>
> >>> For example, Rainer wondered out loud about the possibility of
> using
> >>> this
> >>> library for a rsyslog log parser instead of writing the parsers
> >>> directly
> >>> in C. in that case, most of what you want in terms of normalization
> >>> won't
> >>> matter, as that parser is just focused on figuring out what part of
> the
> >>> blob that arrives is the message, and what metadata in included,
> not in
> >>> parsing the message itself into different pieces.
> >>>
> >>>>> also, where can I find an example of how to compile the rulebase,
> >>> and
> >>>>> an example program that uses the rulebase?
> >>>>
> >>>>        Check out "normalizer.c" with the liblognorm git.  It's a
> >>> basic
> >>>> program that does normalization.
> >>>
> >>> will do.
> >>>
> >>> can you point me at what file (or files) the datatypes are defined
> in?
> >>>
> >>> David Lang
> >> _______________________________________________
> >> Lognorm mailing list
> >> Lognorm at lists.adiscon.com
> >> http://lists.adiscon.net/mailman/listinfo/lognorm
> >>
> >
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm

From rgerhards at hq.adiscon.com  Wed Jan 19 07:33:32 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Jan 2011 07:33:32 +0100
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
References: <20110118232908.GA30902@bundy.vistech.net><alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>
	<20110119002222.GA31632@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Wednesday, January 19, 2011 1:22 AM
> To: lognorm
> Subject: Re: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
> 
> On Tue, Jan 18, 2011 at 03:47:00PM -0800, david at lang.hm wrote:
> > you would need to escape the % in front of %SNMP, that's probably
> throwing
> > off the parser.
> >
> > I don't know how you would actually _do_ that though.
> 
> 	Yep. Me either.  :) Tried a couple of things, but it failed.
> No biggie.  I'm sure Rainer will have an answer tomorrow.


Out of my head it should be %%. Will check and hope to be able to finally
write the short doc today (but I am in a complex test lab and don't like to
lose momentum -- as usual, it takes longer than expexted... ;)).

Rainer
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From david at lang.hm  Wed Jan 19 07:34:15 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 18 Jan 2011 22:34:15 -0800 (PST)
Subject: [Lognorm] getting started document
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA3E@GRFEXC.intern.adiscon.com>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm><alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm><20110117225313.GA16101@bundy.vistech.net><alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com><alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
	<alpine.DEB.2.00.1101181835070.5754@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.00.1101182227560.9980@asgard.lang.hm>

On Wed, 19 Jan 2011, Rainer Gerhards wrote:

> Just some quick points...
>
>> -----Original Message-----
>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Wednesday, January 19, 2011 3:41 AM
>> To: lognorm
>> Subject: Re: [Lognorm] getting started document
>>
>> based on another thread, some additional things that are needed (or at
>> least need to be documented)
>>
>> how to escape a '%'
>>
>> it would be good to have some regex features available for the rules.
>
> I know this seems desirable, but it is very dangerous. The speed of
> liblognorm depends on very fast processing and limited backtracking. The more
> regexpes are used, the more you get into the "normal" slow processing. I
> agree there is a need for this, but I really intend to make it hard to use
> regexpes so that they are only used if there is no way around. Otherwise, I
> fear people turn back the familiar regexpes, because they can be used to do
> what all other parses do. But they do this at an immense performance cost - a
> cost so high that it can actually ruin the base concept. Having said that, I
> am not even sure if I really intend to support them at all...

agreed, general regex is a problem, but if the regex can compile down to a 
parse tree readily it should not be that bad.

>>
>> ones that I think are very useful
>>
>> alternate words (this|that|other)
>
> *Very* evil -- you can do this with separate rules.

what if the rule->parse tree engine realizes this and takes the one line 
of configuration and submits multiple lines to the next step?

it's a trade off between conversion complexity and maintainability of the 
rulesets by humans. If adding a little complexity in the converter can 
shrink a ruleset by a significant factor (I'm not talking about 1% 
savings, I'm thinking 75% savings), it can be well worth it.

>>
>> in word matches, being able to specify that this must match a pattern
>> as
>> well (eth* or [sh]da[0-9]* for example)
>
> Yeah, that's a possibility.
>
>>
>>
>> I don't know how far it makes sense to go, but if the regex can be
>> compiled down to the parse tree with the other rules it shouldn't hurt
>> performance, and if we can use regex rules that people have created for
>> other tools, it should jumpstart the ruleset development.
>
> I am with you on the jumpstart, but if we base on regexpes, we can use the
> other tools in the first place (because they are probably not bad, just
> comparatively slow).

speed is a feature in and of itself. But that being said, just using the 
regex isn't enough, you have to tag what each of the fields mean as well. 
If we then tag them with the standardized terms we can then output them 
into a standard format.

> But I think I now got your full message: what I said is true iff a parser is
> build using the regular regexp libraries. If liblognorm compiles the regexp
> itself into the parse tree, the problem does not exist. This sounds very
> good, but obviously is quite some work to do. But I keep it on my mind and
> will think how I could best implement it. Something like the multi-word
> functionality could probably be done via a preprocessing stage with
> relatively little effort. Thanks for the useful idea!

exactly, I'm glad I gave enough examples to get the point across.

David Lang

> Rainer
>
>>
>> David Lang
>>
>> On Tue, 18 Jan 2011, david at lang.hm wrote:
>>
>>> a couple things that jump out at me as being useful
>>>
>>> for numbers, be able to specify interger or decimal (sometimes
>> numbers are
>>> ##.## where they are two independant fields with the decimal as the
>> separator
>>>
>>> besides the obvious IPv6 there is a need for predefined options
>>>
>>> such as anyIP = (IPv4 || IPv6)
>>>
>>> or host = (hostname || anyIP)
>>>
>>> there are also times when you want to specify multiple words as a
>> single
>>> field (i.e. next 2 words as field named X)
>>>
>>> wildcard to end of line
>>>
>>> with word matches, define allowed characters (hostname can be 0-9A-
>> Za-z-_.
>>> but not +)(*&^%$#@!~`[]{};:'"/?,<>=+ for example)
>>>
>>> some way for the parser to report that the ruleset is ambiguous (two
>> rules
>>> match the same line)
>>>
>>> can you change the prefix during the ruleset?
>>>
>>>
>>> these things will matter more as useage grows (it will become harder
>> and
>>> harder to specify the difference between things)
>>>
>>>
>>>
>>> I see two forks to this project
>>>
>>> fork 1, the parser library (keeping it fast while letting it specify
>> all the
>>> nuances needed to match everything without ambiguity.
>>>
>>>
>>> fork 2. the rulesets for common log messages, defining them with
>> common
>>> termenology.
>>>
>>>
>>> I can see other projects using the parser, without using the rulesets
>> (the
>>> rsyslog parser module for example, it's mostly having to define all
>> the
>>> variations of possible message formats to just figure out how to
>> reformat it
>>> to a 'standard' syslog format for the other rules to handle)
>>>
>>>
>>> David Lang
>>>
>>>
>>> On Tue, 18 Jan 2011, Rainer Gerhards wrote:
>>>
>>>> Date: Tue, 18 Jan 2011 08:04:26 +0100
>>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>>> Reply-To: lognorm <lognorm at lists.adiscon.com>
>>>> To: lognorm <lognorm at lists.adiscon.com>
>>>> Subject: Re: [Lognorm] getting started document
>>>>
>>>> The web site is currently being built (any help on content, design,
>> etc is
>>>> deeply appreciated). It is available at
>>>>
>>>> http://www.liblognorm.com
>>>>
>>>> In the menu there is a link to the current *very sparse*
>> documentation.
>>>>
>>>> On data types: there is not yet really such a thing as a "data type"
>> -- all
>>>> are strings. Liblognorm uses a different concept right now, that is
>>>> "parsers". A parser actually describes a syntax that a string must
>> fullfil
>>>> in
>>>> order to be treated as "correct". There are a number of parsers,
>> which
>>>> should
>>>> be in the doc (but I see Florian did still not include the new ones,
>> hope
>>>> he
>>>> will do soon...). It is these types that the rules are build around.
>>>>
>>>> I have recently asked which additional "types" (parsers) are
>> considered
>>>> useful and I am ready to add new ones (I am sure there must be more
>> than we
>>>> currently have).
>>>>
>>>> We also need standard field names. I have started an effort on what
>> to use
>>>> and Florian started a sample directory. We are looking for feedback
>> here.
>>>>
>>>> Finally, but very importantly, I'd like to add a section with rule
>> bases to
>>>> the web site, where device-specifc rules can be found ... and
>> contributed.
>>>> Suggestion on how to do this best would be very appreciated.
>>>>
>>>> I'll also see if I can write a small getting started document. I
>> guess it
>>>> should focus on the technical issues. Right now, I am doing a lot of
>>>> writing,
>>>> so this should be possible as a side-activity (though I would prefer
>> to
>>>> code
>>>> a little me, but... ;)).
>>>>
>>>> Hope that helps at least a little bit.
>>>>
>>>> Rainer
>>>>
>>>>> -----Original Message-----
>>>>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>> Sent: Tuesday, January 18, 2011 12:26 AM
>>>>> To: lognorm
>>>>> Subject: Re: [Lognorm] getting started document
>>>>>
>>>>> From: "Champ Clark III [Softwink]" <champ at softwink.com>
>>>>>
>>>>>> On Mon, Jan 17, 2011 at 01:33:00PM -0800, david at lang.hm wrote:
>>>>>>> the format of the rulebase files seems pretty straightforward,
>> but
>>>>>>> where can I find the list of what datatypes are supported?
>>>>>>
>>>>>>        Keep in mind that liblognorm is pretty young at this point.
>>>>>> So more documentation probably needs to be done.
>>>>>
>>>>> that's what I figured, and why I posted what i found needed to be
>> done
>>>>> rather than just griping about it :-)
>>>>>
>>>>>> Also, there's not much in the way of rulebase files yet, but that
>> will
>>>>>> likely change in the future.  Right now, you're probably going to
>> have
>>>>> to
>>>>>> write your own rulebase files.
>>>>>
>>>>> I expected to have to write my own rules, but it's hard to write
>> rules
>>>>> without knowing what datatypes are available.
>>>>>
>>>>>> When you do that, keep a copy so that you
>>>>>> can send them in later.  :)
>>>>>
>>>>> I expect to send a bundch in as I work on things, although I also
>>>>> expect
>>>>> to be using the parsing library in ways that don't directly tie in
>> to
>>>>> the 'big normalization' project.
>>>>>
>>>>> For example, Rainer wondered out loud about the possibility of
>> using
>>>>> this
>>>>> library for a rsyslog log parser instead of writing the parsers
>>>>> directly
>>>>> in C. in that case, most of what you want in terms of normalization
>>>>> won't
>>>>> matter, as that parser is just focused on figuring out what part of
>> the
>>>>> blob that arrives is the message, and what metadata in included,
>> not in
>>>>> parsing the message itself into different pieces.
>>>>>
>>>>>>> also, where can I find an example of how to compile the rulebase,
>>>>> and
>>>>>>> an example program that uses the rulebase?
>>>>>>
>>>>>>        Check out "normalizer.c" with the liblognorm git.  It's a
>>>>> basic
>>>>>> program that does normalization.
>>>>>
>>>>> will do.
>>>>>
>>>>> can you point me at what file (or files) the datatypes are defined
>> in?
>>>>>
>>>>> David Lang
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>

From champ at softwink.com  Wed Jan 19 15:37:37 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Wed, 19 Jan 2011 09:37:37 -0500
Subject: [Lognorm] liblognorm default field names
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA0E@GRFEXC.intern.adiscon.com>
References: <20110113153409.GA15921@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9F7@GRFEXC.intern.adiscon.com>
	<20110113173334.GA19911@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DD9FD@GRFEXC.intern.adiscon.com>
	<20110113183413.GA22721@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA04@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA0E@GRFEXC.intern.adiscon.com>
Message-ID: <20110119143737.GA27809@bundy.vistech.net>

On Fri, Jan 14, 2011 at 04:33:51PM +0100, Florian Riedl wrote:
> Dear liblognorm fans,
> 
> we are currently trying to compile some sort of list for field names that
> should be a standard in liblognorm. 
> 	
> We already thought about a few. Like the following:
> Src-ip = Source IP -> IPv4
> Dst-ip = Destination IP -> IPv4
> 
> http://www.liblognorm.com/field-names/

	Aside from the new fields (quoted-string, etc),  we might want
to consider a src-host/dst-host (or something of the likes).  In some
cases,  I'll see sshd (for example) with UseDNS enabled.   In those
cases rather than a src-ip,  i'd grab a src-host.  


-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110119/2fbc5a45/attachment.pgp>

From champ at softwink.com  Wed Jan 19 15:53:04 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Wed, 19 Jan 2011 09:53:04 -0500
Subject: [Lognorm] getting started document
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA3E@GRFEXC.intern.adiscon.com>
References: <alpine.DEB.2.00.1101171308060.9746@asgard.lang.hm>
	<alpine.DEB.2.00.1101171330300.9746@asgard.lang.hm>
	<20110117225313.GA16101@bundy.vistech.net>
	<alpine.DEB.2.00.1101171520250.9980@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA1C@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.00.1101180019500.9980@asgard.lang.hm>
	<alpine.DEB.2.00.1101181835070.5754@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA3E@GRFEXC.intern.adiscon.com>
Message-ID: <20110119145304.GB27809@bundy.vistech.net>

> > based on another thread, some additional things that are needed (or at
> > least need to be documented)
> > 
> > how to escape a '%'
> > 
> > it would be good to have some regex features available for the rules.
> 
> I know this seems desirable, but it is very dangerous. The speed of
> liblognorm depends on very fast processing and limited backtracking. The more
> regexpes are used, the more you get into the "normal" slow processing. I
> agree there is a need for this, but I really intend to make it hard to use
> regexpes so that they are only used if there is no way around. Otherwise, I
> fear people turn back the familiar regexpes, because they can be used to do
> what all other parses do. But they do this at an immense performance cost - a
> cost so high that it can actually ruin the base concept. Having said that, I
> am not even sure if I really intend to support them at all...

	I'm with Rainer on this one.   I know we talk about log
normalization,  but really,  to me,  lognorm does more "masking" of the
log entires to extract useable/desired information out of.   I haven't
seen a situation __YET__ that regex/pcre would be "helpful".   That's
not to say it _wouldn't_ be,  but I've just not seen it. 

> > 
> > ones that I think are very useful
> > 
> > alternate words (this|that|other)
> 
> *Very* evil -- you can do this with separate rules.

	I ran into a similar situation yesterday.... Like this:

Domain of sender example.com does not exist|resolv 

	Break it down to one rule:

Domain of sender example.com does not %reason:word%

	While probably not desirable 100% of the time,  it'll allow
you to deal with the this|that|other situation.  It _might_ be nice
to do something like...

Domain of sender example.com does not %mutli-word:resolv|exist%

	That could be slick :)

> > in word matches, being able to specify that this must match a pattern
> > as
> > well (eth* or [sh]da[0-9]* for example)
> 
> Yeah, that's a possibility.

	We're getting back into regex :)

> > I don't know how far it makes sense to go, but if the regex can be
> > compiled down to the parse tree with the other rules it shouldn't hurt
> > performance, and if we can use regex rules that people have created for
> > other tools, it should jumpstart the ruleset development.
> 
> I am with you on the jumpstart, but if we base on regexpes, we can use the
> other tools in the first place (because they are probably not bad, just
> comparatively slow).
> 
> But I think I now got your full message: what I said is true iff a parser is
> build using the regular regexp libraries. If liblognorm compiles the regexp
> itself into the parse tree, the problem does not exist. This sounds very
> good, but obviously is quite some work to do. But I keep it on my mind and
> will think how I could best implement it. Something like the multi-word
> functionality could probably be done via a preprocessing stage with
> relatively little effort. Thanks for the useful idea!

	I do like the multi-word idea.   That'd be pretty nice.  I can
only speak from experience of PCRE ( ie - www.pcre.org ),  but even if
you take the "rule" and pre - pcre_compile down the patterns,  you still 
take a preformance hit.  On the surface,  it doesn't look like much.
However,  once you start thowing millions of log lines at it,  that's
when you'll notice.  This is the same reason software like Snort
(IPS/IDS) prefer the use of "content:" over "pcre:",  when possible. 

	I've toyed with this in my head a little bit,  and I can see in 
certain situation where pcre/regex might be useful.  My fear is that 
lognorm will end up as 'yet another regex log parser'.   I'm just not
100% sure if regexp will be worth it (?).  May or may not be. 

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110119/76461d1a/attachment.pgp>

From champ at softwink.com  Wed Jan 19 15:55:25 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Wed, 19 Jan 2011 09:55:25 -0500
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com>
References: <20110118232908.GA30902@bundy.vistech.net>
	<alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>
	<20110119002222.GA31632@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com>
Message-ID: <20110119145525.GC27809@bundy.vistech.net>

> Out of my head it should be %%. Will check and hope to be able to finally
> write the short doc today (but I am in a complex test lab and don't like to
> lose momentum -- as usual, it takes longer than expexted... ;)).

	Thanks,  that did it.  I can take a crack at writing the doc, 
if that'd help lighten the load?

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110119/7dcaa93f/attachment-0001.pgp>

From rgerhards at hq.adiscon.com  Wed Jan 19 16:30:05 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Jan 2011 16:30:05 +0100
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
References: <20110118232908.GA30902@bundy.vistech.net><alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm><20110119002222.GA31632@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com>
	<20110119145525.GC27809@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA53@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Wednesday, January 19, 2011 3:55 PM
> To: lognorm
> Subject: Re: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
> 
> > Out of my head it should be %%. Will check and hope to be able to
> finally
> > write the short doc today (but I am in a complex test lab and don't
> like to
> > lose momentum -- as usual, it takes longer than expexted... ;)).
> 
> 	Thanks,  that did it.  I can take a crack at writing the doc,
> if that'd help lighten the load?

That would be *very* useful and a big aid in adoption!

Rainer

From champ at softwink.com  Wed Jan 19 16:49:31 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Wed, 19 Jan 2011 10:49:31 -0500
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA53@GRFEXC.intern.adiscon.com>
References: <20110118232908.GA30902@bundy.vistech.net>
	<alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm>
	<20110119002222.GA31632@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com>
	<20110119145525.GC27809@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA53@GRFEXC.intern.adiscon.com>
Message-ID: <20110119154931.GA1396@bundy.vistech.net>

> > 	Thanks,  that did it.  I can take a crack at writing the doc,
> > if that'd help lighten the load?
> 
> That would be *very* useful and a big aid in adoption!

	I'll start writing something up this evening.  I'm might have 
a couple of questions, but I think I have the hang of the bulk of it.
Once I'm done,  I'll send it to you for review/posting. 

	Good?
-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110119/e5b23149/attachment.pgp>

From rgerhards at hq.adiscon.com  Wed Jan 19 16:57:50 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 19 Jan 2011 16:57:50 +0100
Subject: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
References: <20110118232908.GA30902@bundy.vistech.net><alpine.DEB.2.00.1101181546230.5754@asgard.lang.hm><20110119002222.GA31632@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA40@GRFEXC.intern.adiscon.com><20110119145525.GC27809@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA53@GRFEXC.intern.adiscon.com>
	<20110119154931.GA1396@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA54@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Wednesday, January 19, 2011 4:50 PM
> To: lognorm
> Subject: Re: [Lognorm] Cisco %SNMP-3-AUTHFAIL:
> 
> > > 	Thanks,  that did it.  I can take a crack at writing the doc,
> > > if that'd help lighten the load?
> >
> > That would be *very* useful and a big aid in adoption!
> 
> 	I'll start writing something up this evening.  I'm might have
> a couple of questions, but I think I have the hang of the bulk of it.
> Once I'm done,  I'll send it to you for review/posting.
> 
> 	Good?
Excellent!

Rainer

> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Thu Jan 20 16:15:19 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Thu, 20 Jan 2011 10:15:19 -0500
Subject: [Lognorm] Thought about %word%
Message-ID: <20110120151519.GA18423@bundy.vistech.net>


	Hello, 

	The other day I was working on some Sendmail rules (doing some
testing) and I noticed something about the %field:word%.  Here's a 
quick example:

arg1=<bob at example.com>

	What I'd planned on doing was:

arg1=<%email:word%>

	But that obviously won't work.  In the end,  you end up grabbing 
the entire line (due to spaces).  Perhaps %word% should not _just_ 
grab between spacing,  but any non-alpha/numberic.  This might lead into
other problem.  For example,  %word% still wouldn't work due to the 
"@".  Or,  maybe it's a bad idea all the way around.  Perhaps a preset
of defined delimiters ( < > [ ] (space) = , ) ?

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110120/e4c19b47/attachment.pgp>

From rgerhards at hq.adiscon.com  Thu Jan 20 16:24:37 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Jan 2011 16:24:37 +0100
Subject: [Lognorm] Thought about %word%
References: <20110120151519.GA18423@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA6A@GRFEXC.intern.adiscon.com>

Word is actually a catch-all parser, and a thing to avoid as better parsers
come it. I will soon make sure it will be tried as the last one. An email
parsers seems to make a lot of sense. And probably a "alpha" parser, which
takes a-zA-Z only. The bottom line is that the more precise, the better.

On the other hand, David's comments on regexpes was very interesting.
Regexpes are compiled to DFAs (deterministic finite automatons). Executing
the DFA takes some time. Executing many of them takes *a lot* of time. In
liblognorm, I combine all of those rules into a single DFA, thus it is much
quicker. But it may be possible to use the "regular" regexp logic and compile
many regexpes into a single DFA (liblognorms parse tree) and gain ease of use
of regexpes plus the speed of liblognorm. But this for sure sound like a very
challenging effort...

Rainer

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Thursday, January 20, 2011 4:15 PM
> To: lognorm at lists.adiscon.com
> Subject: [Lognorm] Thought about %word%
> 
> 
> 	Hello,
> 
> 	The other day I was working on some Sendmail rules (doing some
> testing) and I noticed something about the %field:word%.  Here's a
> quick example:
> 
> arg1=<bob at example.com>
> 
> 	What I'd planned on doing was:
> 
> arg1=<%email:word%>
> 
> 	But that obviously won't work.  In the end,  you end up grabbing
> the entire line (due to spaces).  Perhaps %word% should not _just_
> grab between spacing,  but any non-alpha/numberic.  This might lead
> into
> other problem.  For example,  %word% still wouldn't work due to the
> "@".  Or,  maybe it's a bad idea all the way around.  Perhaps a preset
> of defined delimiters ( < > [ ] (space) = , ) ?
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.

From champ at softwink.com  Thu Jan 20 16:29:53 2011
From: champ at softwink.com (Champ Clark III [Softwink])
Date: Thu, 20 Jan 2011 10:29:53 -0500
Subject: [Lognorm] Thought about %word%
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDA6A@GRFEXC.intern.adiscon.com>
References: <20110120151519.GA18423@bundy.vistech.net>
	<9B6E2A8877C38245BFB15CC491A11DA71DDA6A@GRFEXC.intern.adiscon.com>
Message-ID: <20110120152953.GA19342@bundy.vistech.net>

On Thu, Jan 20, 2011 at 04:24:37PM +0100, Rainer Gerhards wrote:
> Word is actually a catch-all parser, and a thing to avoid as better parsers
> come it. I will soon make sure it will be tried as the last one. An email
> parsers seems to make a lot of sense. And probably a "alpha" parser, which
> takes a-zA-Z only. The bottom line is that the more precise, the better.

	Okay.. I just wanted to throw that out there.  Technically,  it
really hasn't been a issue for me. 

-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110120/fb577d16/attachment.pgp>

From rgerhards at hq.adiscon.com  Thu Jan 20 16:31:47 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 20 Jan 2011 16:31:47 +0100
Subject: [Lognorm] Thought about %word%
References: <20110120151519.GA18423@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DDA6A@GRFEXC.intern.adiscon.com>
	<20110120152953.GA19342@bundy.vistech.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDA6D@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Thursday, January 20, 2011 4:30 PM
> To: lognorm
> Subject: Re: [Lognorm] Thought about %word%
> 
> On Thu, Jan 20, 2011 at 04:24:37PM +0100, Rainer Gerhards wrote:
> > Word is actually a catch-all parser, and a thing to avoid as better
> parsers
> > come it. I will soon make sure it will be tried as the last one. An
> email
> > parsers seems to make a lot of sense. And probably a "alpha" parser,
> which
> > takes a-zA-Z only. The bottom line is that the more precise, the
> better.
> 
> 	Okay.. I just wanted to throw that out there.  Technically,  it
> really hasn't been a issue for me.
> 

I guess it becomes an issue when the rulebases become larger. Parsers like
word increase the chance of rule interference -- greatly (IMHO, but
untested).

RAiner