[Lognorm] Sonicwall Normalization tests & other....
Rainer Gerhards
rgerhards at hq.adiscon.com
Fri Jan 14 09:01:35 CET 2011
I just committed a couple of new parsers (only the hex number is still
missing), see these commits:
http://git.adiscon.com/?p=libee.git;a=commitdiff;h=429915802b1a37bd6e0283888f
59978e7258f3a3
http://git.adiscon.com/?p=liblognorm.git;a=commitdiff;h=f63f04cfd08ab1611b761
93cf0a80b036c3cd6e2
I have done basic testing, but it would be nice if you could play a little
bit with them. Doc is upcoming, but their use should be straightforward from
the source comments. QuotedString pulls the quotes, but does not make them
part of the extracted field. The parse pointer is placed right after the
closing quote.
I intend to do new releases early next week with the new features and bug
fixes.
As always, feedback is appreciated.
Rainer
> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Thursday, January 13, 2011 5:24 PM
> To: Champ Clark III [Softwink]
> Cc: lognorm at lists.adiscon.com
> Subject: Re: [Lognorm] Sonicwall Normalization tests & other....
>
> > -----Original Message-----
> > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> > Sent: Thursday, January 13, 2011 4:34 PM
> > To: lognorm at lists.adiscon.com
> > Subject: [Lognorm] Sonicwall Normalization tests & other....
> >
> >
> > Yay! The new liblognorm mailing list!
>
> Welcome, you managed the first-ever post ;)
>
> >
> > FP! Woo. Okay, silliness aside.
> >
> > I started doing some "serious" log normalization with liblognorm
> while
> > working
> > a new sonicwall.rules for Sagan (and a sonicwall.rulebase for
> > liblognorm).
> > The good news, after about 20 hours of running, it's been
> normalizing
> > wonderfully and hasn't "blown up". Here's an example of what it's
> > currently
> > "normalizing" (sanitized):
> >
> > id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1
> > pri=1 c=32 m=608 msg="IPS Detection Alert: ICMP Echo Reply" sid=316
> > ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0
> >
> > I have a pretty generic Sagan rule to be triggered:
> >
> > alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS
> > Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-
> > event; normalize: sonicwall; reference:
> > url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)
> >
> > (Note the "normalize: sonicwall" flag).
> >
> > The liblognorm rule looks like thus (from my sonicwall.rulebase).
> >
> > --<snip>----
> >
> > prefix=id=%firewall:word% sn=%serial:word% time="%date:word%
> > %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4%
> > pri=%pri:number% c=%c:number% m=%m:number%
> >
> > rule=: msg="IPS Detection Alert: %t1:word% %t2:word %t3:word%
> > sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number%
> > n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word%
> > dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%
> >
> > --<snip>----
> >
> > Actually, for the "IPS Detection Alert", I had to write
> > multiple liblognorm
> > rules.
>
> I came across this with iptables as welll. The root cause is that the
> data is
> (at least) semi-structured: it already has name-value pairs, but the
> order is
> not necessarily normalized. One idea I have is to create a special
> parser
> that pulls all name-value pairs and assigns the appropriate fields.
> However,
> there are many subtle issues with that (the most important one probably
> is
> the need to rename fields so that they have a canonical form -- I have
> not
> yet thought enough about that [in terms of actual config syntax]).
>
> > This it might be "IPS Detection: ICMP PING" or "IPS Detection:
> > ICMP PING
> > reply" (ie - multiple fields). Over all, I'm okay with this.
>
> Oh yeah, that's another one, but I think can not be solved by keeping
> it
> efficient.
>
> > However, I did
> > run into some interesting issues:
> >
> > 1. Note the "time="%date:word%
> > %hour:number%:%minute:number%:%seconds:number%"
> > in the prefix. There probably needs to be a parser for date
> (2011-
> > 01-13)
> > format. There probably also needs to be a parser for the 24 clock
> > (10:05:14). Doing something like "%hour:word% doesn't work, so
> > I'm
> > breaking up the fields manually. I'm assuming this is caused by
> the
> > :'s.
>
> OK, parsers needed:
> ISO-Date
> Timestamps (12/24 hour clocks)
> integer (don't we have this already)
>
> Word is terminated by a space, the integer would be terminated by the
> first
> non-digit. Hex numbers would probably be useful as well.
>
> I'll see that I implement these.
>
> >
> > 2. Check out the "msg=" portion. If you look closely/compare the
> > "real"
> > output with the lognorm rule, it's missing a ". For example,
> the
> > log line is actually:
> >
> > msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....
> >
> > But the rule is:
> >
> > IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%
>
> OT: there it is "number" is an integer (oh, the joys of vacation and
> forgetting things... ;)
>
> >
> > The last %t3:word% captures not only the item in the string but
> the
> > " as well. Not sure if that's a "issue", but thought I'd
> mention
> > it.
>
> Good point! Next parser:
> quoted String! Very important one.
>
> >
> > Lastly, on a side note, I've had to alter my rsyslog.conf to the
> > following:
> >
> > $template sagan, "%fromhost-ip%|%syslogfacility-
> text%|%syslogpriority-
> > text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-
> > rfc3339%|%timegenerated:12:19:date-
> > rfc3339%|%programname%|%syslogtag%%msg%\n"
> >
> > Note the end of the line. For some reason, syslog-ng likes to
> > prepend the 'program[pid]' as part of the message. I researched
> this,
> > and
> > I don't see any way to remove it via syslog-ng, so my "fix" was to
> > have rsyslog do the same. It's annoying, but I have a feeling that
> > bringing
> > it up with the syslog-ng team will just lead to bickering about what
> > "should"
> > be part of the %msg% (or $MSG) and what shouldn't be.
>
> probably, but I can't do much against this ;) (RFC3164 says it is
> not...)
> >
> > Also, without the CEE definitions, I'm obviously "making up"
> > field
> > names as I go. If you have any pointers to "real" CEE definitions
> of
> > fields,
> > that would help. If I recall, they haven't been finalized/released
> > yet (?)
>
> still the same status. I think it takes another few weeks until this
> changes.
> I suggest that in the interim we create a dictionary of field
> names/semantics
> on liblognorm.org as an interim solution. The good news is that I could
> add
> them as an alias to the then-finalized CEE dictionary and make libee
> replace
> aliases with "the real thing". That means you do not need to change
> things
> manually, at least at this level.
>
> I'll see that we setup an area for the dictionary on the website very
> soon. I
> hope for your feedback on getting this right ;)
>
> Rainer
> >
> >
> > Anyways, that's all.. Let me know what you thing.
> >
> >
> > --
> > Champ Clark III | Softwink, Inc | 800-538-9357 x 101
> > http://www.softwink.com
> >
> > GPG Key ID: 58A2A58F
> > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F
> > If it wasn't for C, we'd be using BASI, PASAL and OBOL.
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
More information about the Lognorm
mailing list