[Lognorm] Sonicwall Normalization tests & other....

[Rainer Gerhards]

> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink]
> Sent: Thursday, January 13, 2011 4:34 PM
> To: lognorm at lists.adiscon.com
> Subject: [Lognorm] Sonicwall Normalization tests & other....
> 
> 
> Yay!  The new liblognorm mailing list!

Welcome, you managed the first-ever post ;)
 
> 
> FP!  Woo.  Okay,  silliness aside.
> 
> I started doing some "serious" log normalization with liblognorm while
> working
> a new sonicwall.rules for Sagan (and a sonicwall.rulebase for
> liblognorm).
> The good news,  after about 20 hours of running,  it's been normalizing
> wonderfully and hasn't "blown up".   Here's an example of what it's
> currently
> "normalizing" (sanitized):
> 
> id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1
> pri=1 c=32 m=608  msg="IPS Detection Alert: ICMP Echo Reply" sid=316
> ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0
> 
> I have a pretty generic Sagan rule to be triggered:
> 
> alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS
> Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-
> event; normalize: sonicwall; reference:
> url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)
> 
> (Note the "normalize: sonicwall" flag).
> 
> The liblognorm rule looks like thus (from my sonicwall.rulebase).
> 
> --<snip>----
> 
> prefix=id=%firewall:word% sn=%serial:word% time="%date:word%
> %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4%
> pri=%pri:number% c=%c:number% m=%m:number%
> 
> rule=:  msg="IPS Detection Alert: %t1:word% %t2:word %t3:word%
> sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number%
> n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word%
> dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%
> 
> --<snip>----
> 
> 	Actually,  for the "IPS Detection Alert",  I had to write
> multiple liblognorm
> rules.

I came across this with iptables as welll. The root cause is that the data is
(at least) semi-structured: it already has name-value pairs, but the order is
not necessarily normalized. One idea I have is to create a special parser
that pulls all name-value pairs and assigns the appropriate fields. However,
there are many subtle issues with that (the most important one probably is
the need to rename fields so that they have a canonical form -- I have not
yet thought enough about that [in terms of actual config syntax]).

>  This it might be "IPS Detection: ICMP PING" or "IPS Detection:
> ICMP PING
> reply" (ie - multiple fields).  Over all,  I'm okay with this.

Oh yeah, that's another one, but I think can not be solved by keeping it
efficient.

> However,  I did
> run into some interesting issues:
> 
> 1. Note the "time="%date:word%
> %hour:number%:%minute:number%:%seconds:number%"
>    in the prefix.  There probably needs to be a parser for date (2011-
> 01-13)
>    format.  There probably also needs to be a parser for the 24 clock
>    (10:05:14).   Doing something like "%hour:word% doesn't work,  so
> I'm
>    breaking up the fields manually.  I'm assuming this is caused by the
>    :'s.

OK, parsers needed:
ISO-Date
Timestamps (12/24 hour clocks)
integer (don't we have this already)

Word is terminated by a space, the integer would be terminated by the first
non-digit. Hex numbers would probably be useful as well.

I'll see that I implement these.

> 
> 2. Check out the "msg=" portion.  If you look closely/compare the
> "real"
>    output with the lognorm rule,  it's missing a ".   For example,  the
>    log line is actually:
> 
>    msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....
> 
>    But the rule is:
> 
>    IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%

OT: there it is "number" is an integer (oh, the joys of vacation and
forgetting things... ;)

> 
>    The last %t3:word% captures not only the item in the string but the
>    " as well.   Not sure if that's a "issue",  but thought I'd mention
>    it.

Good point! Next parser:
quoted String! Very important one.

> 
>    Lastly,  on a side note,  I've had to alter my rsyslog.conf to the
> following:
> 
> $template sagan, "%fromhost-ip%|%syslogfacility-text%|%syslogpriority-
> text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-
> rfc3339%|%timegenerated:12:19:date-
> rfc3339%|%programname%|%syslogtag%%msg%\n"
> 
> 	Note the end of the line.  For some reason,  syslog-ng likes to
> prepend the 'program[pid]' as part of the message.  I researched this,
> and
> I don't see any way to remove it via syslog-ng,  so my "fix" was to
> have rsyslog do the same.  It's annoying,  but I have a feeling that
> bringing
> it up with the syslog-ng team will just lead to bickering about what
> "should"
> be part of the %msg% (or $MSG) and what shouldn't be.

probably, but I can't do much against this ;) (RFC3164 says it is not...)
> 
> 	Also,  without the CEE definitions,  I'm obviously "making up"
> field
> names as I go.   If you have any pointers to "real" CEE definitions of
> fields,
> that would help.  If I recall,   they haven't been finalized/released
> yet (?)

still the same status. I think it takes another few weeks until this changes.
I suggest that in the interim we create a dictionary of field names/semantics
on liblognorm.org as an interim solution. The good news is that I could add
them as an alias to the then-finalized CEE dictionary and make libee replace
aliases with "the real thing". That means you do not need to change things
manually, at least at this level.

I'll see that we setup an area for the dictionary on the website very soon. I
hope for your feedback on getting this right ;)

Rainer
> 
> 
> 	Anyways,  that's all.. Let me know what you thing.
> 
> 
> --
>         Champ Clark III | Softwink, Inc | 800-538-9357 x 101
>                      http://www.softwink.com
> 
> GPG Key ID: 58A2A58F
> Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
> If it wasn't for C, we'd be using BASI, PASAL and OBOL.