[Lognorm] Sonicwall Normalization tests & other....

[Champ Clark III [Softwink]]

Yay!  The new liblognorm mailing list!  

FP!  Woo.  Okay,  silliness aside. 

I started doing some "serious" log normalization with liblognorm while working
a new sonicwall.rules for Sagan (and a sonicwall.rulebase for liblognorm). 
The good news,  after about 20 hours of running,  it's been normalizing 
wonderfully and hasn't "blown up".   Here's an example of what it's currently
"normalizing" (sanitized):

id=firewall sn=012346788ABC time="2011-01-13 10:05:14" fw=192.168.0.1 pri=1 c=32 m=608  msg="IPS Detection Alert: ICMP Echo Reply" sid=316 ipscat=ICMP ipspri=3 n=0 src=10.1.0.1:8:X1 dst=10.2.0.1:1:X0

I have a pretty generic Sagan rule to be triggered: 

alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"[SONICWALL] IPS Detection Alert"; content: "IPS Detection Alert"; classtype: icmp-event; normalize: sonicwall; reference: url,wiki.softwink.com/bin/view/Main/5001084; sid: 5001084; rev:1;)

(Note the "normalize: sonicwall" flag).

The liblognorm rule looks like thus (from my sonicwall.rulebase). 

--<snip>----

prefix=id=%firewall:word% sn=%serial:word% time="%date:word% %hour:number%:%minute:number%:%seconds:number%" fw=%fire-ip:ipv4% pri=%pri:number% c=%c:number% m=%m:number%

rule=:  msg="IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number% ipscat=%proto:word% ipspri=%ipspri:number% n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%interface:word% dst=%dst-ip:ipv4%:%dst-port:number%:%interface:word%

--<snip>----

	Actually,  for the "IPS Detection Alert",  I had to write multiple liblognorm
rules.  This it might be "IPS Detection: ICMP PING" or "IPS Detection: ICMP PING 
reply" (ie - multiple fields).  Over all,  I'm okay with this.   However,  I did 
run into some interesting issues: 

1. Note the "time="%date:word% %hour:number%:%minute:number%:%seconds:number%"
   in the prefix.  There probably needs to be a parser for date (2011-01-13)
   format.  There probably also needs to be a parser for the 24 clock 
   (10:05:14).   Doing something like "%hour:word% doesn't work,  so I'm
   breaking up the fields manually.  I'm assuming this is caused by the
   :'s. 

2. Check out the "msg=" portion.  If you look closely/compare the "real"
   output with the lognorm rule,  it's missing a ".   For example,  the
   log line is actually: 

   msg="IPS Detection Alert: ICMP Echo Reply" sid=316 .....

   But the rule is: 

   IPS Detection Alert: %t1:word% %t2:word %t3:word% sid=%sid:number%

   The last %t3:word% captures not only the item in the string but the
   " as well.   Not sure if that's a "issue",  but thought I'd mention
   it. 

   Lastly,  on a side note,  I've had to alter my rsyslog.conf to the
following: 

$template sagan, "%fromhost-ip%|%syslogfacility-text%|%syslogpriority-text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-rfc3339%|%timegenerated:12:19:date-rfc3339%|%programname%|%syslogtag%%msg%\n"

	Note the end of the line.  For some reason,  syslog-ng likes to 
prepend the 'program[pid]' as part of the message.  I researched this,  and
I don't see any way to remove it via syslog-ng,  so my "fix" was to 
have rsyslog do the same.  It's annoying,  but I have a feeling that bringing
it up with the syslog-ng team will just lead to bickering about what "should"
be part of the %msg% (or $MSG) and what shouldn't be.  

	Also,  without the CEE definitions,  I'm obviously "making up" field
names as I go.   If you have any pointers to "real" CEE definitions of fields,
that would help.  If I recall,   they haven't been finalized/released yet (?)


	Anyways,  that's all.. Let me know what you thing. 


-- 
        Champ Clark III | Softwink, Inc | 800-538-9357 x 101
                     http://www.softwink.com

GPG Key ID: 58A2A58F
Key fingerprint = 7734 2A1C 007D 581E BDF7  6AD5 0F1F 655F 58A2 A58F
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20110113/b6779009/attachment.pgp>