[Lognorm] Libnormalize issue

[cclark]

On Wed, 2 Nov 2011 13:11:34 -0600, James Lay wrote:
>>
>> The rule does not match, because "(Unhandled..." is not matching the
>> sample.
>> So it did not extract any fields at all.
>>
>> I'll elaborate a bit later why we need to have perfect matches. 
>> Think
>> about
>> false positives...
>>
>> rainer
>
> Thanks for responding so quickly.  As I look at my test setup, I see 
> that
> you are spot on...if it doesn't match the WHOLE thing, nothing gets
> parsed.  That leaves me with two options as it relates to the below
> examples:

Technically,  it matches _up to_ the "unparsed" information.  This will 
give you a "hint"
about where the rule went wrong.

>
> IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00
> PREC=0x00 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 
> RES=0x00
> SYN URGP=0
>
> IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00
> PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384
> RES=0x00 SYN URGP=0
>
> Easy to miss, but the DF there is where I have an issue...some have 
> it,
> and some don't.  Without a regex to ignore junk (LEN=.*DF), then what 
> are
> my options?  I can create 2 different rules, one to match the above 
> with a
> %DF:word%, and one without, but now I have two seperate entries for 
> pretty
> much the same info...not optimal.
>
> I'm guessing that my questions and comments are from my ignorance of 
> how
> this all works.  From my dealings so far with Sagan, it looks like my 
> rule
> file should match first, then send to normlize yes?  I would think 
> that
> would reduce false positives since my rule has already done the job 
> of
> matching, and liblognorm's job is to parse out the specific 
> info..yes?
> Again, maybe I'm TOTALLY missing something.

If the first normalize rule doesn't match,  it'll move on to the second 
rule.  That is,
_right_ when liblognorm "see's" it's not going to match,  it's already 
moving on to the next rule.
If you have pcre/regexp,  it would then have to pump that data via 
libpcre.  That'd create
more overhead than you think.   Hence the reason I encourage users (for 
Sagan/Snort rules)
to use "content:" over "pcre:".  Because pcre adds extra CPU overhead.

I'm sure Rainer can explain better,  and I know this has come up on the 
list before,
but adding regexp/libprec to the mix will actually make it more complex 
and less efficient.
Efficiency is the name of the game here.

I actually think of liblognorm as more of a "mask" than a rule.  That 
is,  if my
log is:

Invalid connection from bob at 192.168.0.1

My "mask" or "rule" become...

Invalid connection from %username:word% at %src-ip:ipv4%

Nice and straight forward.

> I'll continue to test this out...my goal is corelate snort entires 
> with
> firewall rules, but so far it's been an uphill battle.  Again, thanks 
> for
> any light you can shed, and for taking your time to make liblognorm.
>

I've created some bridge related iptables/netfilter rules just fine.  
It actually wasn't
that hard (but I'm use to liblognorm by now).  When I have a chance,  
I'll look at
creating more netfilter rules when I get home (on site for work ATM).

Also,  things like openssh/bind/etc normalization should still work 
fine.  Sounds like
to me it's just your iptables/netfilter information not getting 
normalized.   You should be
getting some correlatable information.