[Lognorm] Hello! Liblognorn stuff [no problems, more general question]

[Champ Clark III]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello Rainer,

I hope things are going well for you.   I just thought I'd let you
know what I'm up to.  Tomorrow I'm releasing Sagan 0.2.1,  which
should be a lot of fun :)

With future versions of Sagan, I've ended up running into a strange
situation.  Currently,  the only time liblognorm is called is after a
event meets certain criteria.   For example -> rule is triggered ->
liblognorm is called -> event it normalized.

That's been working great.  My strange situation is the need for
"pre-processors".  For example,  let's say an event (log line) comes
in.  It may _not_ have triggered a rule,  but it might have some
characteristics I want to know about.  For this example,  let say I'm
interested in GeoIP information.   Basically,  this means that
liblognorm has to be call _before_ it hits the "rule set" section of
code.

This means _all_ data must pass through liblognorm.

As a quick test,  I moved my log normalization code right behind where
it reads data in from the FIFO.  This was done on a fairly busy
machine running Sagan (about 3k event per/sec) just to get an idea of
how much of a performance hit.  To my surprise,  there wasn't as much
of a "hit" as i'd expected.   So first off,  kudo's for the efficient
code!

My two questions are:

1.  For this pre-processor idea to work,  I'll need to do a lot more
normalization.  As the normalization files "grow",  what sort of
performance impact do you think liblognorm will have?

2. Have you come up with a means (repository/format/etc) that
normalization rules should go by?  I ask this as I might be writing a
_lot_ more normalization rules!

My thought (right now) is that if pre-processors are not being used,
then normalization can go to the "back of the line" (how Sagan
currently work).  If pre-processors are being used,  then I'll push it
up.  That way it gives the user an option depending on the load.

I hope this makes sense,  and thanks again!


- -- 
- - Champ Clark III (cclark at quadrantsec.com)
  Quadrant Information Security (http://quadrantsec.com)
  Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A
  GPG Key ID: 0381878A
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPfQ2fAAoJENnmXt7Lmc3Kw0kH/3NQ7WSWzN0BNfm8RVcZjbJw
NYhPt3nhY/DhfsK/j//I7J+iOc4YxJIkgfx9O+4YyQxxnkylvQUBwA2xGuoWG/0K
PCanvPovg6wcVAkmBhPSAQRgE1PbQxwEoBUOhXQ2EzOgpYKXLwXdh45cFb1HI2DR
8dFVRGn7snajU+TPL/bcBib2diJ5+ZGoW+yzBXBv2TEb/rG3aTFb2/BMjzoLUE3V
0ybVR8xvwKvykQUbxdUaVhcZONvbfO/j1frCnv5oLg9ViI3D12XrJtgbmJRRaokk
komxtUwgfcUEGzHC/u2sjaxemxnmq8vZYa/PSbUfO0H+PVdrUDU5IqOAK00PVS8=
=11Fw
-----END PGP SIGNATURE-----