From cclark at quadrantsec.com Thu Apr 5 05:12:31 2012 From: cclark at quadrantsec.com (Champ Clark III) Date: Wed, 04 Apr 2012 23:12:31 -0400 Subject: [Lognorm] Hello! Liblognorn stuff [no problems, more general question] Message-ID: <4F7D0D9F.3030705@quadrantsec.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Rainer, I hope things are going well for you. I just thought I'd let you know what I'm up to. Tomorrow I'm releasing Sagan 0.2.1, which should be a lot of fun :) With future versions of Sagan, I've ended up running into a strange situation. Currently, the only time liblognorm is called is after a event meets certain criteria. For example -> rule is triggered -> liblognorm is called -> event it normalized. That's been working great. My strange situation is the need for "pre-processors". For example, let's say an event (log line) comes in. It may _not_ have triggered a rule, but it might have some characteristics I want to know about. For this example, let say I'm interested in GeoIP information. Basically, this means that liblognorm has to be call _before_ it hits the "rule set" section of code. This means _all_ data must pass through liblognorm. As a quick test, I moved my log normalization code right behind where it reads data in from the FIFO. This was done on a fairly busy machine running Sagan (about 3k event per/sec) just to get an idea of how much of a performance hit. To my surprise, there wasn't as much of a "hit" as i'd expected. So first off, kudo's for the efficient code! My two questions are: 1. For this pre-processor idea to work, I'll need to do a lot more normalization. As the normalization files "grow", what sort of performance impact do you think liblognorm will have? 2. Have you come up with a means (repository/format/etc) that normalization rules should go by? I ask this as I might be writing a _lot_ more normalization rules! My thought (right now) is that if pre-processors are not being used, then normalization can go to the "back of the line" (how Sagan currently work). If pre-processors are being used, then I'll push it up. That way it gives the user an option depending on the load. I hope this makes sense, and thanks again! - -- - - Champ Clark III (cclark at quadrantsec.com) Quadrant Information Security (http://quadrantsec.com) Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A GPG Key ID: 0381878A -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJPfQ2fAAoJENnmXt7Lmc3Kw0kH/3NQ7WSWzN0BNfm8RVcZjbJw NYhPt3nhY/DhfsK/j//I7J+iOc4YxJIkgfx9O+4YyQxxnkylvQUBwA2xGuoWG/0K PCanvPovg6wcVAkmBhPSAQRgE1PbQxwEoBUOhXQ2EzOgpYKXLwXdh45cFb1HI2DR 8dFVRGn7snajU+TPL/bcBib2diJ5+ZGoW+yzBXBv2TEb/rG3aTFb2/BMjzoLUE3V 0ybVR8xvwKvykQUbxdUaVhcZONvbfO/j1frCnv5oLg9ViI3D12XrJtgbmJRRaokk komxtUwgfcUEGzHC/u2sjaxemxnmq8vZYa/PSbUfO0H+PVdrUDU5IqOAK00PVS8= =11Fw -----END PGP SIGNATURE----- From friedl at hq.adiscon.com Mon Apr 16 17:18:01 2012 From: friedl at hq.adiscon.com (Florian Riedl) Date: Mon, 16 Apr 2012 17:18:01 +0200 Subject: [Lognorm] liblognorm 0.3.4 released Message-ID: Hi everyone. We have just released liblognorm 0.3.4. This is a bug fixing release, targeting a single bug that prevented building on many platforms. Changes: Version 0.3.4 (rgerhards), 2012-04-16 bugfix: normalizer tool had a memory leak Thanks to Brian Know for alerting me and helping to debug Download: http://www.liblognorm.com/download/liblognorm-0-3-4/ As always, feedback is appreciated. Best regards, Florian Riedl