From jlay at slave-tothe-box.net Wed Nov 2 16:19:42 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 2 Nov 2011 09:19:42 -0600 Subject: [Lognorm] Libnormalize issue Message-ID: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> Cross posted from sagan: Well...here's the normalize rule: rule=: Deny %iface1:word% %iface2:word% %unused1:number% %proto:word% %unused2:number% %unused3:number% %src-ip:ipv4% %dst-ip:ipv4% %src- prt:number% %dst-prt:number% offset %unused4:number% %unused5:number% %unused6:number% win %unused7:number% and from debug: [*] Normalize output: [cee at 115 originalmsg=" Deny 0-Integra Firebox 78 udp 20 128 ext_ip ext_ip 137 137 (Unhandled External Packet-00)" unparsed-data=" (Unhandled External Packet-00)"] All entries in the db show the IP of the database host (not the same as the system running sagan) as the src/dst, and the port as 514. It's like normalize just isn't parsing anything at all. Any hints where I could trouble shoot this? Thank you. James From rgerhards at hq.adiscon.com Wed Nov 2 19:01:12 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Nov 2011 19:01:12 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of James Lay > Sent: Wednesday, November 02, 2011 4:20 PM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] Libnormalize issue > > Cross posted from sagan: > > Well...here's the normalize rule: > rule=: Deny %iface1:word% %iface2:word% %unused1:number% %proto:word% > %unused2:number% %unused3:number% %src-ip:ipv4% %dst-ip:ipv4% %src- > prt:number% %dst-prt:number% offset %unused4:number% %unused5:number% > %unused6:number% win %unused7:number% > > and from debug: > [*] Normalize output: [cee at 115 originalmsg=" Deny 0-Integra Firebox 78 > udp 20 128 ext_ip ext_ip 137 137 (Unhandled External Packet-00)" > unparsed-data=" (Unhandled External Packet-00)"] > > All entries in the db show the IP of the database host (not the same > as the system running sagan) as the src/dst, and the port as 514. > It's like normalize just isn't parsing anything at all. Any hints > where I could trouble shoot this? The rule does not match, because "(Unhandled..." is not matching the sample. So it did not extract any fields at all. I'll elaborate a bit later why we need to have perfect matches. Think about false positives... rainer > > Thank you. > > James > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From jlay at slave-tothe-box.net Wed Nov 2 20:11:34 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 2 Nov 2011 13:11:34 -0600 Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> Message-ID: <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> > > The rule does not match, because "(Unhandled..." is not matching the > sample. > So it did not extract any fields at all. > > I'll elaborate a bit later why we need to have perfect matches. Think > about > false positives... > > rainer Thanks for responding so quickly. As I look at my test setup, I see that you are spot on...if it doesn't match the WHOLE thing, nothing gets parsed. That leaves me with two options as it relates to the below examples: IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 PREC=0x00 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 Easy to miss, but the DF there is where I have an issue...some have it, and some don't. Without a regex to ignore junk (LEN=.*DF), then what are my options? I can create 2 different rules, one to match the above with a %DF:word%, and one without, but now I have two seperate entries for pretty much the same info...not optimal. I'm guessing that my questions and comments are from my ignorance of how this all works. From my dealings so far with Sagan, it looks like my rule file should match first, then send to normlize yes? I would think that would reduce false positives since my rule has already done the job of matching, and liblognorm's job is to parse out the specific info..yes? Again, maybe I'm TOTALLY missing something. I'll continue to test this out...my goal is corelate snort entires with firewall rules, but so far it's been an uphill battle. Again, thanks for any light you can shed, and for taking your time to make liblognorm. James From cclark at quadrantsec.com Thu Nov 3 01:18:27 2011 From: cclark at quadrantsec.com (cclark) Date: Wed, 02 Nov 2011 20:18:27 -0400 Subject: [Lognorm] Libnormalize issue In-Reply-To: <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> Message-ID: <6309835fd2df4a5144c4cca29906f3fd@quadrantsec.com> On Wed, 2 Nov 2011 13:11:34 -0600, James Lay wrote: >> >> The rule does not match, because "(Unhandled..." is not matching the >> sample. >> So it did not extract any fields at all. >> >> I'll elaborate a bit later why we need to have perfect matches. >> Think >> about >> false positives... >> >> rainer > > Thanks for responding so quickly. As I look at my test setup, I see > that > you are spot on...if it doesn't match the WHOLE thing, nothing gets > parsed. That leaves me with two options as it relates to the below > examples: Technically, it matches _up to_ the "unparsed" information. This will give you a "hint" about where the rule went wrong. > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > PREC=0x00 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 > RES=0x00 > SYN URGP=0 > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 > RES=0x00 SYN URGP=0 > > Easy to miss, but the DF there is where I have an issue...some have > it, > and some don't. Without a regex to ignore junk (LEN=.*DF), then what > are > my options? I can create 2 different rules, one to match the above > with a > %DF:word%, and one without, but now I have two seperate entries for > pretty > much the same info...not optimal. > > I'm guessing that my questions and comments are from my ignorance of > how > this all works. From my dealings so far with Sagan, it looks like my > rule > file should match first, then send to normlize yes? I would think > that > would reduce false positives since my rule has already done the job > of > matching, and liblognorm's job is to parse out the specific > info..yes? > Again, maybe I'm TOTALLY missing something. If the first normalize rule doesn't match, it'll move on to the second rule. That is, _right_ when liblognorm "see's" it's not going to match, it's already moving on to the next rule. If you have pcre/regexp, it would then have to pump that data via libpcre. That'd create more overhead than you think. Hence the reason I encourage users (for Sagan/Snort rules) to use "content:" over "pcre:". Because pcre adds extra CPU overhead. I'm sure Rainer can explain better, and I know this has come up on the list before, but adding regexp/libprec to the mix will actually make it more complex and less efficient. Efficiency is the name of the game here. I actually think of liblognorm as more of a "mask" than a rule. That is, if my log is: Invalid connection from bob at 192.168.0.1 My "mask" or "rule" become... Invalid connection from %username:word% at %src-ip:ipv4% Nice and straight forward. > I'll continue to test this out...my goal is corelate snort entires > with > firewall rules, but so far it's been an uphill battle. Again, thanks > for > any light you can shed, and for taking your time to make liblognorm. > I've created some bridge related iptables/netfilter rules just fine. It actually wasn't that hard (but I'm use to liblognorm by now). When I have a chance, I'll look at creating more netfilter rules when I get home (on site for work ATM). Also, things like openssh/bind/etc normalization should still work fine. Sounds like to me it's just your iptables/netfilter information not getting normalized. You should be getting some correlatable information. From david at lang.hm Thu Nov 3 03:05:19 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 2 Nov 2011 19:05:19 -0700 (PDT) Subject: [Lognorm] Libnormalize issue In-Reply-To: <6309835fd2df4a5144c4cca29906f3fd@quadrantsec.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> <6309835fd2df4a5144c4cca29906f3fd@quadrantsec.com> Message-ID: On Wed, 2 Nov 2011, cclark wrote: > On Wed, 2 Nov 2011 13:11:34 -0600, James Lay wrote: >> I'm guessing that my questions and comments are from my ignorance of how >> this all works. From my dealings so far with Sagan, it looks like my rule >> file should match first, then send to normlize yes? I would think that >> would reduce false positives since my rule has already done the job of >> matching, and liblognorm's job is to parse out the specific info..yes? >> Again, maybe I'm TOTALLY missing something. > > If the first normalize rule doesn't match, it'll move on to the second > rule. That is, _right_ when liblognorm "see's" it's not going to match, > it's already moving on to the next rule. If you have pcre/regexp, it > would then have to pump that data via libpcre. That'd create more > overhead than you think. Hence the reason I encourage users (for > Sagan/Snort rules) to use "content:" over "pcre:". Because pcre adds > extra CPU overhead. > > I'm sure Rainer can explain better, and I know this has come up on the > list before, but adding regexp/libprec to the mix will actually make it > more complex and less efficient. Efficiency is the name of the game > here. > > I actually think of liblognorm as more of a "mask" than a rule. That > is, if my log is: The thing to remember is that liblognorm is creating a parse tree, not a set of regex rules to match. So it's not evaluating the rules one at a time as each line arrives. Instead it's evaluating them all at the same time. It's essentially creating a mini program where it looks at the first character of the input and says 'this character means that it could match this set of rules, but not this other set', then it looks at the next character and says 'of the rules that were possible after the last step, this set is still possible' and repeats this until there is only one rule left in the 'possible' set. Then it goes through that rule to assign values to variables. This process makes it so that it takes very close to the same amount of time to evaluate a large number of rules as a small number of rules. David Lang From cclark at quadrantsec.com Thu Nov 3 07:09:37 2011 From: cclark at quadrantsec.com (Champ Clark III [Quadrant]) Date: Thu, 3 Nov 2011 02:09:37 -0400 Subject: [Lognorm] Libnormalize issue In-Reply-To: References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> <6309835fd2df4a5144c4cca29906f3fd@quadrantsec.com> Message-ID: <8AB53EC2-D74A-4D9E-820B-56A350183310@quadrantsec.com> > > The thing to remember is that liblognorm is creating a parse tree, not a set of regex rules to match. > > So it's not evaluating the rules one at a time as each line arrives. > > Instead it's evaluating them all at the same time. > > It's essentially creating a mini program where it looks at the first character of the input and says 'this character means that it could match this set of rules, but not this other set', then it looks at the next character and says 'of the rules that were possible after the last step, this set is still possible' and repeats this until there is only one rule left in the 'possible' set. Then it goes through that rule to assign values to variables. > > This process makes it so that it takes very close to the same amount of time to evaluate a large number of rules as a small number of rules. David, Thanks for the response. Sagan has been using liblognorm for a while, but you've summed up how it works very well. Actually, it even helped me! I think I'll archive your response for future Sagan/liblognorm questions that come up on our mailing list. Keep it up, awesome work.... Champ Clark III (office) 904.253.7856 (mobile) 850.443.2440 (SOC) 800.538.9357 ext 101 cclark at quadrantsec.com www.quadrantsec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quadrant.png Type: image/png Size: 17273 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Nov 3 08:40:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Nov 2011 08:40:51 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com><6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1><6309835fd2df4a5144c4cca29906f3fd@quadrantsec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728141C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 03, 2011 3:05 AM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > On Wed, 2 Nov 2011, cclark wrote: > > > On Wed, 2 Nov 2011 13:11:34 -0600, James Lay wrote: > >> I'm guessing that my questions and comments are from my ignorance of > how > >> this all works. From my dealings so far with Sagan, it looks like > my rule > >> file should match first, then send to normlize yes? I would think > that > >> would reduce false positives since my rule has already done the job > of > >> matching, and liblognorm's job is to parse out the specific > info..yes? > >> Again, maybe I'm TOTALLY missing something. > > > > If the first normalize rule doesn't match, it'll move on to the > second > > rule. That is, _right_ when liblognorm "see's" it's not going to > match, > > it's already moving on to the next rule. If you have pcre/regexp, it > > would then have to pump that data via libpcre. That'd create more > > overhead than you think. Hence the reason I encourage users (for > > Sagan/Snort rules) to use "content:" over "pcre:". Because pcre adds > > extra CPU overhead. > > > > I'm sure Rainer can explain better, and I know this has come up on > the > > list before, but adding regexp/libprec to the mix will actually make > it > > more complex and less efficient. Efficiency is the name of the game > > here. > > > > I actually think of liblognorm as more of a "mask" than a rule. That > > is, if my log is: > > The thing to remember is that liblognorm is creating a parse tree, not > a > set of regex rules to match. > > So it's not evaluating the rules one at a time as each line arrives. > > Instead it's evaluating them all at the same time. > > It's essentially creating a mini program where it looks at the first > character of the input and says 'this character means that it could > match > this set of rules, but not this other set', then it looks at the next > character and says 'of the rules that were possible after the last > step, > this set is still possible' and repeats this until there is only one > rule > left in the 'possible' set. Then it goes through that rule to assign > values to variables. > > This process makes it so that it takes very close to the same amount of > time to evaluate a large number of rules as a small number of rules. That's an excellent description, David actually said it better than I could ;) rainer From rgerhards at hq.adiscon.com Thu Nov 3 09:04:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Nov 2011 09:04:51 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of James Lay > Sent: Wednesday, November 02, 2011 8:12 PM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > > > > The rule does not match, because "(Unhandled..." is not matching the > > sample. > > So it did not extract any fields at all. > > > > I'll elaborate a bit later why we need to have perfect matches. Think > > about > > false positives... > > > > rainer > > Thanks for responding so quickly. As I look at my test setup, I see > that > you are spot on...if it doesn't match the WHOLE thing, nothing gets > parsed. That leaves me with two options as it relates to the below > examples: > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > PREC=0x00 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 > RES=0x00 > SYN URGP=0 > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 > RES=0x00 SYN URGP=0 > > Easy to miss, but the DF there is where I have an issue...some have it, > and some don't. Without a regex to ignore junk (LEN=.*DF), then what > are > my options? I can create 2 different rules, one to match the above > with a > %DF:word%, and one without, but now I have two seperate entries for > pretty > much the same info...not optimal. The problem here is that liblognorm primarily aims at semi-structured data, that is text data without an easily parsable structure. Iptables actually provides structured data and liblognorm is not great at processing that kind of data. It becomes even worse if there are any permutations in field order. In that case, you need exponentionally many rules in the worst case. I was thinking about adding a special name/value parsing capability to support that type of data. But then it is vitally important that the data has a header that clearly identifies the message, otherwise normalization will result in a big mess of garbage. Because the chance that such a very generic parser mis-interprets things is very high, especially in the uptables case as a single word (like "df" above) is a valid (binary) "name/value-pair", so it is hard to detect during parsing if that really is iptables or not. Even if we assume it is: the parser consumes probably a lot of data before it detects a mismatch. So we need to backtrack over a lot of data. In essence, one such rule could probably double the processing speed of all rules. And if you have 10 such rules, you could come up with a 1024-times slower rule parsing in the worst case (that's the problem that bugs the usual regex approach and severely limits extraction speed). So iptables actually presents a pretty hard problem. I'd still like to tackle it, but unfortunately I am short on time at the moment. In any case, normalization is still up on my agenda, so probably one of the first things to look at when there is time left (CEE has a new draft standard out and I'd like to make the necessary adaptions). Probably a solution is to provide this "iptables" normalizer, maybe even as a different api call, and the controlling application must first select the normalize to use based on other information. > > I'm guessing that my questions and comments are from my ignorance of > how > this all works. From my dealings so far with Sagan, it looks like my > rule > file should match first, then send to normlize yes? I would think that > would reduce false positives since my rule has already done the job of > matching, and liblognorm's job is to parse out the specific info..yes? > Again, maybe I'm TOTALLY missing something. I don't now how it is implemented in Sagan. But if you do that, you'll loose all of liblognorm's performance benefits (which it gains from doing all in a *single* pass). Note: I am not saying what you intend to do is bad for your context, I am just saying how it relates to liblognorm. rainer > > I'll continue to test this out...my goal is corelate snort entires with > firewall rules, but so far it's been an uphill battle. Again, thanks > for > any light you can shed, and for taking your time to make liblognorm. > > James > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From rgerhards at hq.adiscon.com Thu Nov 3 11:42:52 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Nov 2011 11:42:52 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com><6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728142A@GRFEXC.intern.adiscon.com> I have just uploaded an overview paper that may help to provide context for this discussion (though admittedly not touching it precisely): http://www.gerhards.net/download/LogNormalizationV2.pdf HTH Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, November 03, 2011 9:05 AM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of James Lay > > Sent: Wednesday, November 02, 2011 8:12 PM > > To: lognorm > > Subject: Re: [Lognorm] Libnormalize issue > > > > > > > > The rule does not match, because "(Unhandled..." is not matching the > > > sample. > > > So it did not extract any fields at all. > > > > > > I'll elaborate a bit later why we need to have perfect matches. > > > Think about false positives... > > > > > > rainer > > > > Thanks for responding so quickly. As I look at my test setup, I see > > that you are spot on...if it doesn't match the WHOLE thing, nothing > > gets parsed. That leaves me with two options as it relates to the > > below > > examples: > > > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > > PREC=0x00 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 > WINDOW=16384 > > RES=0x00 > > SYN URGP=0 > > > > IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 > > PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 > WINDOW=16384 > > RES=0x00 SYN URGP=0 > > > > Easy to miss, but the DF there is where I have an issue...some have > > it, and some don't. Without a regex to ignore junk (LEN=.*DF), then > > what are my options? I can create 2 different rules, one to match the > > above with a %DF:word%, and one without, but now I have two seperate > > entries for pretty much the same info...not optimal. > > The problem here is that liblognorm primarily aims at semi-structured data, > that is text data without an easily parsable structure. Iptables actually > provides structured data and liblognorm is not great at processing that kind > of data. It becomes even worse if there are any permutations in field order. > In that case, you need exponentionally many rules in the worst case. > > I was thinking about adding a special name/value parsing capability to > support that type of data. But then it is vitally important that the data has a > header that clearly identifies the message, otherwise normalization will > result in a big mess of garbage. Because the chance that such a very generic > parser mis-interprets things is very high, especially in the uptables case as a > single word (like "df" above) is a valid (binary) "name/value-pair", so it is hard > to detect during parsing if that really is iptables or not. Even if we assume it is: > the parser consumes probably a lot of data before it detects a mismatch. So > we need to backtrack over a lot of data. In essence, one such rule could > probably double the processing speed of all rules. And if you have > 10 such rules, you could come up with a 1024-times slower rule parsing in the > worst case (that's the problem that bugs the usual regex approach and > severely limits extraction speed). > > So iptables actually presents a pretty hard problem. I'd still like to tackle it, > but unfortunately I am short on time at the moment. In any case, > normalization is still up on my agenda, so probably one of the first things to > look at when there is time left (CEE has a new draft standard out and I'd like > to make the necessary adaptions). Probably a solution is to provide this > "iptables" normalizer, maybe even as a different api call, and the controlling > application must first select the normalize to use based on other information. > > > > > I'm guessing that my questions and comments are from my ignorance of > > how this all works. From my dealings so far with Sagan, it looks like > > my rule file should match first, then send to normlize yes? I would > > think that would reduce false positives since my rule has already done > > the job of matching, and liblognorm's job is to parse out the specific > > info..yes? > > Again, maybe I'm TOTALLY missing something. > > I don't now how it is implemented in Sagan. But if you do that, you'll loose all > of liblognorm's performance benefits (which it gains from doing all in a > *single* pass). Note: I am not saying what you intend to do is bad for your > context, I am just saying how it relates to liblognorm. > > rainer > > > > I'll continue to test this out...my goal is corelate snort entires > > with firewall rules, but so far it's been an uphill battle. Again, > > thanks for any light you can shed, and for taking your time to make > > liblognorm. > > > > James > > > > _______________________________________________ > > Lognorm mailing list > > Lognorm at lists.adiscon.com > > http://lists.adiscon.net/mailman/listinfo/lognorm > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From jlay at slave-tothe-box.net Thu Nov 3 12:11:38 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 03 Nov 2011 05:11:38 -0600 Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728142A@GRFEXC.intern.adiscon.com> Message-ID: On 11/3/11 4:42 AM, "Rainer Gerhards" wrote: >I have just uploaded an overview paper that may help to provide context >for >this discussion (though admittedly not touching it precisely): > >http://www.gerhards.net/download/LogNormalizationV2.pdf > >HTH >Rainer Wow...thanks Rainer, Champ and David...this is a LOT of great information for me to use. Again, I'm only using liblognorm as it relates to Sagan, so I'll get with Champ and ask a few more questions about how Sagan and liblognorm relate. As a final thought to think about..as it relates to false positives and the like, whose job is it to verify....the app sending the data, or the library parsing the data? Food for thought. I'll post some results in a different thread soon...already working with more types of data and rules...firewall rules come first, then others later :) Thanks again all. James From rgerhards at hq.adiscon.com Thu Nov 3 12:17:27 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Nov 2011 12:17:27 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA728142A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728142B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of James Lay > Sent: Thursday, November 03, 2011 12:12 PM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > > > On 11/3/11 4:42 AM, "Rainer Gerhards" wrote: > > >I have just uploaded an overview paper that may help to provide > >context for this discussion (though admittedly not touching it > >precisely): > > > >http://www.gerhards.net/download/LogNormalizationV2.pdf > > > >HTH > >Rainer > > Wow...thanks Rainer, Champ and David...this is a LOT of great information for > me to use. Again, I'm only using liblognorm as it relates to Sagan, so I'll get > with Champ and ask a few more questions about how Sagan and liblognorm > relate. As a final thought to think about..as it relates to false positives and > the like, whose job is it to verify....the app sending the data, or the library > parsing the data? Well, the problem is that a false positive is a false positive because the library interpreted it in a way that is present inside the rule base, but not in the way the user thought it would. As the thought-reader is still under development [;-)], the lib has no way of knowing it did not meet user expectations. However, we try to reduce the chance for false positives by using more precise parsers before trying more generic parsers. At least that is the plan. I am not sure right now if that is 100% implemented. That somewhat mitigates the iptables-parser dilemma, but it still exists. rainer Food for thought. I'll post some results in a different > thread soon...already working with more types of data and rules...firewall > rules come first, then others later :) Thanks again all. > > James > > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From david at lang.hm Thu Nov 3 18:19:50 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Nov 2011 10:19:50 -0700 (PDT) Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 3 Nov 2011, Rainer Gerhards wrote: > The problem here is that liblognorm primarily aims at semi-structured data, > that is text data without an easily parsable structure. Iptables actually > provides structured data and liblognorm is not great at processing that kind > of data. It becomes even worse if there are any permutations in field order. > In that case, you need exponentionally many rules in the worst case. > > I was thinking about adding a special name/value parsing capability to > support that type of data. But then it is vitally important that the data has > a header that clearly identifies the message, otherwise normalization will > result in a big mess of garbage. Because the chance that such a very generic > parser mis-interprets things is very high, especially in the uptables case as > a single word (like "df" above) is a valid (binary) "name/value-pair", so it > is hard to detect during parsing if that really is iptables or not. Even if > we assume it is: the parser consumes probably a lot of data before it detects > a mismatch. So we need to backtrack over a lot of data. In essence, one such > rule could probably double the processing speed of all rules. And if you have > 10 such rules, you could come up with a 1024-times slower rule parsing in the > worst case (that's the problem that bugs the usual regex approach and > severely limits extraction speed). how about adding a couple of new tag types 1. name=value pair 2. one or more name=value pairs then you could make a rule that would match the fixed part of a log and then let the log specify the rest of it > > So iptables actually presents a pretty hard problem. I'd still like to tackle > it, but unfortunately I am short on time at the moment. In any case, > normalization is still up on my agenda, so probably one of the first things > to look at when there is time left (CEE has a new draft standard out and I'd > like to make the necessary adaptions). Probably a solution is to provide this > "iptables" normalizer, maybe even as a different api call, and the > controlling application must first select the normalize to use based on other > information. one key thing is that iptables lets you specify a tag for the log messages. by default it is just 'kernel' but if you are wanting to identify them for parsing, you really should set this (and then you can unambigously match them) David Lang From rgerhards at hq.adiscon.com Thu Nov 3 18:24:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Nov 2011 18:24:50 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com><6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728142D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, November 03, 2011 6:20 PM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > On Thu, 3 Nov 2011, Rainer Gerhards wrote: > > > The problem here is that liblognorm primarily aims at semi-structured > > data, that is text data without an easily parsable structure. Iptables > > actually provides structured data and liblognorm is not great at > > processing that kind of data. It becomes even worse if there are any > permutations in field order. > > In that case, you need exponentionally many rules in the worst case. > > > > I was thinking about adding a special name/value parsing capability to > > support that type of data. But then it is vitally important that the > > data has a header that clearly identifies the message, otherwise > > normalization will result in a big mess of garbage. Because the chance > > that such a very generic parser mis-interprets things is very high, > > especially in the uptables case as a single word (like "df" above) is > > a valid (binary) "name/value-pair", so it is hard to detect during > > parsing if that really is iptables or not. Even if we assume it is: > > the parser consumes probably a lot of data before it detects a > > mismatch. So we need to backtrack over a lot of data. In essence, one > > such rule could probably double the processing speed of all rules. And > > if you have > > 10 such rules, you could come up with a 1024-times slower rule parsing > > in the worst case (that's the problem that bugs the usual regex > > approach and severely limits extraction speed). > > how about adding a couple of new tag types > > 1. name=value pair > > 2. one or more name=value pairs > > then you could make a rule that would match the fixed part of a log and then > let the log specify the rest of it That's (especially 2) what I am thinking about. > > > > > So iptables actually presents a pretty hard problem. I'd still like to > > tackle it, but unfortunately I am short on time at the moment. In any > > case, normalization is still up on my agenda, so probably one of the > > first things to look at when there is time left (CEE has a new draft > > standard out and I'd like to make the necessary adaptions). Probably a > > solution is to provide this "iptables" normalizer, maybe even as a > > different api call, and the controlling application must first select > > the normalize to use based on other information. > > one key thing is that iptables lets you specify a tag for the log messages. by > default it is just 'kernel' but if you are wanting to identify them for parsing, > you really should set this (and then you can unambigously match them) Yup, but doesn't help against something malicious (but granted, that's also the case for other things, but not tot hat extent as a no-match would occur in most cases). rainer > > David Lang > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From rgerhards at hq.adiscon.com Mon Nov 7 18:31:05 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Nov 2011 18:31:05 +0100 Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728142D@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com><6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA728142D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728143A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, November 03, 2011 6:25 PM > To: lognorm > Subject: Re: [Lognorm] Libnormalize issue > > > -----Original Message----- > > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, November 03, 2011 6:20 PM > > To: lognorm > > Subject: Re: [Lognorm] Libnormalize issue > > > > On Thu, 3 Nov 2011, Rainer Gerhards wrote: > > > > > The problem here is that liblognorm primarily aims at > > > semi-structured data, that is text data without an easily parsable > > > structure. Iptables actually provides structured data and liblognorm > > > is not great at processing that kind of data. It becomes even worse > > > if there are any > > permutations in field order. > > > In that case, you need exponentionally many rules in the worst case. > > > > > > I was thinking about adding a special name/value parsing capability > > > to support that type of data. But then it is vitally important that > > > the data has a header that clearly identifies the message, otherwise > > > normalization will result in a big mess of garbage. Because the > > > chance that such a very generic parser mis-interprets things is very > > > high, especially in the uptables case as a single word (like "df" > > > above) is a valid (binary) "name/value-pair", so it is hard to > > > detect during parsing if that really is iptables or not. Even if we assume it > is: > > > the parser consumes probably a lot of data before it detects a > > > mismatch. So we need to backtrack over a lot of data. In essence, > > > one such rule could probably double the processing speed of all > > > rules. And if you have > > > 10 such rules, you could come up with a 1024-times slower rule > > > parsing in the worst case (that's the problem that bugs the usual > > > regex approach and severely limits extraction speed). > > > > how about adding a couple of new tag types > > > > 1. name=value pair > > > > 2. one or more name=value pairs > > > > then you could make a rule that would match the fixed part of a log > > and > then > > let the log specify the rest of it > > That's (especially 2) what I am thinking about. I have added some experimental code to liblognorm to handle this case. The code is currently available via git, only. This rule: rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: %dummy:iptables% used with this message: Apr 8 13:58:26 host.example.net iptables: IN=ppp0 OUT= MAC= SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 PREC=0x00 TTL=108 ID=256 DF PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 Leads to this format (json-formatted): '{"IN": "ppp0", "OUT": "", "MAC": "", "SRC": "121.11.80.101", "DST": "my_ext_ip", "LEN": "40", "TOS": "0x00", "PREC": "0x00", "TTL": "108", "ID": "256", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "6000", "DPT": "1433", "WINDOW": "16384", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP": "0", "tag": "iptables", "host": "host.example.net", "date": "Apr 8 13:58:26"}' Note that things like DF show up with value "[*PRESENT*]". The code currently does not check malformdness of the iptables part. Most probably the code will segfault if something is malformed. I have not been able to conduct broader tests, especially as part of a larger rule base. I'd deeply appreciate if someone (Champ?) could try out the new code in a real-world setting. I'd expect that it would considerably reduce the effort required to handle iptables logs inside a semi-structured log stream. Just make sure that you assign a unique tag as suggested by David for iptables, else recognition will be a mess. Feedback deeply appreciated. Rainer From jlay at slave-tothe-box.net Mon Nov 7 18:40:55 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 7 Nov 2011 10:40:55 -0700 Subject: [Lognorm] Libnormalize issue In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728143A@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com><6e447a869df6f1f9ec538a7e45ff1eb4.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA728141E@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA728142D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA728143A@GRFEXC.intern.adiscon.com> Message-ID: <5060f676c14db80b7bbe14550354b7ee.squirrel@127.0.0.1> > > >> -----Original Message----- >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Thursday, November 03, 2011 6:25 PM >> To: lognorm >> Subject: Re: [Lognorm] Libnormalize issue >> >> > -----Original Message----- >> > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> > Sent: Thursday, November 03, 2011 6:20 PM >> > To: lognorm >> > Subject: Re: [Lognorm] Libnormalize issue >> > >> > On Thu, 3 Nov 2011, Rainer Gerhards wrote: >> > >> > > The problem here is that liblognorm primarily aims at >> > > semi-structured data, that is text data without an easily parsable >> > > structure. Iptables actually provides structured data and liblognorm >> > > is not great at processing that kind of data. It becomes even worse >> > > if there are any >> > permutations in field order. >> > > In that case, you need exponentionally many rules in the worst case. >> > > >> > > I was thinking about adding a special name/value parsing capability >> > > to support that type of data. But then it is vitally important that >> > > the data has a header that clearly identifies the message, otherwise >> > > normalization will result in a big mess of garbage. Because the >> > > chance that such a very generic parser mis-interprets things is very >> > > high, especially in the uptables case as a single word (like "df" >> > > above) is a valid (binary) "name/value-pair", so it is hard to >> > > detect during parsing if that really is iptables or not. Even if we > assume it >> is: >> > > the parser consumes probably a lot of data before it detects a >> > > mismatch. So we need to backtrack over a lot of data. In essence, >> > > one such rule could probably double the processing speed of all >> > > rules. And if you have >> > > 10 such rules, you could come up with a 1024-times slower rule >> > > parsing in the worst case (that's the problem that bugs the usual >> > > regex approach and severely limits extraction speed). >> > >> > how about adding a couple of new tag types >> > >> > 1. name=value pair >> > >> > 2. one or more name=value pairs >> > >> > then you could make a rule that would match the fixed part of a log >> > and >> then >> > let the log specify the rest of it >> >> That's (especially 2) what I am thinking about. > > I have added some experimental code to liblognorm to handle this case. The > code is currently available via git, only. > > This rule: > rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: %dummy:iptables% > > used with this message: > Apr 8 13:58:26 host.example.net iptables: IN=ppp0 OUT= MAC= > SRC=121.11.80.101 DST=my_ext_ip LEN=40 TOS=0x00 PREC=0x00 TTL=108 ID=256 > DF > PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 > > Leads to this format (json-formatted): > '{"IN": "ppp0", "OUT": "", "MAC": "", "SRC": "121.11.80.101", "DST": > "my_ext_ip", "LEN": "40", "TOS": "0x00", "PREC": "0x00", "TTL": "108", > "ID": > "256", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "6000", "DPT": "1433", > "WINDOW": "16384", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP": "0", > "tag": > "iptables", "host": "host.example.net", "date": "Apr 8 13:58:26"}' > > Note that things like DF show up with value "[*PRESENT*]". > > The code currently does not check malformdness of the iptables part. Most > probably the code will segfault if something is malformed. I have not been > able to conduct broader tests, especially as part of a larger rule base. > I'd > deeply appreciate if someone (Champ?) could try out the new code in a > real-world setting. I'd expect that it would considerably reduce the > effort > required to handle iptables logs inside a semi-structured log stream. Just > make sure that you assign a unique tag as suggested by David for iptables, > else recognition will be a mess. > > Feedback deeply appreciated. > Rainer > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > Count me in Rainer...I'll get this rockin at the home machine either today or tomorrow and see the impact. I'll let you know my findings..thanks for all your hard work on this! James From cclark at quadrantsec.com Mon Nov 7 22:35:35 2011 From: cclark at quadrantsec.com (Champ Clark III [Quadrant Information Security]) Date: Mon, 7 Nov 2011 16:35:35 -0500 Subject: [Lognorm] Issues with :'s in fields (?) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1> <9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> Message-ID: <20111107213535.GA15911@a> Howdy all, I'm noticing a strange issue when I'm having to deal with Sonicwall logs. Here's the run down. The log lines look like this: id=firewall sn=001111111111 time="2011-11-07 21:23:04 UTC" fw=192.168.1.1 pri=3 c=4 m=14 msg="Web site access denied" n=0 src=10.110.14.117:54426:X0: dst=216.163.137.68:80:X1: dstname=www.playboy.com arg=/ code=4 Category="Pornography" I'm trying to parse the src-ip/dst-ip & src-port/dst-port. Those are the most important to me. This is the rule I'm using (no prefix as of yet): rule=: id=%id:word% sn=%sn:word% time="%time:char-to:\x22%" fw=%fw:ipv4% pri=%pri:number% c=%c:number% m=%m:number% msg="%msg:char-to:\x22%" n=%n:number% src=%src-ip:ipv4%:%src-port:number%:%inteface:word%: dst=%dst-ip:ipv4%:%interface:word%: dstname=%website:word% arg=%arg:word% code=%code:number% Category="%cat:char-to:\x22%" When I run it, with liblognorm debugging, I get: [*] Normalize output: [cee at 115 originalmsg=" id=firewall sn=001111111111 time=\"2011-11-07 21:22:24 UTC\" fw=192.168.1.1 pri=3 c=4 m=14 msg=\"Web site access denied\" n=0 src=10.110.14.117:24423:X0: dst=216.163.137.68:80:X1: dstname=www.playboy.com arg=/ code=4 Category=\"Pornography\" " unparsed-data=" dst=216.163.137.68:80:X1: dstname=www.playboy.com arg=/ code=4 Category=\"Pornography\" "] No matter how I move around the rule, the "dst=" always comes back unparsed. I'll look at it more tonight, but my thought is that the :'s fields might be messing things up. What do you think? Thanks. Oh - Rainer, I'll test out that new liblognorm feature ASAP. -- Champ Clark III | Quadrant Information Security | 904-253-7856 http://www.quadrantsec.com GPG Key ID: 0B30A6A7 Key fingerprint = A154 17D5 F16D 8C09 69FA 618B 3877 B04C 0B30 A6A7 If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Tue Nov 8 08:40:23 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Nov 2011 08:40:23 +0100 Subject: [Lognorm] Issues with :'s in fields (?) In-Reply-To: <20111107213535.GA15911@a> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <20111107213535.GA15911@a> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728143C@GRFEXC.intern.adiscon.com> Champ, I'll try to have a look later today. But as a general suggestion, it may be useful to try such things out with the "normalize" program from liblognorm. You can feed it the rule base (or a single rule ;)) and the actual message. If you call normalize with -vv, you'll get a verbose output that tells you how the actual matching process takes place. As a side-note, I hope I will have some time to look at the normalizing tools over the next couple of weeks. Maybe not that many hours per day, but hopefully a bit every day (or every other ;)). rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant > Information Security] > Sent: Monday, November 07, 2011 10:36 PM > To: lognorm > Subject: [Lognorm] Issues with :'s in fields (?) > > > Howdy all, > > I'm noticing a strange issue when I'm having to deal with > Sonicwall logs. Here's the run down. > > The log lines look like this: > > id=firewall sn=001111111111 time="2011-11-07 21:23:04 UTC" > fw=192.168.1.1 pri=3 c=4 m=14 msg="Web site access denied" n=0 > src=10.110.14.117:54426:X0: dst=216.163.137.68:80:X1: > dstname=www.playboy.com arg=/ code=4 Category="Pornography" > > I'm trying to parse the src-ip/dst-ip & src-port/dst-port. Those > are > the most important to me. This is the rule I'm using (no prefix as of > yet): > > rule=: id=%id:word% sn=%sn:word% time="%time:char-to:\x22%" > fw=%fw:ipv4% pri=%pri:number% c=%c:number% m=%m:number% msg="%msg:char- > to:\x22%" n=%n:number% src=%src-ip:ipv4%:%src- > port:number%:%inteface:word%: dst=%dst-ip:ipv4%:%interface:word%: > dstname=%website:word% arg=%arg:word% code=%code:number% > Category="%cat:char-to:\x22%" > > When I run it, with liblognorm debugging, I get: > > [*] Normalize output: [cee at 115 originalmsg=" id=firewall > sn=001111111111 time=\"2011-11-07 21:22:24 UTC\" fw=192.168.1.1 pri=3 > c=4 m=14 msg=\"Web site access denied\" n=0 src=10.110.14.117:24423:X0: > dst=216.163.137.68:80:X1: dstname=www.playboy.com arg=/ code=4 > Category=\"Pornography\" " unparsed-data=" dst=216.163.137.68:80:X1: > dstname=www.playboy.com arg=/ code=4 Category=\"Pornography\" "] > > No matter how I move around the rule, the "dst=" always comes > back unparsed. I'll look at it more tonight, but my thought is that > the :'s fields might be messing things up. What do you think? Thanks. > > Oh - Rainer, I'll test out that new liblognorm feature ASAP. > > -- > > Champ Clark III | Quadrant Information Security | 904-253-7856 > http://www.quadrantsec.com > > > GPG Key ID: 0B30A6A7 > Key fingerprint = A154 17D5 F16D 8C09 69FA 618B 3877 B04C 0B30 A6A7 > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From cclark at quadrantsec.com Tue Nov 8 16:13:10 2011 From: cclark at quadrantsec.com (Champ Clark III [Quadrant]) Date: Tue, 8 Nov 2011 10:13:10 -0500 Subject: [Lognorm] Issues with :'s in fields (?) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728143C@GRFEXC.intern.adiscon.com> References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <20111107213535.GA15911@a> <9B6E2A8877C38245BFB15CC491A11DA728143C@GRFEXC.intern.adiscon.com> Message-ID: Ryan, Just for fun, I added in the IDS sensors as well as Sagan into the Quadrant / Sagan interface. This will give you access to what our IDS sensor is seeing, and what your Sonicwall firewalls are seeing. On Nov 8, 2011, at 2:40 AM, Rainer Gerhards wrote: > Champ, > > I'll try to have a look later today. But as a general suggestion, it may be > useful to try such things out with the "normalize" program from liblognorm. > You can feed it the rule base (or a single rule ;)) and the actual message. > If you call normalize with -vv, you'll get a verbose output that tells you > how the actual matching process takes place. > > As a side-note, I hope I will have some time to look at the normalizing tools > over the next couple of weeks. Maybe not that many hours per day, but > hopefully a bit every day (or every other ;)). > > rainer > >> -----Original Message----- >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant >> Information Security] >> Sent: Monday, November 07, 2011 10:36 PM >> To: lognorm >> Subject: [Lognorm] Issues with :'s in fields (?) >> >> >> Howdy all, >> >> I'm noticing a strange issue when I'm having to deal with >> Sonicwall logs. Here's the run down. >> >> The log lines look like this: >> >> id=firewall sn=001111111111 time="2011-11-07 21:23:04 UTC" >> fw=192.168.1.1 pri=3 c=4 m=14 msg="Web site access denied" n=0 >> src=10.110.14.117:54426:X0: dst=216.163.137.68:80:X1: >> dstname=www.playboy.com arg=/ code=4 Category="Pornography" >> >> I'm trying to parse the src-ip/dst-ip & src-port/dst-port. Those >> are >> the most important to me. This is the rule I'm using (no prefix as of >> yet): >> >> rule=: id=%id:word% sn=%sn:word% time="%time:char-to:\x22%" >> fw=%fw:ipv4% pri=%pri:number% c=%c:number% m=%m:number% msg="%msg:char- >> to:\x22%" n=%n:number% src=%src-ip:ipv4%:%src- >> port:number%:%inteface:word%: dst=%dst-ip:ipv4%:%interface:word%: >> dstname=%website:word% arg=%arg:word% code=%code:number% >> Category="%cat:char-to:\x22%" >> >> When I run it, with liblognorm debugging, I get: >> >> [*] Normalize output: [cee at 115 originalmsg=" id=firewall >> sn=001111111111 time=\"2011-11-07 21:22:24 UTC\" fw=192.168.1.1 pri=3 >> c=4 m=14 msg=\"Web site access denied\" n=0 src=10.110.14.117:24423:X0: >> dst=216.163.137.68:80:X1: dstname=www.playboy.com arg=/ code=4 >> Category=\"Pornography\" " unparsed-data=" dst=216.163.137.68:80:X1: >> dstname=www.playboy.com arg=/ code=4 Category=\"Pornography\" "] >> >> No matter how I move around the rule, the "dst=" always comes >> back unparsed. I'll look at it more tonight, but my thought is that >> the :'s fields might be messing things up. What do you think? Thanks. >> >> Oh - Rainer, I'll test out that new liblognorm feature ASAP. >> >> -- >> >> Champ Clark III | Quadrant Information Security | 904-253-7856 >> http://www.quadrantsec.com >> >> >> GPG Key ID: 0B30A6A7 >> Key fingerprint = A154 17D5 F16D 8C09 69FA 618B 3877 B04C 0B30 A6A7 >> If it wasn't for C, we'd be using BASI, PASAL and OBOL. > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Champ Clark III (office) 904.253.7856 (mobile) 850.443.2440 (SOC) 800.538.9357 ext 101 cclark at quadrantsec.com www.quadrantsec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quadrant.png Type: image/png Size: 17273 bytes Desc: not available URL: From cclark at quadrantsec.com Tue Nov 8 16:34:02 2011 From: cclark at quadrantsec.com (cclark) Date: Tue, 08 Nov 2011 11:34:02 -0400 Subject: [Lognorm] =?utf-8?q?Issues_with_=3A=27s_in_fields_=28=3F=29?= In-Reply-To: References: <662575ac28ea0190f013679347c66f75.squirrel@127.0.0.1><9B6E2A8877C38245BFB15CC491A11DA7281417@GRFEXC.intern.adiscon.com> <20111107213535.GA15911@a> <9B6E2A8877C38245BFB15CC491A11DA728143C@GRFEXC.intern.adiscon.com> Message-ID: <938112a8177af667189d1d9445a60ce5@quadrantsec.com> Sorry that wasn't meant for the list. On Tue, 8 Nov 2011 10:13:10 -0500, Champ Clark III [Quadrant] wrote: > Ryan, > Just for fun, I added in the IDS sensors as well as Sagan into the > Quadrant / Sagan interface. This will give you access to what > our IDS sensor is seeing, and what your Sonicwall firewalls are > seeing. > > On Nov 8, 2011, at 2:40 AM, Rainer Gerhards wrote: > >> Champ, >> >> I'll try to have a look later today. But as a general suggestion, it >> may be >> useful to try such things out with the "normalize" program from >> liblognorm. >> You can feed it the rule base (or a single rule ;)) and the actual >> message. >> If you call normalize with -vv, you'll get a verbose output that >> tells you >> how the actual matching process takes place. >> >> As a side-note, I hope I will have some time to look at the >> normalizing tools >> over the next couple of weeks. Maybe not that many hours per day, >> but >> hopefully a bit every day (or every other ;)). >> >> rainer >> >>> -----Original Message----- >> >>> From: lognorm-bounces at lists.adiscon.com [1] [mailto:lognorm- >> ists.adiscon.com">bounces at lists.adiscon.com] On Behalf Of Champ >>> Clark III [Quadrant >> nformation Security] >> Sent: Monday, N >> >>> M >> te">To: lognorm >> [Lognorm] Issues with :'s in fields (?) >> blockquote type="cite"> >> /blockquote> >> ="cite"> Howdy all, >> ite"> >>> >> le-tab-span" style="white-space:pre"> iss >> aving to deal with >> >>> e type="cite">Sonicwall logs. Here's the run down. >> e="cite"> >> te"> >> look like this: >> >>>> id=firewall sn=001111111111 time="2011-11-07 21:23:04 UTC" >> ite">fw=192.1 >> >>> g="Web site access denied" n=0 >> type="cite">src=10.110.14.117:54426:X0: dst=216.163.137.68:80:X1: >> type >> me=www.playboy.com arg=/ co >> >>> hy" >> r> clas >> span" style="white-space:pre"> I'm tryi >> >>> st-ip & src-port/dst-port. Those >> ="cite">are >> "cite">the most important to me. This is the rule I'm using >> of >> yet >> >>> kquote type="cite"> >> %id:word% sn=%s >> >>> -to:x22%" >>> fw=%fw:ipv4% >> er% c=%c:number% m=%m:number% msg="%msg:char- ote >> o:x22%" n=%n:number% src=%src-ip:ipv4%:%src- >> e="cite">port:number%:%inteface:word%: >>> dst=%dst-ip:ipv4%:%interface:word%: >>> dstname=%website:word% a >> code=%code:number% >> >>> e type= >> ry="%cat:char-to:x22%" >>> >> e"> When I run >> >>> rm debugg >> get: >> >> [*] Normal >> >>> gina >> ewall >>> sn=001111111111 time="2011-11-07 21:22:24 UTC" fw=192.168.1.1 >> lockquote> g="Web site access denied" n=0 >> src=10.110.14.117:24423:X0: >> ype="cit >> >>> :X1: dstname=www.playboy.com arg=/ code=4 >> ckquote type="cite">Category >> >>> d-data=" dst=216.163.137.68:80:X1: >> =www.playboy.com arg=/ code=4 >> >>> "] >>> >>>> >> ype="cite"> 'll >> re tonight, but >> >>> /blockquote>the :'s fields might be messing things up. What do you >>> think? Thanks. >> ckquote type="cite"> te t >> pan class="Apple-tab-span" style="white-space:pre"> l test out that >> new liblognorm feature ASAP. >> type="cite"> >> ="cite">-- >>> >>>> >> ="cite"> Champ Clark III >> >>> ecurity | 904-253-7856 >>> &nb >> p; &nb >> >>> sp; ht >> rantsec.com >> te type="cite"> >>> GPG Key ID: 0B >> lockquote>Key fingerprint = A154 17D5 F16D 8C09 69FA &nbs >> >>> A7 >>> If it wasn't for C, we' >> SI, PASAL and OBOL. >> _______ >> >>> ____ >> br>Lognorm mailing list > > Champ Clark III > (office) 904.253.7856 > > (mobile) 850.443.2440 > (SOC) 800.538.9357 ext 101 > cclark at quadrantsec.com [2] > www.quadrantsec.com > > > > Links: > ------ > [1] mailto:lognorm-bounces at lists.adiscon.com > [2] mailto:cclark at quadrantsec.com From rgerhards at hq.adiscon.com Tue Nov 8 18:10:59 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Nov 2011 18:10:59 +0100 Subject: [Lognorm] new capability: filler fields Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728144F@GRFEXC.intern.adiscon.com> I just added the capability to have an "official" filler field type. Details here: http://blog.gerhards.net/2011/11/filler-fields-in-log-normalization.html rainer From jlay at slave-tothe-box.net Wed Nov 9 01:37:03 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 08 Nov 2011 17:37:03 -0700 Subject: [Lognorm] new capability: filler fields In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728144F@GRFEXC.intern.adiscon.com> Message-ID: On 11/8/11 10:10 AM, "Rainer Gerhards" wrote: >I just added the capability to have an "official" filler field type. >Details >here: > >http://blog.gerhards.net/2011/11/filler-fields-in-log-normalization.html > >rainer >_______________________________________________ >Lognorm mailing list >Lognorm at lists.adiscon.com >http://lists.adiscon.net/mailman/listinfo/lognorm Very cool news...thanks Rainer. I've got the latest git installed and up and running. Regarding the -, would a sample look like say: %-:word% Thanks again...very cool stuff going on! James From rgerhards at hq.adiscon.com Wed Nov 9 08:38:55 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Nov 2011 08:38:55 +0100 Subject: [Lognorm] new capability: filler fields In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA728144F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281451@GRFEXC.intern.adiscon.com> > Very cool news...thanks Rainer. I've got the latest git installed and > up > and running. Regarding the -, would a sample look like say: > > %-:word% Yes, exactly :) rainer From rgerhards at hq.adiscon.com Wed Nov 9 18:47:36 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Nov 2011 18:47:36 +0100 Subject: [Lognorm] Some more news & feedback request Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728145D@GRFEXC.intern.adiscon.com> Hi all, the list becomes a bit busy after being dormant for a while ;) I have thought about how to best implement support for the new version of CEE and think I found a generally useful method. It is described here: http://blog.gerhards.net/2011/11/liblognorm-event-annotation-and-cee.html Feedback appreciated. Rainer From james.lay at wincofoods.com Tue Nov 22 16:31:50 2011 From: james.lay at wincofoods.com (Lay, James) Date: Tue, 22 Nov 2011 08:31:50 -0700 Subject: [Lognorm] Simple tests seem not to work Message-ID: <360E0F1A6850C74D89B37C3A22C9DE1F04DB42E4@GOMAIL.go.winco.local> Hey all! So...been battling trying to get some asa stuff to fly. As I'm testing things, I think I need some help in understanding more on how liblognorm works. Here's the rules below: Normalize-rulebase: alert udp $EXTERNAL_NET any -> $HOME_NET any (msg:"[ASA] TCP EXTERNAL BLOCK"; program: TEST; content: TCP; normalize: asa; classtype: bad-unknown; sid: 6000006; rev:1;) Rule: prefix= rule=: TCP There is a space at the end of the TCP. That being shown, here's what happens when I test this: echo "1.1.1.1|local0|warning|warning|84|2011-11-21|15:03:02|TEST| TCP " > sagan.fifo [*] Normalize output: [cee at 115 originalmsg=" TCP " unparsed-data=""] I've tried: echo "1.1.1.1|local0|warning|warning|84|2011-11-21|15:03:02|TEST| TCP" > sagan.fifo echo "1.1.1.1|local0|warning|warning|84|2011-11-21|15:03:02|TEST|TCP" > sagan.fifo echo "1.1.1.1|local0|warning|warning|84|2011-11-21|15:03:02|TEST|TCP " > sagan.fifo None of which work. [*] Normalize output: [cee at 115 originalmsg=" TCP" unparsed-data=""] [*] Normalize output: [cee at 115 originalmsg="TCP" unparsed-data="TCP"] [*] Normalize output: [cee at 115 originalmsg="TCP " unparsed-data="TCP "] My question is, why not, and where is the issue? Why would a simple word like this not match? Even chaning "TCP" in the rulebase to %-:word% gives me the same output. What could I be missing here? Thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: From friedl at hq.adiscon.com Thu Nov 24 17:00:32 2011 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 24 Nov 2011 17:00:32 +0100 Subject: [Lognorm] liblognorm 0.3.2 and libee 0.3.2 released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728150D@GRFEXC.intern.adiscon.com> We have just released liblognorm 0.3.2 and libee 0.3.2. This release includes a new major features. Changes: Liblognorm Version 0.3.2 (rgerhards), 2011-11-21 - added rfc5424 parser (requires libee >= 0.3.2) - added "-" to serve as name for filler fields. Value is extracted, but no field is written - special handling for iptables log via %iptables% parser added (currently experimental pending practical verification) - normalizer tool on its way to a full-blow stand-alone tool - support for annotations added, for the time being see http://blog.gerhards.net/2011/11/log-annotation-with-liblognorm.html Download: http://www.liblognorm.com/files/download/liblognorm-0.3.2.tar.gz Libee Version 0.3.2 (rgerhards), 2011-11-22 - API enhancements: * added capability to enumerate tags inside a tagbucket * added capability to obtain tagbucket for an event --> ee_EventGetTagbucket() * added capability to add a string value to a field in one call --> ee_addStrValueToField() * added ee_getTag(), ee_setTags() - added additional parser * RFC5424Date - potentially problematic API change: in earlier versions, the function ee_TagbucketHasTag() was errornously name ee_BucketHasTag(). This has been corrected, potentially resulting in API incompatibility. We have accepted this, because we are at level 0.x AND know that potentially no current user has ever used this function, but instead the upper-layer equivalents. But if thinks break, you now know why ;) - flat tags are no longer encoded in the CEE encoders as CEE does not support flat tags. However, this can be turned on via context flags, if desired - bugfix: compile problems under Solaris closes: http://bugzilla.adiscon.com/show_bug.cgi?id=253 - bugfix: ee_EventGetTagbucket() always returned -1 (error) Download: http://www.libee.org/files/download/libee-0.3.2.tar.gz As always, feedback is appreciated. Best regards, Florian Riedl From james.lay at wincofoods.com Thu Nov 24 18:42:32 2011 From: james.lay at wincofoods.com (Lay, James) Date: Thu, 24 Nov 2011 10:42:32 -0700 Subject: [Lognorm] liblognorm 0.3.2 and libee 0.3.2 released Message-ID: <360E0F1A6850C74D89B37C3A22C9DE1F04DB4315@GOMAIL.go.winco.local> Excellent! Thanks for the hard work on this. I'll be installing this weekend for testing. Thanks again. James > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-bounces at lists.adiscon.com] On Behalf Of Florian Riedl > Sent: Thursday, November 24, 2011 9:01 AM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] liblognorm 0.3.2 and libee 0.3.2 released > > We have just released liblognorm 0.3.2 and libee 0.3.2. > > This release includes a new major features. > > Changes: > > Liblognorm Version 0.3.2 (rgerhards), 2011-11-21 > > - added rfc5424 parser (requires libee >= 0.3.2) > - added "-" to serve as name for filler fields. Value is extracted, > but no field is written > - special handling for iptables log via %iptables% parser added > (currently experimental pending practical verification) > - normalizer tool on its way to a full-blow stand-alone tool > - support for annotations added, for the time being see > http://blog.gerhards.net/2011/11/log-annotation-with-liblognorm.html > > Download: > http://www.liblognorm.com/files/download/liblognorm-0.3.2.tar.gz > > > Libee Version 0.3.2 (rgerhards), 2011-11-22 > > - API enhancements: > * added capability to enumerate tags inside a tagbucket > * added capability to obtain tagbucket for an event > --> ee_EventGetTagbucket() > * added capability to add a string value to a field in one call > --> ee_addStrValueToField() > * added ee_getTag(), ee_setTags() > - added additional parser > * RFC5424Date > - potentially problematic API change: in earlier versions, the > function ee_TagbucketHasTag() was errornously name ee_BucketHasTag(). > This has been corrected, potentially resulting in API > incompatibility. We have accepted this, because we are at level 0.x > AND know that potentially no current user has ever used this > function, but instead the upper-layer equivalents. But if thinks > break, you now know why ;) > - flat tags are no longer encoded in the CEE encoders as CEE does not > support flat tags. However, this can be turned on via context flags, > if desired > - bugfix: compile problems under Solaris > closes: http://bugzilla.adiscon.com/show_bug.cgi?id=253 > - bugfix: ee_EventGetTagbucket() always returned -1 (error) > > Download: > http://www.libee.org/files/download/libee-0.3.2.tar.gz > > > As always, feedback is appreciated. > > Best regards, > Florian Riedl > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From james.lay at wincofoods.com Thu Nov 24 19:32:09 2011 From: james.lay at wincofoods.com (Lay, James) Date: Thu, 24 Nov 2011 11:32:09 -0700 Subject: [Lognorm] Question on special characters Message-ID: <360E0F1A6850C74D89B37C3A22C9DE1F04DB4318@GOMAIL.go.winco.local> Hey all! So...I think I'm getting down to the bottom of something that I've had an issue with. Here's some tests: Log contents to pass to normalizer, blick.txt: Test one) Rulebase file blick-rulebase: prefix= rule=: Test one) normalizer -r blick-rulebase < blick.txt this works, and returns nothing, since no normalizing was required (as I understand it). Now...if I make the below change to the blick-rulebase file: prefix= rule=: Test %-:word%) normalizer -r blick-rulebase < blick.txt [cee at 115 originalmsg=" Test one)" unparsed-data=" "] Then it looks like something isn't working. If I remove the ")" in both blick.txt and blick-rulebase to reflect: Test one prefix= rule=: Test %-:word% then it works: normalizer -r blick-rulebase < blick.txt [cee at 115 -="one"] This seems to happen with matching %word% within parenthesis. Is there something I can do to check this on my end? Thank you. James -------------- next part -------------- An HTML attachment was scrubbed... URL: From laanwj at gmail.com Thu Nov 24 21:40:33 2011 From: laanwj at gmail.com (Wladimir) Date: Thu, 24 Nov 2011 21:40:33 +0100 Subject: [Lognorm] liblognorm 0.3.2 and libee 0.3.2 released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728150D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA728150D@GRFEXC.intern.adiscon.com> Message-ID: Congrats with the release! On Thu, Nov 24, 2011 at 5:00 PM, Florian Riedl wrote: > We have just released liblognorm 0.3.2 and libee 0.3.2. > > This release includes a new major features. > > Changes: > > Liblognorm Version 0.3.2 (rgerhards), 2011-11-21 > > - added rfc5424 parser (requires libee >= 0.3.2) > - added "-" to serve as name for filler fields. Value is extracted, > but no field is written > - special handling for iptables log via %iptables% parser added > (currently experimental pending practical verification) > - normalizer tool on its way to a full-blow stand-alone tool > - support for annotations added, for the time being see > http://blog.gerhards.net/2011/11/log-annotation-with-liblognorm.html > > Download: > http://www.liblognorm.com/files/download/liblognorm-0.3.2.tar.gz > > > Libee Version 0.3.2 (rgerhards), 2011-11-22 > > - API enhancements: > * added capability to enumerate tags inside a tagbucket > * added capability to obtain tagbucket for an event > --> ee_EventGetTagbucket() > * added capability to add a string value to a field in one call > --> ee_addStrValueToField() > * added ee_getTag(), ee_setTags() > - added additional parser > * RFC5424Date > - potentially problematic API change: in earlier versions, the > function ee_TagbucketHasTag() was errornously name ee_BucketHasTag(). > This has been corrected, potentially resulting in API > incompatibility. We have accepted this, because we are at level 0.x > AND know that potentially no current user has ever used this > function, but instead the upper-layer equivalents. But if thinks > break, you now know why ;) > - flat tags are no longer encoded in the CEE encoders as CEE does not > support flat tags. However, this can be turned on via context flags, > if desired > - bugfix: compile problems under Solaris > closes: http://bugzilla.adiscon.com/show_bug.cgi?id=253 > - bugfix: ee_EventGetTagbucket() always returned -1 (error) > > Download: > http://www.libee.org/files/download/libee-0.3.2.tar.gz > > > As always, feedback is appreciated. > > Best regards, > Florian Riedl > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cclark at quadrantsec.com Fri Nov 25 22:08:15 2011 From: cclark at quadrantsec.com (Champ Clark III [Quadrant]) Date: Fri, 25 Nov 2011 16:08:15 -0500 Subject: [Lognorm] Missing header in git/make install Message-ID: <1CC0956D-E04D-4704-BB31-F4C5B7226486@quadrantsec.com> Doing some testing today with Sagan + new version (via git) of liblognorm. I noticed this: cc -DHAVE_CONFIG_H -I. -I.. -g -O2 -MT sagan.o -MD -MP -MF .deps/sagan.Tpo -c -o sagan.o sagan.c In file included from sagan.c:58: /usr/include/lognorm.h:31:19: error: annot.h: No such file or directory The "make install" doesn't appear to move the annot.h header in place (?). Easy to correct by manually copying. Just FYI. Thank you. Champ Clark III (office) 904.253.7856 (mobile) 850.443.2440 (SOC) 800.538.9357 ext 101 cclark at quadrantsec.com www.quadrantsec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quadrant.png Type: image/png Size: 17273 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Fri Nov 25 22:11:46 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Nov 2011 22:11:46 +0100 Subject: [Lognorm] Missing header in git/make install In-Reply-To: <1CC0956D-E04D-4704-BB31-F4C5B7226486@quadrantsec.com> References: <1CC0956D-E04D-4704-BB31-F4C5B7226486@quadrantsec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728152E@GRFEXC.intern.adiscon.com> Ummm.. looks like I forgot the makefile again :-( Will look at this and other open questions on Monday or Tuesday. As you know, things have been very busy and I have left open quite some things that are urgent at least for me ;) Rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant] > Sent: Friday, November 25, 2011 10:08 PM > To: lognorm > Subject: [Lognorm] Missing header in git/make install > > Doing some testing today with Sagan + new version (via git) of > liblognorm. I noticed this: > > cc -DHAVE_CONFIG_H -I. -I.. -g -O2 -MT sagan.o -MD -MP -MF > .deps/sagan.Tpo -c -o sagan.o sagan.c > In file included from sagan.c:58: > /usr/include/lognorm.h:31:19: error: annot.h: No such file or directory > > The "make install" doesn't appear to move the annot.h header in place > (?). Easy to correct by manually copying. Just FYI. Thank you. > > > > Champ Clark III > (office) 904.253.7856 > > (mobile) 850.443.2440 > (SOC) 800.538.9357 ext 101 > cclark at quadrantsec.com > www.quadrantsec.com