From jlay at slave-tothe-box.net Fri Oct 28 20:24:16 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 28 Oct 2011 12:24:16 -0600 Subject: [Lognorm] Matching problem Message-ID: <992c63f111d8d04e903bc4c93df63237.squirrel@127.0.0.1> Hey all! So..I've posted this on the Sagan mailling list, but I was told here might be good as well since it's related to liblognorm. Here's the scoop below: I've tried various version of the below...I just can't seem to get a match on this. I've tried making almost everything that can have a possible variable reflect it. as well as moving stuff into prefix= just to see what happens...it's just not matching. Any one have any ideas? Thanks. James raw syslog entry: Oct 28 11:13:48 gateway kernel: [110475.092235] New,invalid IN=ppp0 OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 normalize rule: prefix=[%garbage:number%.%garbage:number%] New,invalid IN=%int:word% OUT= MAC= rule=: SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS= %tos:number% PREC=%prec:word% TTL=%ttl:number% ID=%garbage:number% DF PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% WINDOW= %garbage:number% RES=%res:word% SYN URGP=%ugrp:number% normalize debug: [*] Normalize output: [cee at 115 originalmsg="[110475.092235\] New \,invalid IN=ppp0 OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 " unparsed-data="SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 "] From cclark at quadrantsec.com Mon Oct 31 00:36:47 2011 From: cclark at quadrantsec.com (Champ Clark III [Quadrant]) Date: Sun, 30 Oct 2011 19:36:47 -0400 Subject: [Lognorm] Matching problem In-Reply-To: <992c63f111d8d04e903bc4c93df63237.squirrel@127.0.0.1> References: <992c63f111d8d04e903bc4c93df63237.squirrel@127.0.0.1> Message-ID: <24B79877-827E-4DC1-B6A5-8067F31D7CA1@quadrantsec.com> Hello Liblognorm list! I've looked over this problem. I've posted about it, and think I have it working. Heres the post (scroll to the bottom) http://groups.google.com/group/sagan-users/browse_thread/thread/57f771ecbbe7984b Normalization rules are at: https://github.com/beave/sagan-rules/blob/master/linux-kernel-normalize.rulebase On Oct 28, 2011, at 2:24 PM, James Lay wrote: > Hey all! > > So..I've posted this on the Sagan mailling list, but I was told here might > be good as well since it's related to liblognorm. Here's the scoop below: > > I've tried various version of the below...I just can't seem to get a match > on this. I've tried making almost everything that can have a possible > variable reflect it. as well as moving stuff into prefix= just to see what > happens...it's just not matching. Any one have any ideas? Thanks. > > James > > raw syslog entry: > Oct 28 11:13:48 gateway kernel: [110475.092235] New,invalid IN=ppp0 > OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 > TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 RES=0x00 > SYN URGP=0 > > normalize rule: > prefix=[%garbage:number%.%garbage:number%] New,invalid IN=%int:word% > OUT= MAC= > rule=: SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS= > %tos:number% PREC=%prec:word% TTL=%ttl:number% ID=%garbage:number% DF > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% WINDOW= > %garbage:number% RES=%res:word% SYN URGP=%ugrp:number% > > normalize debug: > [*] Normalize output: [cee at 115 originalmsg="[110475.092235\] New > \,invalid IN=ppp0 OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 > TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 > WINDOW=14600 RES=0x00 SYN URGP=0 " unparsed-data="SRC=70.56.158.130 > DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP > SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 "] > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Champ Clark III (office) 904.253.7856 (mobile) 850.443.2440 (SOC) 800.538.9357 ext 101 cclark at quadrantsec.com www.quadrantsec.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quadrant.png Type: image/png Size: 17273 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Mon Oct 31 10:34:01 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 31 Oct 2011 10:34:01 +0100 Subject: [Lognorm] Matching problem In-Reply-To: <24B79877-827E-4DC1-B6A5-8067F31D7CA1@quadrantsec.com> References: <992c63f111d8d04e903bc4c93df63237.squirrel@127.0.0.1> <24B79877-827E-4DC1-B6A5-8067F31D7CA1@quadrantsec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72813F2@GRFEXC.intern.adiscon.com> I'll go through it in more detail soon. Just let me add the comment that these things already come with name-value pairs. I wonder if it would make sense to have special rules that just say (in a sense): "grab these already existing N/V pairs and be done with them". That of course has the subtle problem that there is little text less to prevent false positives. rainer > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant] > Sent: Monday, October 31, 2011 12:37 AM > To: lognorm > Subject: Re: [Lognorm] Matching problem > > Hello Liblognorm list! > > I've looked over this problem. I've posted about it, and think I have > it working. Heres the post (scroll to the bottom) > > http://groups.google.com/group/sagan- > users/browse_thread/thread/57f771ecbbe7984b > > Normalization rules are at: > > https://github.com/beave/sagan-rules/blob/master/linux-kernel- > normalize.rulebase > > > > On Oct 28, 2011, at 2:24 PM, James Lay wrote: > > > Hey all! > > So..I've posted this on the Sagan mailling list, but I was told > here might > be good as well since it's related to liblognorm. Here's the > scoop below: > > I've tried various version of the below...I just can't seem to > get a match > on this. I've tried making almost everything that can have a > possible > variable reflect it. as well as moving stuff into prefix= just to > see what > happens...it's just not matching. Any one have any ideas? > Thanks. > > James > > raw syslog entry: > Oct 28 11:13:48 gateway kernel: [110475.092235] New,invalid > IN=ppp0 > OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 > PREC=0x00 > TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 > RES=0x00 > SYN URGP=0 > > normalize rule: > prefix=[%garbage:number%.%garbage:number%] New,invalid > IN=%int:word% > OUT= MAC= > rule=: SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS= > %tos:number% PREC=%prec:word% TTL=%ttl:number% > ID=%garbage:number% DF > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% > WINDOW= > %garbage:number% RES=%res:word% SYN URGP=%ugrp:number% > > normalize debug: > [*] Normalize output: [cee at 115 originalmsg="[110475.092235\] New > \,invalid IN=ppp0 OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip > LEN=60 > TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 > WINDOW=14600 RES=0x00 SYN URGP=0 " unparsed- > data="SRC=70.56.158.130 > DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF > PROTO=TCP > SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 "] > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > > > > > > Champ Clark III > (office) 904.253.7856 > > (mobile) 850.443.2440 > (SOC) 800.538.9357 ext 101 > cclark at quadrantsec.com > www.quadrantsec.com From jlay at slave-tothe-box.net Mon Oct 31 16:57:29 2011 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 31 Oct 2011 09:57:29 -0600 Subject: [Lognorm] Matching problem In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72813F2@GRFEXC.intern.adiscon.com> References: <992c63f111d8d04e903bc4c93df63237.squirrel@127.0.0.1> <24B79877-827E-4DC1-B6A5-8067F31D7CA1@quadrantsec.com> <9B6E2A8877C38245BFB15CC491A11DA72813F2@GRFEXC.intern.adiscon.com> Message-ID: Hi Rainer, Neat app first off. I'm not a programmer...just user :) So...as I've been looking at this particular issue...would it not make sense to...revamp liblognorm to only match what we want? If we're only extracting src/dst ip and src/dst port, couldn't we have something that goes from this: rule=: %unusedtime:word% IN=%in:word% OUT=%out:word% MAC=%mac:word% SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% WINDOW=%window:number% RES=%res:word% %pkt-type:word% URGP=%urgp:number% to rule=: SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% SPT=%src-port:number% DPT=%dst-port:number% My kernel.rule file has already identified this previously has a stream to send to normalize yes? Perhaps even change the above so a code is given to begin parsing data: rule=: the <> could be the start/stop maybe. Just some thoughts....having to match EVERY single item just to get four pieces of data seems a bit much. Thanks again....I will continue hammering this out to see if I can get it to match. James > I'll go through it in more detail soon. Just let me add the comment that > these things already come with name-value pairs. I wonder if it would make > sense to have special rules that just say (in a sense): "grab these > already > existing N/V pairs and be done with them". That of course has the subtle > problem that there is little text less to prevent false positives. > > rainer >> -----Original Message----- >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant] >> Sent: Monday, October 31, 2011 12:37 AM >> To: lognorm >> Subject: Re: [Lognorm] Matching problem >> >> Hello Liblognorm list! >> >> I've looked over this problem. I've posted about it, and think I have >> it working. Heres the post (scroll to the bottom) >> >> http://groups.google.com/group/sagan- >> users/browse_thread/thread/57f771ecbbe7984b >> >> Normalization rules are at: >> >> https://github.com/beave/sagan-rules/blob/master/linux-kernel- >> normalize.rulebase >> >> >> >> On Oct 28, 2011, at 2:24 PM, James Lay wrote: >> >> >> Hey all! >> >> So..I've posted this on the Sagan mailling list, but I was told >> here might >> be good as well since it's related to liblognorm. Here's the >> scoop below: >> >> I've tried various version of the below...I just can't seem to >> get a match >> on this. I've tried making almost everything that can have a >> possible >> variable reflect it. as well as moving stuff into prefix= just to >> see what >> happens...it's just not matching. Any one have any ideas? >> Thanks. >> >> James >> >> raw syslog entry: >> Oct 28 11:13:48 gateway kernel: [110475.092235] New,invalid >> IN=ppp0 >> OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip LEN=60 TOS=0x10 >> PREC=0x00 >> TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 WINDOW=14600 >> RES=0x00 >> SYN URGP=0 >> >> normalize rule: >> prefix=[%garbage:number%.%garbage:number%] New,invalid >> IN=%int:word% >> OUT= MAC= >> rule=: SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS= >> %tos:number% PREC=%prec:word% TTL=%ttl:number% >> ID=%garbage:number% DF >> PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% >> WINDOW= >> %garbage:number% RES=%res:word% SYN URGP=%ugrp:number% >> >> normalize debug: >> [*] Normalize output: [cee at 115 originalmsg="[110475.092235\] New >> \,invalid IN=ppp0 OUT= MAC= SRC=70.56.158.130 DST=my_ext_ip >> LEN=60 >> TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF PROTO=TCP SPT=59786 DPT=23 >> WINDOW=14600 RES=0x00 SYN URGP=0 " unparsed- >> data="SRC=70.56.158.130 >> DST=my_ext_ip LEN=60 TOS=0x10 PREC=0x00 TTL=60 ID=55693 DF >> PROTO=TCP >> SPT=59786 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 "] >> >> _______________________________________________ >> Lognorm mailing list >> Lognorm at lists.adiscon.com >> http://lists.adiscon.net/mailman/listinfo/lognorm >> >> >> >> >> >> Champ Clark III >> (office) 904.253.7856 >> >> (mobile) 850.443.2440 >> (SOC) 800.538.9357 ext 101 >> cclark at quadrantsec.com >> www.quadrantsec.com > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm >