From jlay at slave-tothe-box.net Wed Jul 4 16:10:25 2012 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 4 Jul 2012 08:10:25 -0600 Subject: [Lognorm] Special characters revisited In-Reply-To: <827E2018-77A9-4E3D-8514-798893ED1605@slave-tothe-box.net> References: <09e54deeee391553381e6c7f8dfb4f7f@192.168.1.253> <827E2018-77A9-4E3D-8514-798893ED1605@slave-tothe-box.net> Message-ID: <2447E5E6-C891-4FC9-BC1A-C42C3AE49479@slave-tothe-box.net> On Jun 13, 2012, at 7:11 AM, James Lay wrote: > On May 25, 2012, at 12:44 PM, James Lay wrote: >> On Fri, 25 May 2012 12:33:15 -0600, James Lay wrote: >>> Hey all! >>> >>> So...here's where I'm at. Here's my rulebase: >>> >>> prefix= >>> rule=: %-:word% request discarded from >>> %src-ip:ipv4%/%src-port:number% to >>> %-:word%:%dst-ip:ipv4%/%dst-port:number% >>> >>> >>> Sample firewall hit (quotes added for clarity): >>> >>> " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" >>> >>> and my result: >>> >>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>> to Interface:my_ip/443" unparsed-data=""] >>> >>> I've tried the below (thinking the : would be included in word) >>> rule=: %-:word% request discarded from >>> %src-ip:ipv4%/%src-port:number% to >>> %-:word%%dst-ip:ipv4%/%dst-port:number% >>> >>> And I get the same result. >>> >>> I've also tried: >>> rule=: %-:word% request discarded from >>> %src-ip:ipv4%/%src-port:number% to >>> Interface:%dst-ip:ipv4%/%dst-port:number% >>> >>> And this works, so I'm pretty sure it's the : that is my issue. A >>> full -vv below. Any thoughts on how to get this to go? Thanks all: >>> >>> James >>> >>> liblognorm: read sample line: 'prefix=' >>> liblognorm: read sample line: 'rule=: %-:word% request discarded from >>> %src-ip:ipv4%/%src-port:number% to >>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>> liblognorm: sample line to add: ': %-:word% request discarded from >>> %src-ip:ipv4%/%src-port:number% to >>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>> >>> liblognorm: addSampToTree 0 of 108 >>> liblognorm: parsed literal: ' ' >>> liblognorm: buildPTree: begin at 0x9308030, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 1, offs 0 >>> liblognorm: addSampToTree 1 of 108 >>> liblognorm: parsed field: '-' >>> liblognorm: got new subtree 0x9308810 >>> liblognorm: prev subtree 0x9308030 >>> liblognorm: new subtree 0x9308810 >>> liblognorm: addSampToTree 9 of 108 >>> liblognorm: parsed literal: ' request discarded from ' >>> liblognorm: buildPTree: begin at 0x9308810, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 24, offs 0 >>> liblognorm: addSampToTree 33 of 108 >>> liblognorm: parsed field: 'src-ip' >>> liblognorm: got new subtree 0x9308ca0 >>> liblognorm: prev subtree 0x9308810 >>> liblognorm: new subtree 0x9308ca0 >>> liblognorm: addSampToTree 46 of 108 >>> liblognorm: parsed literal: '/' >>> liblognorm: buildPTree: begin at 0x9308ca0, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 1, offs 0 >>> liblognorm: addSampToTree 47 of 108 >>> liblognorm: parsed field: 'src-port' >>> liblognorm: got new subtree 0x9309110 >>> liblognorm: prev subtree 0x9308ca0 >>> liblognorm: new subtree 0x9309110 >>> liblognorm: addSampToTree 64 of 108 >>> liblognorm: parsed literal: ' to ' >>> liblognorm: buildPTree: begin at 0x9309110, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 4, offs 0 >>> liblognorm: addSampToTree 68 of 108 >>> liblognorm: parsed field: '-' >>> liblognorm: got new subtree 0x9309580 >>> liblognorm: prev subtree 0x9309110 >>> liblognorm: new subtree 0x9309580 >>> liblognorm: addSampToTree 76 of 108 >>> liblognorm: parsed literal: ':' >>> liblognorm: buildPTree: begin at 0x9309580, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 1, offs 0 >>> liblognorm: addSampToTree 77 of 108 >>> liblognorm: parsed field: 'dst-ip' >>> liblognorm: got new subtree 0x93099f0 >>> liblognorm: prev subtree 0x9309580 >>> liblognorm: new subtree 0x93099f0 >>> liblognorm: addSampToTree 90 of 108 >>> liblognorm: parsed literal: '/' >>> liblognorm: buildPTree: begin at 0x93099f0, offs 0 >>> liblognorm: case 3.1 >>> liblognorm: addPTree: offs 0 >>> liblognorm: setPrefix lenBuf 1, offs 0 >>> liblognorm: addSampToTree 91 of 108 >>> liblognorm: parsed field: 'dst-port' >>> liblognorm: got new subtree 0x9309e60 >>> liblognorm: prev subtree 0x93099f0 >>> liblognorm: new subtree 0x9309e60 >>> liblognorm: end addSampToTree 108 of 108 >>> number of tree nodes: 7 >>> To normalize: ' TCP request discarded from 96.18.101.250/1077 to >>> Interface:my_ip/443' >>> liblognorm: 0: prefix compare ' ', ' ' >>> liblognorm: 1: prefix compare succeeded, still valid >>> liblognorm: 1:trying parser for field '-': 0xb786c450 >>> liblognorm: potential hit, trying subtree >>> liblognorm: 4: prefix compare ' ', ' ' >>> liblognorm: 5: prefix compare 'r', 'r' >>> liblognorm: 6: prefix compare 'e', 'e' >>> liblognorm: 7: prefix compare 'q', 'q' >>> liblognorm: 8: prefix compare 'u', 'u' >>> liblognorm: 9: prefix compare 'e', 'e' >>> liblognorm: 10: prefix compare 's', 's' >>> liblognorm: 11: prefix compare 't', 't' >>> liblognorm: 12: prefix compare ' ', ' ' >>> liblognorm: 13: prefix compare 'd', 'd' >>> liblognorm: 14: prefix compare 'i', 'i' >>> liblognorm: 15: prefix compare 's', 's' >>> liblognorm: 16: prefix compare 'c', 'c' >>> liblognorm: 17: prefix compare 'a', 'a' >>> liblognorm: 18: prefix compare 'r', 'r' >>> liblognorm: 19: prefix compare 'd', 'd' >>> liblognorm: 20: prefix compare 'e', 'e' >>> liblognorm: 21: prefix compare 'd', 'd' >>> liblognorm: 22: prefix compare ' ', ' ' >>> liblognorm: 23: prefix compare 'f', 'f' >>> liblognorm: 24: prefix compare 'r', 'r' >>> liblognorm: 25: prefix compare 'o', 'o' >>> liblognorm: 26: prefix compare 'm', 'm' >>> liblognorm: 27: prefix compare ' ', ' ' >>> liblognorm: 28: prefix compare succeeded, still valid >>> liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 >>> liblognorm: potential hit, trying subtree >>> liblognorm: 41: prefix compare '/', '/' >>> liblognorm: 42: prefix compare succeeded, still valid >>> liblognorm: 42:trying parser for field 'src-port': 0xb786c510 >>> liblognorm: potential hit, trying subtree >>> liblognorm: 46: prefix compare ' ', ' ' >>> liblognorm: 47: prefix compare 't', 't' >>> liblognorm: 48: prefix compare 'o', 'o' >>> liblognorm: 49: prefix compare ' ', ' ' >>> liblognorm: 50: prefix compare succeeded, still valid >>> liblognorm: 50:trying parser for field '-': 0xb786c450 >>> liblognorm: potential hit, trying subtree >>> liblognorm: 74 returns -1 >>> liblognorm: 50 nonmatch, backtracking required, left=-1 >>> liblognorm: 50 no field, trying subtree char 'I': (nil) >>> liblognorm: 50 returns -1 >>> liblognorm: 42 nonmatch, backtracking required, left=-1 >>> liblognorm: 42 no field, trying subtree char '1': (nil) >>> liblognorm: 42 returns -1 >>> liblognorm: 28 nonmatch, backtracking required, left=-1 >>> liblognorm: 28 no field, trying subtree char '9': (nil) >>> liblognorm: 28 returns -1 >>> liblognorm: 1 nonmatch, backtracking required, left=-1 >>> liblognorm: 1 no field, trying subtree char 'T': (nil) >>> liblognorm: 1 returns -1 >>> liblognorm: final result for normalizer: left -1, endNode 0x9309580 >>> normalized: '[cee at 115 originalmsg=" TCP request discarded from >>> 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' >>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>> to Interface:my_ip/443" unparsed-data=""] >>> >>> _______________________________________________ >>> Lognorm mailing list >>> Lognorm at lists.adiscon.com >>> http://lists.adiscon.net/mailman/listinfo/lognorm >> >> An interesting tidbit...with the below: >> >> >> " TCP request discarded from 96.18.101.250/1077 to 1234:my_ip/443" >> >> and >> >> rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% to %-:number%:%dst-ip:ipv4%/%dst-port:number% >> >> This works and correctly processes. Hope that helps. >> >> James >> >> _______________________________________________ >> Lognorm mailing list >> Lognorm at lists.adiscon.com >> http://lists.adiscon.net/mailman/listinfo/lognorm > > Any movement on this? I'm at an impasse until this is addressed?thank you. > > James > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Would have been nice to at least get some response of some kind on this? James From rgerhards at hq.adiscon.com Wed Jul 4 17:32:16 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Jul 2012 15:32:16 +0000 Subject: [Lognorm] Special characters revisited In-Reply-To: <2447E5E6-C891-4FC9-BC1A-C42C3AE49479@slave-tothe-box.net> References: <09e54deeee391553381e6c7f8dfb4f7f@192.168.1.253> <827E2018-77A9-4E3D-8514-798893ED1605@slave-tothe-box.net> <2447E5E6-C891-4FC9-BC1A-C42C3AE49479@slave-tothe-box.net> Message-ID: <2E7D85FDF00FA545A325B6B2EC73A12C6FA0AA@vmexch2.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of James Lay > Sent: Wednesday, July 04, 2012 4:10 PM > To: lognorm > Subject: Re: [Lognorm] Special characters revisited > > On Jun 13, 2012, at 7:11 AM, James Lay wrote: > > On May 25, 2012, at 12:44 PM, James Lay wrote: > >> On Fri, 25 May 2012 12:33:15 -0600, James Lay wrote: > >>> Hey all! > >>> > >>> So...here's where I'm at. Here's my rulebase: > >>> > >>> prefix= > >>> rule=: %-:word% request discarded from > >>> %src-ip:ipv4%/%src-port:number% to > >>> %-:word%:%dst-ip:ipv4%/%dst-port:number% > >>> > >>> > >>> Sample firewall hit (quotes added for clarity): > >>> > >>> " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" > >>> > >>> and my result: > >>> > >>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 > >>> to Interface:my_ip/443" unparsed-data=""] > >>> > >>> I've tried the below (thinking the : would be included in word) > >>> rule=: %-:word% request discarded from > >>> %src-ip:ipv4%/%src-port:number% to > >>> %-:word%%dst-ip:ipv4%/%dst-port:number% > >>> > >>> And I get the same result. > >>> > >>> I've also tried: > >>> rule=: %-:word% request discarded from > >>> %src-ip:ipv4%/%src-port:number% to > >>> Interface:%dst-ip:ipv4%/%dst-port:number% > >>> > >>> And this works, so I'm pretty sure it's the : that is my issue. A > >>> full -vv below. Any thoughts on how to get this to go? Thanks all: > >>> > >>> James > >>> > >>> liblognorm: read sample line: 'prefix=' > >>> liblognorm: read sample line: 'rule=: %-:word% request discarded from > >>> %src-ip:ipv4%/%src-port:number% to > >>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' > >>> liblognorm: sample line to add: ': %-:word% request discarded from > >>> %src-ip:ipv4%/%src-port:number% to > >>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' > >>> > >>> liblognorm: addSampToTree 0 of 108 > >>> liblognorm: parsed literal: ' ' > >>> liblognorm: buildPTree: begin at 0x9308030, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 1, offs 0 > >>> liblognorm: addSampToTree 1 of 108 > >>> liblognorm: parsed field: '-' > >>> liblognorm: got new subtree 0x9308810 > >>> liblognorm: prev subtree 0x9308030 > >>> liblognorm: new subtree 0x9308810 > >>> liblognorm: addSampToTree 9 of 108 > >>> liblognorm: parsed literal: ' request discarded from ' > >>> liblognorm: buildPTree: begin at 0x9308810, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 24, offs 0 > >>> liblognorm: addSampToTree 33 of 108 > >>> liblognorm: parsed field: 'src-ip' > >>> liblognorm: got new subtree 0x9308ca0 > >>> liblognorm: prev subtree 0x9308810 > >>> liblognorm: new subtree 0x9308ca0 > >>> liblognorm: addSampToTree 46 of 108 > >>> liblognorm: parsed literal: '/' > >>> liblognorm: buildPTree: begin at 0x9308ca0, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 1, offs 0 > >>> liblognorm: addSampToTree 47 of 108 > >>> liblognorm: parsed field: 'src-port' > >>> liblognorm: got new subtree 0x9309110 > >>> liblognorm: prev subtree 0x9308ca0 > >>> liblognorm: new subtree 0x9309110 > >>> liblognorm: addSampToTree 64 of 108 > >>> liblognorm: parsed literal: ' to ' > >>> liblognorm: buildPTree: begin at 0x9309110, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 4, offs 0 > >>> liblognorm: addSampToTree 68 of 108 > >>> liblognorm: parsed field: '-' > >>> liblognorm: got new subtree 0x9309580 > >>> liblognorm: prev subtree 0x9309110 > >>> liblognorm: new subtree 0x9309580 > >>> liblognorm: addSampToTree 76 of 108 > >>> liblognorm: parsed literal: ':' > >>> liblognorm: buildPTree: begin at 0x9309580, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 1, offs 0 > >>> liblognorm: addSampToTree 77 of 108 > >>> liblognorm: parsed field: 'dst-ip' > >>> liblognorm: got new subtree 0x93099f0 > >>> liblognorm: prev subtree 0x9309580 > >>> liblognorm: new subtree 0x93099f0 > >>> liblognorm: addSampToTree 90 of 108 > >>> liblognorm: parsed literal: '/' > >>> liblognorm: buildPTree: begin at 0x93099f0, offs 0 > >>> liblognorm: case 3.1 > >>> liblognorm: addPTree: offs 0 > >>> liblognorm: setPrefix lenBuf 1, offs 0 > >>> liblognorm: addSampToTree 91 of 108 > >>> liblognorm: parsed field: 'dst-port' > >>> liblognorm: got new subtree 0x9309e60 > >>> liblognorm: prev subtree 0x93099f0 > >>> liblognorm: new subtree 0x9309e60 > >>> liblognorm: end addSampToTree 108 of 108 > >>> number of tree nodes: 7 > >>> To normalize: ' TCP request discarded from 96.18.101.250/1077 to > >>> Interface:my_ip/443' > >>> liblognorm: 0: prefix compare ' ', ' ' > >>> liblognorm: 1: prefix compare succeeded, still valid > >>> liblognorm: 1:trying parser for field '-': 0xb786c450 > >>> liblognorm: potential hit, trying subtree > >>> liblognorm: 4: prefix compare ' ', ' ' > >>> liblognorm: 5: prefix compare 'r', 'r' > >>> liblognorm: 6: prefix compare 'e', 'e' > >>> liblognorm: 7: prefix compare 'q', 'q' > >>> liblognorm: 8: prefix compare 'u', 'u' > >>> liblognorm: 9: prefix compare 'e', 'e' > >>> liblognorm: 10: prefix compare 's', 's' > >>> liblognorm: 11: prefix compare 't', 't' > >>> liblognorm: 12: prefix compare ' ', ' ' > >>> liblognorm: 13: prefix compare 'd', 'd' > >>> liblognorm: 14: prefix compare 'i', 'i' > >>> liblognorm: 15: prefix compare 's', 's' > >>> liblognorm: 16: prefix compare 'c', 'c' > >>> liblognorm: 17: prefix compare 'a', 'a' > >>> liblognorm: 18: prefix compare 'r', 'r' > >>> liblognorm: 19: prefix compare 'd', 'd' > >>> liblognorm: 20: prefix compare 'e', 'e' > >>> liblognorm: 21: prefix compare 'd', 'd' > >>> liblognorm: 22: prefix compare ' ', ' ' > >>> liblognorm: 23: prefix compare 'f', 'f' > >>> liblognorm: 24: prefix compare 'r', 'r' > >>> liblognorm: 25: prefix compare 'o', 'o' > >>> liblognorm: 26: prefix compare 'm', 'm' > >>> liblognorm: 27: prefix compare ' ', ' ' > >>> liblognorm: 28: prefix compare succeeded, still valid > >>> liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 > >>> liblognorm: potential hit, trying subtree > >>> liblognorm: 41: prefix compare '/', '/' > >>> liblognorm: 42: prefix compare succeeded, still valid > >>> liblognorm: 42:trying parser for field 'src-port': 0xb786c510 > >>> liblognorm: potential hit, trying subtree > >>> liblognorm: 46: prefix compare ' ', ' ' > >>> liblognorm: 47: prefix compare 't', 't' > >>> liblognorm: 48: prefix compare 'o', 'o' > >>> liblognorm: 49: prefix compare ' ', ' ' > >>> liblognorm: 50: prefix compare succeeded, still valid > >>> liblognorm: 50:trying parser for field '-': 0xb786c450 > >>> liblognorm: potential hit, trying subtree > >>> liblognorm: 74 returns -1 > >>> liblognorm: 50 nonmatch, backtracking required, left=-1 > >>> liblognorm: 50 no field, trying subtree char 'I': (nil) > >>> liblognorm: 50 returns -1 > >>> liblognorm: 42 nonmatch, backtracking required, left=-1 > >>> liblognorm: 42 no field, trying subtree char '1': (nil) > >>> liblognorm: 42 returns -1 > >>> liblognorm: 28 nonmatch, backtracking required, left=-1 > >>> liblognorm: 28 no field, trying subtree char '9': (nil) > >>> liblognorm: 28 returns -1 > >>> liblognorm: 1 nonmatch, backtracking required, left=-1 > >>> liblognorm: 1 no field, trying subtree char 'T': (nil) > >>> liblognorm: 1 returns -1 > >>> liblognorm: final result for normalizer: left -1, endNode 0x9309580 > >>> normalized: '[cee at 115 originalmsg=" TCP request discarded from > >>> 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' > >>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 > >>> to Interface:my_ip/443" unparsed-data=""] > >>> > >>> _______________________________________________ > >>> Lognorm mailing list > >>> Lognorm at lists.adiscon.com > >>> http://lists.adiscon.net/mailman/listinfo/lognorm > >> > >> An interesting tidbit...with the below: > >> > >> > >> " TCP request discarded from 96.18.101.250/1077 to 1234:my_ip/443" > >> > >> and > >> > >> rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > to %-:number%:%dst-ip:ipv4%/%dst-port:number% > >> > >> This works and correctly processes. Hope that helps. > >> > >> James > >> > >> _______________________________________________ > >> Lognorm mailing list > >> Lognorm at lists.adiscon.com > >> http://lists.adiscon.net/mailman/listinfo/lognorm > > > > Any movement on this? I'm at an impasse until this is addressed...thank you. > > > > James > > _______________________________________________ > > Lognorm mailing list > > Lognorm at lists.adiscon.com > > http://lists.adiscon.net/mailman/listinfo/lognorm > > Would have been nice to at least get some response of some kind on this... Oh, damn, sorry - I have totally lost track of this. I need to look at another issue today, but will try to tweak it in between the other stuff ASAP. Rainer From rgerhards at hq.adiscon.com Wed Jul 4 18:19:07 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Jul 2012 16:19:07 +0000 Subject: [Lognorm] Special characters revisited In-Reply-To: References: Message-ID: <2E7D85FDF00FA545A325B6B2EC73A12C6FA23F@vmexch2.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of James Lay > Sent: Friday, May 25, 2012 8:33 PM > To: lognorm at lists.adiscon.com > Subject: [Lognorm] Special characters revisited > > Hey all! > > So...here's where I'm at. Here's my rulebase: > > prefix= > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > to %-:word%:%dst-ip:ipv4%/%dst-port:number% > > > Sample firewall hit (quotes added for clarity): > > " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" > > and my result: > > [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 to > Interface:my_ip/443" unparsed-data=""] > > I've tried the below (thinking the : would be included in word) > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > to %-:word%%dst-ip:ipv4%/%dst-port:number% > > And I get the same result. > > I've also tried: > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > to Interface:%dst-ip:ipv4%/%dst-port:number% > > And this works, so I'm pretty sure it's the : that is my issue. A full > -vv below. Any thoughts on how to get this to go? Thanks all: The core problem is that the "word" syntax takes everyting up to the next SPACE as a word. So this is all part of the word: "Interface:my_ip/443" and this is also the reason why there is no data left for the other fields to parse. The "char-to" syntax was created to process characters up to a specific char. I am right now looking at the correct template, but thought I let you know the interim outcome. Rainer > > James > > liblognorm: read sample line: 'prefix=' > liblognorm: read sample line: 'rule=: %-:word% request discarded from > %src-ip:ipv4%/%src-port:number% to > %-:word%:%dst-ip:ipv4%/%dst-port:number%' > liblognorm: sample line to add: ': %-:word% request discarded from > %src-ip:ipv4%/%src-port:number% to > %-:word%:%dst-ip:ipv4%/%dst-port:number%' > > liblognorm: addSampToTree 0 of 108 > liblognorm: parsed literal: ' ' > liblognorm: buildPTree: begin at 0x9308030, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 1 of 108 > liblognorm: parsed field: '-' > liblognorm: got new subtree 0x9308810 > liblognorm: prev subtree 0x9308030 > liblognorm: new subtree 0x9308810 > liblognorm: addSampToTree 9 of 108 > liblognorm: parsed literal: ' request discarded from ' > liblognorm: buildPTree: begin at 0x9308810, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 24, offs 0 > liblognorm: addSampToTree 33 of 108 > liblognorm: parsed field: 'src-ip' > liblognorm: got new subtree 0x9308ca0 > liblognorm: prev subtree 0x9308810 > liblognorm: new subtree 0x9308ca0 > liblognorm: addSampToTree 46 of 108 > liblognorm: parsed literal: '/' > liblognorm: buildPTree: begin at 0x9308ca0, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 47 of 108 > liblognorm: parsed field: 'src-port' > liblognorm: got new subtree 0x9309110 > liblognorm: prev subtree 0x9308ca0 > liblognorm: new subtree 0x9309110 > liblognorm: addSampToTree 64 of 108 > liblognorm: parsed literal: ' to ' > liblognorm: buildPTree: begin at 0x9309110, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 4, offs 0 > liblognorm: addSampToTree 68 of 108 > liblognorm: parsed field: '-' > liblognorm: got new subtree 0x9309580 > liblognorm: prev subtree 0x9309110 > liblognorm: new subtree 0x9309580 > liblognorm: addSampToTree 76 of 108 > liblognorm: parsed literal: ':' > liblognorm: buildPTree: begin at 0x9309580, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 77 of 108 > liblognorm: parsed field: 'dst-ip' > liblognorm: got new subtree 0x93099f0 > liblognorm: prev subtree 0x9309580 > liblognorm: new subtree 0x93099f0 > liblognorm: addSampToTree 90 of 108 > liblognorm: parsed literal: '/' > liblognorm: buildPTree: begin at 0x93099f0, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 91 of 108 > liblognorm: parsed field: 'dst-port' > liblognorm: got new subtree 0x9309e60 > liblognorm: prev subtree 0x93099f0 > liblognorm: new subtree 0x9309e60 > liblognorm: end addSampToTree 108 of 108 > number of tree nodes: 7 > To normalize: ' TCP request discarded from 96.18.101.250/1077 to > Interface:my_ip/443' > liblognorm: 0: prefix compare ' ', ' ' > liblognorm: 1: prefix compare succeeded, still valid > liblognorm: 1:trying parser for field '-': 0xb786c450 > liblognorm: potential hit, trying subtree > liblognorm: 4: prefix compare ' ', ' ' > liblognorm: 5: prefix compare 'r', 'r' > liblognorm: 6: prefix compare 'e', 'e' > liblognorm: 7: prefix compare 'q', 'q' > liblognorm: 8: prefix compare 'u', 'u' > liblognorm: 9: prefix compare 'e', 'e' > liblognorm: 10: prefix compare 's', 's' > liblognorm: 11: prefix compare 't', 't' > liblognorm: 12: prefix compare ' ', ' ' > liblognorm: 13: prefix compare 'd', 'd' > liblognorm: 14: prefix compare 'i', 'i' > liblognorm: 15: prefix compare 's', 's' > liblognorm: 16: prefix compare 'c', 'c' > liblognorm: 17: prefix compare 'a', 'a' > liblognorm: 18: prefix compare 'r', 'r' > liblognorm: 19: prefix compare 'd', 'd' > liblognorm: 20: prefix compare 'e', 'e' > liblognorm: 21: prefix compare 'd', 'd' > liblognorm: 22: prefix compare ' ', ' ' > liblognorm: 23: prefix compare 'f', 'f' > liblognorm: 24: prefix compare 'r', 'r' > liblognorm: 25: prefix compare 'o', 'o' > liblognorm: 26: prefix compare 'm', 'm' > liblognorm: 27: prefix compare ' ', ' ' > liblognorm: 28: prefix compare succeeded, still valid > liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 > liblognorm: potential hit, trying subtree > liblognorm: 41: prefix compare '/', '/' > liblognorm: 42: prefix compare succeeded, still valid > liblognorm: 42:trying parser for field 'src-port': 0xb786c510 > liblognorm: potential hit, trying subtree > liblognorm: 46: prefix compare ' ', ' ' > liblognorm: 47: prefix compare 't', 't' > liblognorm: 48: prefix compare 'o', 'o' > liblognorm: 49: prefix compare ' ', ' ' > liblognorm: 50: prefix compare succeeded, still valid > liblognorm: 50:trying parser for field '-': 0xb786c450 > liblognorm: potential hit, trying subtree > liblognorm: 74 returns -1 > liblognorm: 50 nonmatch, backtracking required, left=-1 > liblognorm: 50 no field, trying subtree char 'I': (nil) > liblognorm: 50 returns -1 > liblognorm: 42 nonmatch, backtracking required, left=-1 > liblognorm: 42 no field, trying subtree char '1': (nil) > liblognorm: 42 returns -1 > liblognorm: 28 nonmatch, backtracking required, left=-1 > liblognorm: 28 no field, trying subtree char '9': (nil) > liblognorm: 28 returns -1 > liblognorm: 1 nonmatch, backtracking required, left=-1 > liblognorm: 1 no field, trying subtree char 'T': (nil) > liblognorm: 1 returns -1 > liblognorm: final result for normalizer: left -1, endNode 0x9309580 > normalized: '[cee at 115 originalmsg=" TCP request discarded from > 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' > [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 to > Interface:my_ip/443" unparsed-data=""] > > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From rgerhards at hq.adiscon.com Wed Jul 4 18:29:13 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 4 Jul 2012 16:29:13 +0000 Subject: [Lognorm] Special characters revisited In-Reply-To: <2E7D85FDF00FA545A325B6B2EC73A12C6FA23F@vmexch2.intern.adiscon.com> References: <2E7D85FDF00FA545A325B6B2EC73A12C6FA23F@vmexch2.intern.adiscon.com> Message-ID: <2E7D85FDF00FA545A325B6B2EC73A12C6FA273@vmexch2.intern.adiscon.com> > > So...here's where I'm at. Here's my rulebase: > > > > prefix= > > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > > to %-:word%:%dst-ip:ipv4%/%dst-port:number% > > > > > > Sample firewall hit (quotes added for clarity): > > > > " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" > > > > and my result: > > > > [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 to > > Interface:my_ip/443" unparsed-data=""] > > > > I've tried the below (thinking the : would be included in word) > > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > > to %-:word%%dst-ip:ipv4%/%dst-port:number% > > > > And I get the same result. > > > > I've also tried: > > rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% > > to Interface:%dst-ip:ipv4%/%dst-port:number% > > > > And this works, so I'm pretty sure it's the : that is my issue. A full > > -vv below. Any thoughts on how to get this to go? Thanks all: > > The core problem is that the "word" syntax takes everyting up to the next SPACE > as a word. So this is all part of the word: "Interface:my_ip/443" and this is also > the reason why there is no data left for the other fields to parse. The "char-to" > syntax was created to process characters up to a specific char. I am right now > looking at the correct template, but thought I let you know the interim > outcome. This works for me: prefix= #rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% to %-:char-to::%:%dst-ip:ipv4%/%dst-port:number% rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% to %-:char-to::%:%dst-ip:char-to:/%/%dst-port:number% Note that the second line is commented out. It does not work because "my_ip" is not an IPv4 address. If you just anonymized the entry, that line should work. The third line accepts a string up to the slash character (and is also a nice example of different char-to syntaxes). Note that the colon in char-to is kind of a hack -- it usually should be escaped, but I overlooked that case when implementing char-to. Please watch further release announcements closely, I may change that to the proper escaped form (without escape, it can be confusing to both humans and parsers...). Sorry it took so long, I have to admit I lost track of the issue. I hope the answer is still useful for you. Rainer From jlay at slave-tothe-box.net Wed Jul 4 18:44:26 2012 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 4 Jul 2012 10:44:26 -0600 Subject: [Lognorm] Special characters revisited In-Reply-To: <2E7D85FDF00FA545A325B6B2EC73A12C6FA0AA@vmexch2.intern.adiscon.com> References: <09e54deeee391553381e6c7f8dfb4f7f@192.168.1.253> <827E2018-77A9-4E3D-8514-798893ED1605@slave-tothe-box.net> <2447E5E6-C891-4FC9-BC1A-C42C3AE49479@slave-tothe-box.net> <2E7D85FDF00FA545A325B6B2EC73A12C6FA0AA@vmexch2.intern.adiscon.com> Message-ID: On Jul 4, 2012, at 9:32 AM, Rainer Gerhards wrote: > > >> -----Original Message----- >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> bounces at lists.adiscon.com] On Behalf Of James Lay >> Sent: Wednesday, July 04, 2012 4:10 PM >> To: lognorm >> Subject: Re: [Lognorm] Special characters revisited >> >> On Jun 13, 2012, at 7:11 AM, James Lay wrote: >>> On May 25, 2012, at 12:44 PM, James Lay wrote: >>>> On Fri, 25 May 2012 12:33:15 -0600, James Lay wrote: >>>>> Hey all! >>>>> >>>>> So...here's where I'm at. Here's my rulebase: >>>>> >>>>> prefix= >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> >>>>> Sample firewall hit (quotes added for clarity): >>>>> >>>>> " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" >>>>> >>>>> and my result: >>>>> >>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>> to Interface:my_ip/443" unparsed-data=""] >>>>> >>>>> I've tried the below (thinking the : would be included in word) >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> And I get the same result. >>>>> >>>>> I've also tried: >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> Interface:%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> And this works, so I'm pretty sure it's the : that is my issue. A >>>>> full -vv below. Any thoughts on how to get this to go? Thanks all: >>>>> >>>>> James >>>>> >>>>> liblognorm: read sample line: 'prefix=' >>>>> liblognorm: read sample line: 'rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>> liblognorm: sample line to add: ': %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>> >>>>> liblognorm: addSampToTree 0 of 108 >>>>> liblognorm: parsed literal: ' ' >>>>> liblognorm: buildPTree: begin at 0x9308030, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 1 of 108 >>>>> liblognorm: parsed field: '-' >>>>> liblognorm: got new subtree 0x9308810 >>>>> liblognorm: prev subtree 0x9308030 >>>>> liblognorm: new subtree 0x9308810 >>>>> liblognorm: addSampToTree 9 of 108 >>>>> liblognorm: parsed literal: ' request discarded from ' >>>>> liblognorm: buildPTree: begin at 0x9308810, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 24, offs 0 >>>>> liblognorm: addSampToTree 33 of 108 >>>>> liblognorm: parsed field: 'src-ip' >>>>> liblognorm: got new subtree 0x9308ca0 >>>>> liblognorm: prev subtree 0x9308810 >>>>> liblognorm: new subtree 0x9308ca0 >>>>> liblognorm: addSampToTree 46 of 108 >>>>> liblognorm: parsed literal: '/' >>>>> liblognorm: buildPTree: begin at 0x9308ca0, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 47 of 108 >>>>> liblognorm: parsed field: 'src-port' >>>>> liblognorm: got new subtree 0x9309110 >>>>> liblognorm: prev subtree 0x9308ca0 >>>>> liblognorm: new subtree 0x9309110 >>>>> liblognorm: addSampToTree 64 of 108 >>>>> liblognorm: parsed literal: ' to ' >>>>> liblognorm: buildPTree: begin at 0x9309110, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 4, offs 0 >>>>> liblognorm: addSampToTree 68 of 108 >>>>> liblognorm: parsed field: '-' >>>>> liblognorm: got new subtree 0x9309580 >>>>> liblognorm: prev subtree 0x9309110 >>>>> liblognorm: new subtree 0x9309580 >>>>> liblognorm: addSampToTree 76 of 108 >>>>> liblognorm: parsed literal: ':' >>>>> liblognorm: buildPTree: begin at 0x9309580, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 77 of 108 >>>>> liblognorm: parsed field: 'dst-ip' >>>>> liblognorm: got new subtree 0x93099f0 >>>>> liblognorm: prev subtree 0x9309580 >>>>> liblognorm: new subtree 0x93099f0 >>>>> liblognorm: addSampToTree 90 of 108 >>>>> liblognorm: parsed literal: '/' >>>>> liblognorm: buildPTree: begin at 0x93099f0, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 91 of 108 >>>>> liblognorm: parsed field: 'dst-port' >>>>> liblognorm: got new subtree 0x9309e60 >>>>> liblognorm: prev subtree 0x93099f0 >>>>> liblognorm: new subtree 0x9309e60 >>>>> liblognorm: end addSampToTree 108 of 108 >>>>> number of tree nodes: 7 >>>>> To normalize: ' TCP request discarded from 96.18.101.250/1077 to >>>>> Interface:my_ip/443' >>>>> liblognorm: 0: prefix compare ' ', ' ' >>>>> liblognorm: 1: prefix compare succeeded, still valid >>>>> liblognorm: 1:trying parser for field '-': 0xb786c450 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 4: prefix compare ' ', ' ' >>>>> liblognorm: 5: prefix compare 'r', 'r' >>>>> liblognorm: 6: prefix compare 'e', 'e' >>>>> liblognorm: 7: prefix compare 'q', 'q' >>>>> liblognorm: 8: prefix compare 'u', 'u' >>>>> liblognorm: 9: prefix compare 'e', 'e' >>>>> liblognorm: 10: prefix compare 's', 's' >>>>> liblognorm: 11: prefix compare 't', 't' >>>>> liblognorm: 12: prefix compare ' ', ' ' >>>>> liblognorm: 13: prefix compare 'd', 'd' >>>>> liblognorm: 14: prefix compare 'i', 'i' >>>>> liblognorm: 15: prefix compare 's', 's' >>>>> liblognorm: 16: prefix compare 'c', 'c' >>>>> liblognorm: 17: prefix compare 'a', 'a' >>>>> liblognorm: 18: prefix compare 'r', 'r' >>>>> liblognorm: 19: prefix compare 'd', 'd' >>>>> liblognorm: 20: prefix compare 'e', 'e' >>>>> liblognorm: 21: prefix compare 'd', 'd' >>>>> liblognorm: 22: prefix compare ' ', ' ' >>>>> liblognorm: 23: prefix compare 'f', 'f' >>>>> liblognorm: 24: prefix compare 'r', 'r' >>>>> liblognorm: 25: prefix compare 'o', 'o' >>>>> liblognorm: 26: prefix compare 'm', 'm' >>>>> liblognorm: 27: prefix compare ' ', ' ' >>>>> liblognorm: 28: prefix compare succeeded, still valid >>>>> liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 41: prefix compare '/', '/' >>>>> liblognorm: 42: prefix compare succeeded, still valid >>>>> liblognorm: 42:trying parser for field 'src-port': 0xb786c510 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 46: prefix compare ' ', ' ' >>>>> liblognorm: 47: prefix compare 't', 't' >>>>> liblognorm: 48: prefix compare 'o', 'o' >>>>> liblognorm: 49: prefix compare ' ', ' ' >>>>> liblognorm: 50: prefix compare succeeded, still valid >>>>> liblognorm: 50:trying parser for field '-': 0xb786c450 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 74 returns -1 >>>>> liblognorm: 50 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 50 no field, trying subtree char 'I': (nil) >>>>> liblognorm: 50 returns -1 >>>>> liblognorm: 42 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 42 no field, trying subtree char '1': (nil) >>>>> liblognorm: 42 returns -1 >>>>> liblognorm: 28 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 28 no field, trying subtree char '9': (nil) >>>>> liblognorm: 28 returns -1 >>>>> liblognorm: 1 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 1 no field, trying subtree char 'T': (nil) >>>>> liblognorm: 1 returns -1 >>>>> liblognorm: final result for normalizer: left -1, endNode 0x9309580 >>>>> normalized: '[cee at 115 originalmsg=" TCP request discarded from >>>>> 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' >>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>> to Interface:my_ip/443" unparsed-data=""] >>>>> >>>>> _______________________________________________ >>>>> Lognorm mailing list >>>>> Lognorm at lists.adiscon.com >>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>> >>>> An interesting tidbit...with the below: >>>> >>>> >>>> " TCP request discarded from 96.18.101.250/1077 to 1234:my_ip/443" >>>> >>>> and >>>> >>>> rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% >> to %-:number%:%dst-ip:ipv4%/%dst-port:number% >>>> >>>> This works and correctly processes. Hope that helps. >>>> >>>> James >>>> >>>> _______________________________________________ >>>> Lognorm mailing list >>>> Lognorm at lists.adiscon.com >>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>> >>> Any movement on this? I'm at an impasse until this is addressed...thank you. >>> >>> James >>> _______________________________________________ >>> Lognorm mailing list >>> Lognorm at lists.adiscon.com >>> http://lists.adiscon.net/mailman/listinfo/lognorm >> >> Would have been nice to at least get some response of some kind on this... > > Oh, damn, sorry - I have totally lost track of this. I need to look at another issue today, but will try to tweak it in between the other stuff ASAP. > > Rainer > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Thanks Rainer?I'll give this a shot when I get to work tomorrow. James From cclark at quadrantsec.com Wed Jul 4 21:50:55 2012 From: cclark at quadrantsec.com (=?utf-8?B?Q2hhbXAgQ2xhcmsgSUlJIChRdWFkcmFudCk=?=) Date: Wed, 04 Jul 2012 15:50:55 -0400 Subject: [Lognorm] =?utf-8?q?Special_characters_revisited?= Message-ID: <201207041950.q64Joap2018823@a.qsecure.net> Rainer, Thanks for Looking into this. I've had similar issues as well. Look forward to reading more about this. Champ Clark III - Quadrant information security ----- Reply message ----- From: "James Lay" Date: Wed, Jul 4, 2012 12:44 Subject: [Lognorm] Special characters revisited To: "lognorm" On Jul 4, 2012, at 9:32 AM, Rainer Gerhards wrote: > > >> -----Original Message----- >> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >> bounces at lists.adiscon.com] On Behalf Of James Lay >> Sent: Wednesday, July 04, 2012 4:10 PM >> To: lognorm >> Subject: Re: [Lognorm] Special characters revisited >> >> On Jun 13, 2012, at 7:11 AM, James Lay wrote: >>> On May 25, 2012, at 12:44 PM, James Lay wrote: >>>> On Fri, 25 May 2012 12:33:15 -0600, James Lay wrote: >>>>> Hey all! >>>>> >>>>> So...here's where I'm at. Here's my rulebase: >>>>> >>>>> prefix= >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> >>>>> Sample firewall hit (quotes added for clarity): >>>>> >>>>> " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" >>>>> >>>>> and my result: >>>>> >>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>> to Interface:my_ip/443" unparsed-data=""] >>>>> >>>>> I've tried the below (thinking the : would be included in word) >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> And I get the same result. >>>>> >>>>> I've also tried: >>>>> rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> Interface:%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> And this works, so I'm pretty sure it's the : that is my issue. A >>>>> full -vv below. Any thoughts on how to get this to go? Thanks all: >>>>> >>>>> James >>>>> >>>>> liblognorm: read sample line: 'prefix=' >>>>> liblognorm: read sample line: 'rule=: %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>> liblognorm: sample line to add: ': %-:word% request discarded from >>>>> %src-ip:ipv4%/%src-port:number% to >>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>> >>>>> liblognorm: addSampToTree 0 of 108 >>>>> liblognorm: parsed literal: ' ' >>>>> liblognorm: buildPTree: begin at 0x9308030, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 1 of 108 >>>>> liblognorm: parsed field: '-' >>>>> liblognorm: got new subtree 0x9308810 >>>>> liblognorm: prev subtree 0x9308030 >>>>> liblognorm: new subtree 0x9308810 >>>>> liblognorm: addSampToTree 9 of 108 >>>>> liblognorm: parsed literal: ' request discarded from ' >>>>> liblognorm: buildPTree: begin at 0x9308810, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 24, offs 0 >>>>> liblognorm: addSampToTree 33 of 108 >>>>> liblognorm: parsed field: 'src-ip' >>>>> liblognorm: got new subtree 0x9308ca0 >>>>> liblognorm: prev subtree 0x9308810 >>>>> liblognorm: new subtree 0x9308ca0 >>>>> liblognorm: addSampToTree 46 of 108 >>>>> liblognorm: parsed literal: '/' >>>>> liblognorm: buildPTree: begin at 0x9308ca0, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 47 of 108 >>>>> liblognorm: parsed field: 'src-port' >>>>> liblognorm: got new subtree 0x9309110 >>>>> liblognorm: prev subtree 0x9308ca0 >>>>> liblognorm: new subtree 0x9309110 >>>>> liblognorm: addSampToTree 64 of 108 >>>>> liblognorm: parsed literal: ' to ' >>>>> liblognorm: buildPTree: begin at 0x9309110, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 4, offs 0 >>>>> liblognorm: addSampToTree 68 of 108 >>>>> liblognorm: parsed field: '-' >>>>> liblognorm: got new subtree 0x9309580 >>>>> liblognorm: prev subtree 0x9309110 >>>>> liblognorm: new subtree 0x9309580 >>>>> liblognorm: addSampToTree 76 of 108 >>>>> liblognorm: parsed literal: ':' >>>>> liblognorm: buildPTree: begin at 0x9309580, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 77 of 108 >>>>> liblognorm: parsed field: 'dst-ip' >>>>> liblognorm: got new subtree 0x93099f0 >>>>> liblognorm: prev subtree 0x9309580 >>>>> liblognorm: new subtree 0x93099f0 >>>>> liblognorm: addSampToTree 90 of 108 >>>>> liblognorm: parsed literal: '/' >>>>> liblognorm: buildPTree: begin at 0x93099f0, offs 0 >>>>> liblognorm: case 3.1 >>>>> liblognorm: addPTree: offs 0 >>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>> liblognorm: addSampToTree 91 of 108 >>>>> liblognorm: parsed field: 'dst-port' >>>>> liblognorm: got new subtree 0x9309e60 >>>>> liblognorm: prev subtree 0x93099f0 >>>>> liblognorm: new subtree 0x9309e60 >>>>> liblognorm: end addSampToTree 108 of 108 >>>>> number of tree nodes: 7 >>>>> To normalize: ' TCP request discarded from 96.18.101.250/1077 to >>>>> Interface:my_ip/443' >>>>> liblognorm: 0: prefix compare ' ', ' ' >>>>> liblognorm: 1: prefix compare succeeded, still valid >>>>> liblognorm: 1:trying parser for field '-': 0xb786c450 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 4: prefix compare ' ', ' ' >>>>> liblognorm: 5: prefix compare 'r', 'r' >>>>> liblognorm: 6: prefix compare 'e', 'e' >>>>> liblognorm: 7: prefix compare 'q', 'q' >>>>> liblognorm: 8: prefix compare 'u', 'u' >>>>> liblognorm: 9: prefix compare 'e', 'e' >>>>> liblognorm: 10: prefix compare 's', 's' >>>>> liblognorm: 11: prefix compare 't', 't' >>>>> liblognorm: 12: prefix compare ' ', ' ' >>>>> liblognorm: 13: prefix compare 'd', 'd' >>>>> liblognorm: 14: prefix compare 'i', 'i' >>>>> liblognorm: 15: prefix compare 's', 's' >>>>> liblognorm: 16: prefix compare 'c', 'c' >>>>> liblognorm: 17: prefix compare 'a', 'a' >>>>> liblognorm: 18: prefix compare 'r', 'r' >>>>> liblognorm: 19: prefix compare 'd', 'd' >>>>> liblognorm: 20: prefix compare 'e', 'e' >>>>> liblognorm: 21: prefix compare 'd', 'd' >>>>> liblognorm: 22: prefix compare ' ', ' ' >>>>> liblognorm: 23: prefix compare 'f', 'f' >>>>> liblognorm: 24: prefix compare 'r', 'r' >>>>> liblognorm: 25: prefix compare 'o', 'o' >>>>> liblognorm: 26: prefix compare 'm', 'm' >>>>> liblognorm: 27: prefix compare ' ', ' ' >>>>> liblognorm: 28: prefix compare succeeded, still valid >>>>> liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 41: prefix compare '/', '/' >>>>> liblognorm: 42: prefix compare succeeded, still valid >>>>> liblognorm: 42:trying parser for field 'src-port': 0xb786c510 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 46: prefix compare ' ', ' ' >>>>> liblognorm: 47: prefix compare 't', 't' >>>>> liblognorm: 48: prefix compare 'o', 'o' >>>>> liblognorm: 49: prefix compare ' ', ' ' >>>>> liblognorm: 50: prefix compare succeeded, still valid >>>>> liblognorm: 50:trying parser for field '-': 0xb786c450 >>>>> liblognorm: potential hit, trying subtree >>>>> liblognorm: 74 returns -1 >>>>> liblognorm: 50 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 50 no field, trying subtree char 'I': (nil) >>>>> liblognorm: 50 returns -1 >>>>> liblognorm: 42 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 42 no field, trying subtree char '1': (nil) >>>>> liblognorm: 42 returns -1 >>>>> liblognorm: 28 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 28 no field, trying subtree char '9': (nil) >>>>> liblognorm: 28 returns -1 >>>>> liblognorm: 1 nonmatch, backtracking required, left=-1 >>>>> liblognorm: 1 no field, trying subtree char 'T': (nil) >>>>> liblognorm: 1 returns -1 >>>>> liblognorm: final result for normalizer: left -1, endNode 0x9309580 >>>>> normalized: '[cee at 115 originalmsg=" TCP request discarded from >>>>> 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' >>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>> to Interface:my_ip/443" unparsed-data=""] >>>>> >>>>> _______________________________________________ >>>>> Lognorm mailing list >>>>> Lognorm at lists.adiscon.com >>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>> >>>> An interesting tidbit...with the below: >>>> >>>> >>>> " TCP request discarded from 96.18.101.250/1077 to 1234:my_ip/443" >>>> >>>> and >>>> >>>> rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% >> to %-:number%:%dst-ip:ipv4%/%dst-port:number% >>>> >>>> This works and correctly processes. Hope that helps. >>>> >>>> James >>>> >>>> _______________________________________________ >>>> Lognorm mailing list >>>> Lognorm at lists.adiscon.com >>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>> >>> Any movement on this? I'm at an impasse until this is addressed...thank you. >>> >>> James >>> _______________________________________________ >>> Lognorm mailing list >>> Lognorm at lists.adiscon.com >>> http://lists.adiscon.net/mailman/listinfo/lognorm >> >> Would have been nice to at least get some response of some kind on this... > > Oh, damn, sorry - I have totally lost track of this. I need to look at another issue today, but will try to tweak it in between the other stuff ASAP. > > Rainer > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Thanks Rainer?I'll give this a shot when I get to work tomorrow. James _______________________________________________ Lognorm mailing list Lognorm at lists.adiscon.com http://lists.adiscon.net/mailman/listinfo/lognorm -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlay at slave-tothe-box.net Fri Jul 6 01:48:35 2012 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 5 Jul 2012 17:48:35 -0600 Subject: [Lognorm] Special characters revisited In-Reply-To: References: <09e54deeee391553381e6c7f8dfb4f7f@192.168.1.253> <827E2018-77A9-4E3D-8514-798893ED1605@slave-tothe-box.net> <2447E5E6-C891-4FC9-BC1A-C42C3AE49479@slave-tothe-box.net> <2E7D85FDF00FA545A325B6B2EC73A12C6FA0AA@vmexch2.intern.adiscon.com> Message-ID: On Jul 4, 2012, at 10:44 AM, James Lay wrote: > On Jul 4, 2012, at 9:32 AM, Rainer Gerhards wrote: >> >> >>> -----Original Message----- >>> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- >>> bounces at lists.adiscon.com] On Behalf Of James Lay >>> Sent: Wednesday, July 04, 2012 4:10 PM >>> To: lognorm >>> Subject: Re: [Lognorm] Special characters revisited >>> >>> On Jun 13, 2012, at 7:11 AM, James Lay wrote: >>>> On May 25, 2012, at 12:44 PM, James Lay wrote: >>>>> On Fri, 25 May 2012 12:33:15 -0600, James Lay wrote: >>>>>> Hey all! >>>>>> >>>>>> So...here's where I'm at. Here's my rulebase: >>>>>> >>>>>> prefix= >>>>>> rule=: %-:word% request discarded from >>>>>> %src-ip:ipv4%/%src-port:number% to >>>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number% >>>>>> >>>>>> >>>>>> Sample firewall hit (quotes added for clarity): >>>>>> >>>>>> " TCP request discarded from 96.18.101.250/1077 to Interface:my_ip/443" >>>>>> >>>>>> and my result: >>>>>> >>>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>>> to Interface:my_ip/443" unparsed-data=""] >>>>>> >>>>>> I've tried the below (thinking the : would be included in word) >>>>>> rule=: %-:word% request discarded from >>>>>> %src-ip:ipv4%/%src-port:number% to >>>>>> %-:word%%dst-ip:ipv4%/%dst-port:number% >>>>>> >>>>>> And I get the same result. >>>>>> >>>>>> I've also tried: >>>>>> rule=: %-:word% request discarded from >>>>>> %src-ip:ipv4%/%src-port:number% to >>>>>> Interface:%dst-ip:ipv4%/%dst-port:number% >>>>>> >>>>>> And this works, so I'm pretty sure it's the : that is my issue. A >>>>>> full -vv below. Any thoughts on how to get this to go? Thanks all: >>>>>> >>>>>> James >>>>>> >>>>>> liblognorm: read sample line: 'prefix=' >>>>>> liblognorm: read sample line: 'rule=: %-:word% request discarded from >>>>>> %src-ip:ipv4%/%src-port:number% to >>>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>>> liblognorm: sample line to add: ': %-:word% request discarded from >>>>>> %src-ip:ipv4%/%src-port:number% to >>>>>> %-:word%:%dst-ip:ipv4%/%dst-port:number%' >>>>>> >>>>>> liblognorm: addSampToTree 0 of 108 >>>>>> liblognorm: parsed literal: ' ' >>>>>> liblognorm: buildPTree: begin at 0x9308030, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>>> liblognorm: addSampToTree 1 of 108 >>>>>> liblognorm: parsed field: '-' >>>>>> liblognorm: got new subtree 0x9308810 >>>>>> liblognorm: prev subtree 0x9308030 >>>>>> liblognorm: new subtree 0x9308810 >>>>>> liblognorm: addSampToTree 9 of 108 >>>>>> liblognorm: parsed literal: ' request discarded from ' >>>>>> liblognorm: buildPTree: begin at 0x9308810, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 24, offs 0 >>>>>> liblognorm: addSampToTree 33 of 108 >>>>>> liblognorm: parsed field: 'src-ip' >>>>>> liblognorm: got new subtree 0x9308ca0 >>>>>> liblognorm: prev subtree 0x9308810 >>>>>> liblognorm: new subtree 0x9308ca0 >>>>>> liblognorm: addSampToTree 46 of 108 >>>>>> liblognorm: parsed literal: '/' >>>>>> liblognorm: buildPTree: begin at 0x9308ca0, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>>> liblognorm: addSampToTree 47 of 108 >>>>>> liblognorm: parsed field: 'src-port' >>>>>> liblognorm: got new subtree 0x9309110 >>>>>> liblognorm: prev subtree 0x9308ca0 >>>>>> liblognorm: new subtree 0x9309110 >>>>>> liblognorm: addSampToTree 64 of 108 >>>>>> liblognorm: parsed literal: ' to ' >>>>>> liblognorm: buildPTree: begin at 0x9309110, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 4, offs 0 >>>>>> liblognorm: addSampToTree 68 of 108 >>>>>> liblognorm: parsed field: '-' >>>>>> liblognorm: got new subtree 0x9309580 >>>>>> liblognorm: prev subtree 0x9309110 >>>>>> liblognorm: new subtree 0x9309580 >>>>>> liblognorm: addSampToTree 76 of 108 >>>>>> liblognorm: parsed literal: ':' >>>>>> liblognorm: buildPTree: begin at 0x9309580, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>>> liblognorm: addSampToTree 77 of 108 >>>>>> liblognorm: parsed field: 'dst-ip' >>>>>> liblognorm: got new subtree 0x93099f0 >>>>>> liblognorm: prev subtree 0x9309580 >>>>>> liblognorm: new subtree 0x93099f0 >>>>>> liblognorm: addSampToTree 90 of 108 >>>>>> liblognorm: parsed literal: '/' >>>>>> liblognorm: buildPTree: begin at 0x93099f0, offs 0 >>>>>> liblognorm: case 3.1 >>>>>> liblognorm: addPTree: offs 0 >>>>>> liblognorm: setPrefix lenBuf 1, offs 0 >>>>>> liblognorm: addSampToTree 91 of 108 >>>>>> liblognorm: parsed field: 'dst-port' >>>>>> liblognorm: got new subtree 0x9309e60 >>>>>> liblognorm: prev subtree 0x93099f0 >>>>>> liblognorm: new subtree 0x9309e60 >>>>>> liblognorm: end addSampToTree 108 of 108 >>>>>> number of tree nodes: 7 >>>>>> To normalize: ' TCP request discarded from 96.18.101.250/1077 to >>>>>> Interface:my_ip/443' >>>>>> liblognorm: 0: prefix compare ' ', ' ' >>>>>> liblognorm: 1: prefix compare succeeded, still valid >>>>>> liblognorm: 1:trying parser for field '-': 0xb786c450 >>>>>> liblognorm: potential hit, trying subtree >>>>>> liblognorm: 4: prefix compare ' ', ' ' >>>>>> liblognorm: 5: prefix compare 'r', 'r' >>>>>> liblognorm: 6: prefix compare 'e', 'e' >>>>>> liblognorm: 7: prefix compare 'q', 'q' >>>>>> liblognorm: 8: prefix compare 'u', 'u' >>>>>> liblognorm: 9: prefix compare 'e', 'e' >>>>>> liblognorm: 10: prefix compare 's', 's' >>>>>> liblognorm: 11: prefix compare 't', 't' >>>>>> liblognorm: 12: prefix compare ' ', ' ' >>>>>> liblognorm: 13: prefix compare 'd', 'd' >>>>>> liblognorm: 14: prefix compare 'i', 'i' >>>>>> liblognorm: 15: prefix compare 's', 's' >>>>>> liblognorm: 16: prefix compare 'c', 'c' >>>>>> liblognorm: 17: prefix compare 'a', 'a' >>>>>> liblognorm: 18: prefix compare 'r', 'r' >>>>>> liblognorm: 19: prefix compare 'd', 'd' >>>>>> liblognorm: 20: prefix compare 'e', 'e' >>>>>> liblognorm: 21: prefix compare 'd', 'd' >>>>>> liblognorm: 22: prefix compare ' ', ' ' >>>>>> liblognorm: 23: prefix compare 'f', 'f' >>>>>> liblognorm: 24: prefix compare 'r', 'r' >>>>>> liblognorm: 25: prefix compare 'o', 'o' >>>>>> liblognorm: 26: prefix compare 'm', 'm' >>>>>> liblognorm: 27: prefix compare ' ', ' ' >>>>>> liblognorm: 28: prefix compare succeeded, still valid >>>>>> liblognorm: 28:trying parser for field 'src-ip': 0xb786bdb0 >>>>>> liblognorm: potential hit, trying subtree >>>>>> liblognorm: 41: prefix compare '/', '/' >>>>>> liblognorm: 42: prefix compare succeeded, still valid >>>>>> liblognorm: 42:trying parser for field 'src-port': 0xb786c510 >>>>>> liblognorm: potential hit, trying subtree >>>>>> liblognorm: 46: prefix compare ' ', ' ' >>>>>> liblognorm: 47: prefix compare 't', 't' >>>>>> liblognorm: 48: prefix compare 'o', 'o' >>>>>> liblognorm: 49: prefix compare ' ', ' ' >>>>>> liblognorm: 50: prefix compare succeeded, still valid >>>>>> liblognorm: 50:trying parser for field '-': 0xb786c450 >>>>>> liblognorm: potential hit, trying subtree >>>>>> liblognorm: 74 returns -1 >>>>>> liblognorm: 50 nonmatch, backtracking required, left=-1 >>>>>> liblognorm: 50 no field, trying subtree char 'I': (nil) >>>>>> liblognorm: 50 returns -1 >>>>>> liblognorm: 42 nonmatch, backtracking required, left=-1 >>>>>> liblognorm: 42 no field, trying subtree char '1': (nil) >>>>>> liblognorm: 42 returns -1 >>>>>> liblognorm: 28 nonmatch, backtracking required, left=-1 >>>>>> liblognorm: 28 no field, trying subtree char '9': (nil) >>>>>> liblognorm: 28 returns -1 >>>>>> liblognorm: 1 nonmatch, backtracking required, left=-1 >>>>>> liblognorm: 1 no field, trying subtree char 'T': (nil) >>>>>> liblognorm: 1 returns -1 >>>>>> liblognorm: final result for normalizer: left -1, endNode 0x9309580 >>>>>> normalized: '[cee at 115 originalmsg=" TCP request discarded from >>>>>> 96.18.101.250/1077 to Interface:my_ip/443" unparsed-data=""]' >>>>>> [cee at 115 originalmsg=" TCP request discarded from 96.18.101.250/1077 >>>>>> to Interface:my_ip/443" unparsed-data=""] >>>>>> >>>>>> _______________________________________________ >>>>>> Lognorm mailing list >>>>>> Lognorm at lists.adiscon.com >>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>> >>>>> An interesting tidbit...with the below: >>>>> >>>>> >>>>> " TCP request discarded from 96.18.101.250/1077 to 1234:my_ip/443" >>>>> >>>>> and >>>>> >>>>> rule=: %-:word% request discarded from %src-ip:ipv4%/%src-port:number% >>> to %-:number%:%dst-ip:ipv4%/%dst-port:number% >>>>> >>>>> This works and correctly processes. Hope that helps. >>>>> >>>>> James >>>>> >>>>> _______________________________________________ >>>>> Lognorm mailing list >>>>> Lognorm at lists.adiscon.com >>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>> >>>> Any movement on this? I'm at an impasse until this is addressed...thank you. >>>> >>>> James >>>> _______________________________________________ >>>> Lognorm mailing list >>>> Lognorm at lists.adiscon.com >>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>> >>> Would have been nice to at least get some response of some kind on this... >> >> Oh, damn, sorry - I have totally lost track of this. I need to look at another issue today, but will try to tweak it in between the other stuff ASAP. >> >> Rainer >> _______________________________________________ >> Lognorm mailing list >> Lognorm at lists.adiscon.com >> http://lists.adiscon.net/mailman/listinfo/lognorm > > Thanks Rainer?I'll give this a shot when I get to work tomorrow. > > James > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm Yea this TOTALLY worked?thanks a BUNCH Rainer! James From alorbach at ro1.adiscon.com Mon Jul 16 18:15:48 2012 From: alorbach at ro1.adiscon.com (Andre Lorbach) Date: Mon, 16 Jul 2012 16:15:48 +0000 Subject: [Lognorm] libee 0.4.0 compilation problems on Centos 5.8 In-Reply-To: References: Message-ID: <5D458DEF8C9D644D8AD24FBF6CA7525F826386@vmexch2.intern.adiscon.com> I had the same Problem on Fedora 17, pkg-config is not working as expected. So I found this workaround which let me compile libee finally, maybe it helps you: http://lists.adiscon.net/pipermail/rsyslog/2012-June/015060.html Remove /local if needed at your end. Best regards, Andre Lorbach From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-bounces at lists.adiscon.com] On Behalf Of Gary Sent: Montag, 19. M?rz 2012 19:38 To: lognorm at lists.adiscon.com Subject: [Lognorm] libee 0.4.0 compilation problems on Centos 5.8 This is my target platform because that's currently collecting all our syslogs for splunk (and eventually sending them to sagan as well if I can get past this hurdle). I've compiled libestr and installed it in /usr/local. The configuration of libee seems to go well. $ LIBESTR_CFLAGS=-I/usr/local/include LIBESTR_LIBS=-L/usr/local/lib ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether gcc and cc understand -c and -o together... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 98304 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for ar... ar checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking for stdlib.h... (cached) yes checking for GNU libc compatible malloc... yes checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for LIBESTR... yes configure: creating ./config.status config.status: creating Makefile config.status: creating libee.pc config.status: creating src/Makefile config.status: creating include/Makefile config.status: creating include/libee/Makefile config.status: creating tests/Makefile config.status: creating config.h config.status: config.h is unchanged config.status: executing depfiles commands config.status: executing libtool commands ***************************************************** libee will be compiled with the following settings: Debug mode enabled: no Testbench enabled: yes However, compiling is another matter... $ make make all-recursive make[1]: Entering directory `/usr/local/src/libee-0.4.0' Making all in tests make[2]: Entering directory `/usr/local/src/libee-0.4.0/tests' make[2]: Nothing to be done for `all'. make[2]: Leaving directory `/usr/local/src/libee-0.4.0/tests' Making all in include make[2]: Entering directory `/usr/local/src/libee-0.4.0/include' Making all in libee make[3]: Entering directory `/usr/local/src/libee-0.4.0/include/libee' make[3]: Nothing to be done for `all'. make[3]: Leaving directory `/usr/local/src/libee-0.4.0/include/libee' make[3]: Entering directory `/usr/local/src/libee-0.4.0/include' make[3]: Nothing to be done for `all-am'. make[3]: Leaving directory `/usr/local/src/libee-0.4.0/include' make[2]: Leaving directory `/usr/local/src/libee-0.4.0/include' Making all in src make[2]: Entering directory `/usr/local/src/libee-0.4.0/src' CC libee_la-cjson.lo CC libee_la-ctx.lo CC libee_la-tag.lo CC libee_la-event.lo CC libee_la-json_event.lo CC libee_la-value.lo value.c: In function 'ee_newValue': value.c:37: warning: unused parameter 'ctx' CC libee_la-tagbucket.lo CC libee_la-field.lo field.c: In function 'ee_getFieldValueAsStr': field.c:182: warning: 'str' may be used uninitialized in this function CC libee_la-fieldbucket.lo CC libee_la-primitivetype.lo CC libee_la-int_dec.lo CC libee_la-json_dec.lo CC libee_la-apache_dec.lo apache_dec.c: In function 'ee_newApache': apache_dec.c:37: warning: unused parameter 'ctx' apache_dec.c: In function 'ee_apacheAddName': apache_dec.c:71: warning: unused parameter 'ctx' apache_dec.c: In function 'processLn': apache_dec.c:205: warning: unused variable 'value' apache_dec.c: In function 'ee_apacheDec': apache_dec.c:143: warning: 'r' may be used uninitialized in this function CC libee_la-syslog_enc.lo CC libee_la-json_enc.lo CC libee_la-csv_enc.lo csv_enc.c: In function 'ee_AddName': csv_enc.c:66: warning: unused parameter 'ctx' CC libee_la-xml_enc.lo xml_enc.c: In function 'ee_addValue_XML': xml_enc.c:60: warning: unused variable 'j' xml_enc.c:59: warning: unused variable 'numbuf' xml_enc.c: At top level: xml_enc.c:40: warning: 'hexdigit' defined but not used CCLD libee.la CC libee_convert-convert.o CCLD libee-convert libee_convert-convert.o: In function `cbNewEvt': /usr/local/src/libee-0.4.0/src/convert.c:109: undefined reference to `es_str2cstr' /usr/local/src/libee-0.4.0/src/convert.c:112: undefined reference to `es_deleteStr' /usr/local/src/libee-0.4.0/src/convert.c:114: undefined reference to `es_str2cstr' /usr/local/src/libee-0.4.0/src/convert.c:117: undefined reference to `es_deleteStr' /usr/local/src/libee-0.4.0/src/convert.c:119: undefined reference to `es_str2cstr' /usr/local/src/libee-0.4.0/src/convert.c:122: undefined reference to `es_deleteStr' /usr/local/src/libee-0.4.0/src/convert.c:86: undefined reference to `es_str2cstr' /usr/local/src/libee-0.4.0/src/convert.c:89: undefined reference to `es_deleteStr' libee_convert-convert.o: In function `cbGetLine': /usr/local/src/libee-0.4.0/src/convert.c:151: undefined reference to `es_newStrFromCStr' /usr/local/src/libee-0.4.0/src/convert.c:156: undefined reference to `es_unescapeStr' libee_convert-convert.o: In function `main': /usr/local/src/libee-0.4.0/src/convert.c:218: undefined reference to `es_newStrFromCStr' /usr/local/src/libee-0.4.0/src/convert.c:221: undefined reference to `es_newStrFromCStr' /usr/local/src/libee-0.4.0/src/convert.c:248: undefined reference to `es_str2cstr' /usr/local/src/libee-0.4.0/src/convert.c:263: undefined reference to `es_str2cstr' ../src/.libs/libee.so: undefined reference to `es_addBuf' ../src/.libs/libee.so: undefined reference to `es_strbufcmp' ../src/.libs/libee.so: undefined reference to `es_newStrFromSubStr' ../src/.libs/libee.so: undefined reference to `es_addChar' ../src/.libs/libee.so: undefined reference to `es_newStr' collect2: ld returned 1 exit status make[2]: *** [libee-convert] Error 1 make[2]: Leaving directory `/usr/local/src/libee-0.4.0/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/local/src/libee-0.4.0' make: *** [all] Error 2 Any ideas what I may be doing wrong? much thanks, Gary -------------- next part -------------- An HTML attachment was scrubbed... URL: