From rgerhards at hq.adiscon.com Sun Aug 26 14:56:33 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 26 Aug 2012 12:56:33 +0000 Subject: [Lognorm] liblognorm API changes Message-ID: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> Hi all, as you probably know, liblognorm uses libee as its "base data model library". Libee was created based on a now quite old CEE spec, on that has considerably changed in the mean time. Today, it looks like a simple JSON object model would take care of the CEE needs - and many more. Rsyslog is also moving towards that model. So I am tempted to modify the next major version of liblognorm to no longer be based on libee but rather directly on the json-c (often packaged as libjson0) library object model. That would of course mean that in order to use new versions the caller apps needs to be modified. I wouldn't expect that much code needs to be changed, but that of course depends. I have not made up my mind. But I'd appreciate feedback from liblognorm users (Champ? ;)) as one of the decision factors. Thanks, Rainer From cclark at quadrantsec.com Tue Aug 28 16:49:15 2012 From: cclark at quadrantsec.com (Champ Clark III) Date: Tue, 28 Aug 2012 10:49:15 -0400 Subject: [Lognorm] liblognorm API changes In-Reply-To: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> Message-ID: <503CDA6B.7030503@quadrantsec.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm fine with that Rainer. As I understand it, we'll just drop the libee dependency and add json-c as a dependency. Correct? From the API, what sort of changes do you see. I'm also not 100% sure how json-c (which Sagan uses in some cases) and liblognorm will work together. Can you briefly explain? Thank you Rainer. I'll certainly modify Sagan to work with the new API as you see fit. Thanks for the hard work. On 8/26/12 8:56 AM, Rainer Gerhards wrote: > Hi all, > > as you probably know, liblognorm uses libee as its "base data model > library". Libee was created based on a now quite old CEE spec, on > that has considerably changed in the mean time. > > Today, it looks like a simple JSON object model would take care of > the CEE needs - and many more. Rsyslog is also moving towards that > model. So I am tempted to modify the next major version of > liblognorm to no longer be based on libee but rather directly on > the json-c (often packaged as libjson0) library object model. That > would of course mean that in order to use new versions the caller > apps needs to be modified. I wouldn't expect that much code needs > to be changed, but that of course depends. > > I have not made up my mind. But I'd appreciate feedback from > liblognorm users (Champ? ;)) as one of the decision factors. > > Thanks, Rainer _______________________________________________ > Lognorm mailing list Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm > - -- - - Champ Clark III (cclark at quadrantsec.com) Quadrant Information Security (http://quadrantsec.com) Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A GPG Key ID: 0381878A -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJQPNprAAoJENnmXt7Lmc3KsPAH/24+P+U9lA49LIWAdZopCa88 4mK7fX4Fh7ikvvRthxrzPCe9A4+ZYsoeFaal721LGHd+F0f7VohArBtmkgNx88IF 0T9VsLVlKZXzgiV378hL+o68jNziTOskZ4OWIx9RnW3xf31ctyQbfKYtuprGWLQz 0vv56M2/xXZRyeF7poVz09SntSk3Bho+zMI6J5dyNHIuWQhi6cBObQXUzB+ew1sb C9moWjlbbNUmLYPDeXZ0qaqS5rsn+2RcU1zXYXSJBLM/G1b6yvw/andXfPc0GkqM fuOBIkgLs4BN9W+VJpDsoEGYOoCA3nfSbcWGR7LBJIjyB5kkxohSGCOTEfAW/U0= =WYTd -----END PGP SIGNATURE----- From jlay at slave-tothe-box.net Tue Aug 28 16:58:52 2012 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 28 Aug 2012 08:58:52 -0600 Subject: [Lognorm] liblognorm API changes In-Reply-To: <503CDA6B.7030503@quadrantsec.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> <503CDA6B.7030503@quadrantsec.com> Message-ID: <7f879f72f03ef0e976fa26c99cfe2965@localhost> Count me in for testing in a real world environment when needed :) James On 2012-08-28 08:49, Champ Clark III wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm fine with that Rainer. As I understand it, we'll just drop the > libee dependency and add json-c as a dependency. Correct? From the > API, what sort of changes do you see. I'm also not 100% sure how > json-c (which Sagan uses in some cases) and liblognorm will work > together. Can you briefly explain? Thank you Rainer. > > I'll certainly modify Sagan to work with the new API as you see fit. > Thanks for the hard work. > > > On 8/26/12 8:56 AM, Rainer Gerhards wrote: >> Hi all, >> >> as you probably know, liblognorm uses libee as its "base data model >> library". Libee was created based on a now quite old CEE spec, on >> that has considerably changed in the mean time. >> >> Today, it looks like a simple JSON object model would take care of >> the CEE needs - and many more. Rsyslog is also moving towards that >> model. So I am tempted to modify the next major version of >> liblognorm to no longer be based on libee but rather directly on >> the json-c (often packaged as libjson0) library object model. That >> would of course mean that in order to use new versions the caller >> apps needs to be modified. I wouldn't expect that much code needs >> to be changed, but that of course depends. >> >> I have not made up my mind. But I'd appreciate feedback from >> liblognorm users (Champ? ;)) as one of the decision factors. >> >> Thanks, Rainer _______________________________________________ >> Lognorm mailing list Lognorm at lists.adiscon.com >> http://lists.adiscon.net/mailman/listinfo/lognorm >> > > > - -- > - - Champ Clark III (cclark at quadrantsec.com) > Quadrant Information Security (http://quadrantsec.com) > Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A > GPG Key ID: 0381878A > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJQPNprAAoJENnmXt7Lmc3KsPAH/24+P+U9lA49LIWAdZopCa88 > 4mK7fX4Fh7ikvvRthxrzPCe9A4+ZYsoeFaal721LGHd+F0f7VohArBtmkgNx88IF > 0T9VsLVlKZXzgiV378hL+o68jNziTOskZ4OWIx9RnW3xf31ctyQbfKYtuprGWLQz > 0vv56M2/xXZRyeF7poVz09SntSk3Bho+zMI6J5dyNHIuWQhi6cBObQXUzB+ew1sb > C9moWjlbbNUmLYPDeXZ0qaqS5rsn+2RcU1zXYXSJBLM/G1b6yvw/andXfPc0GkqM > fuOBIkgLs4BN9W+VJpDsoEGYOoCA3nfSbcWGR7LBJIjyB5kkxohSGCOTEfAW/U0= > =WYTd > -----END PGP SIGNATURE----- > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From rgerhards at hq.adiscon.com Tue Aug 28 18:11:33 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Aug 2012 16:11:33 +0000 Subject: [Lognorm] liblognorm API changes In-Reply-To: <503CDA6B.7030503@quadrantsec.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> <503CDA6B.7030503@quadrantsec.com> Message-ID: <82D3B6EBB984E84DACE6028384F75FB903615C@vmexch2.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III > Sent: Tuesday, August 28, 2012 4:49 PM > To: lognorm at lists.adiscon.com > Subject: Re: [Lognorm] liblognorm API changes > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm fine with that Rainer. As I understand it, we'll just drop the > libee dependency and add json-c as a dependency. Correct? Yes, but... > From the > API, what sort of changes do you see. You currently use libee API to access the data! That would now change. > I'm also not 100% sure how > json-c (which Sagan uses in some cases) and liblognorm will work > together. Can you briefly explain? Thank you Rainer. In essence, you would call the normalizer "as usual". On return, however, you'd receive no longer an libee event, but a json-c json doc. So you would pull data out of the json tree. Note that this is currently at the idea stage. It breaks a couple of things in the libee/lognorm infrastructure, and I am still not totally sure how to best handle it. However, from your PoV the bottom line is that access to the normalized properties changes. I have not definite schedule, depends on the other work going on. Let me know any questions and concerns. Rainer > > I'll certainly modify Sagan to work with the new API as you see fit. > Thanks for the hard work. > > > On 8/26/12 8:56 AM, Rainer Gerhards wrote: > > Hi all, > > > > as you probably know, liblognorm uses libee as its "base data model > > library". Libee was created based on a now quite old CEE spec, on > > that has considerably changed in the mean time. > > > > Today, it looks like a simple JSON object model would take care of > > the CEE needs - and many more. Rsyslog is also moving towards that > > model. So I am tempted to modify the next major version of > > liblognorm to no longer be based on libee but rather directly on > > the json-c (often packaged as libjson0) library object model. That > > would of course mean that in order to use new versions the caller > > apps needs to be modified. I wouldn't expect that much code needs > > to be changed, but that of course depends. > > > > I have not made up my mind. But I'd appreciate feedback from > > liblognorm users (Champ? ;)) as one of the decision factors. > > > > Thanks, Rainer _______________________________________________ > > Lognorm mailing list Lognorm at lists.adiscon.com > > http://lists.adiscon.net/mailman/listinfo/lognorm > > > > > - -- > - - Champ Clark III (cclark at quadrantsec.com) > Quadrant Information Security (http://quadrantsec.com) > Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A > GPG Key ID: 0381878A > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJQPNprAAoJENnmXt7Lmc3KsPAH/24+P+U9lA49LIWAdZopCa88 > 4mK7fX4Fh7ikvvRthxrzPCe9A4+ZYsoeFaal721LGHd+F0f7VohArBtmkgNx88IF > 0T9VsLVlKZXzgiV378hL+o68jNziTOskZ4OWIx9RnW3xf31ctyQbfKYtuprGWLQz > 0vv56M2/xXZRyeF7poVz09SntSk3Bho+zMI6J5dyNHIuWQhi6cBObQXUzB+ew1sb > C9moWjlbbNUmLYPDeXZ0qaqS5rsn+2RcU1zXYXSJBLM/G1b6yvw/andXfPc0GkqM > fuOBIkgLs4BN9W+VJpDsoEGYOoCA3nfSbcWGR7LBJIjyB5kkxohSGCOTEfAW/U0= > =WYTd > -----END PGP SIGNATURE----- > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From cclark at quadrantsec.com Wed Aug 29 18:35:04 2012 From: cclark at quadrantsec.com (Champ Clark III) Date: Wed, 29 Aug 2012 12:35:04 -0400 Subject: [Lognorm] liblognorm API changes In-Reply-To: <82D3B6EBB984E84DACE6028384F75FB903615C@vmexch2.intern.adiscon.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> <503CDA6B.7030503@quadrantsec.com> <82D3B6EBB984E84DACE6028384F75FB903615C@vmexch2.intern.adiscon.com> Message-ID: <503E44B8.2060907@quadrantsec.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Okay. I think I understand now. So rather than libee handing back the element, liblognorm will return JSON back. So, software like Sagan, will need to use json-c to parse and extract the value(s) it needs from the returned liblognorm JSON string. Correct? The only issue I have is that the current API makes it extremely simple to pull the values needed. The json-c stuff will add some extra overhead (granted, not very much at all). Think of it this way.. I'm using liblognorm to "parse" and "extract" values out of a syslog line. Now, I need to "parse" and "extract" values of that syslog line information from a JSON format :) However, it's really not a big deal either way. If it'll improve the liblognorm and make it more useable for more people, then I'm all for it. - -- - - Champ Clark III (cclark at quadrantsec.com) Quadrant Information Security (http://quadrantsec.com) Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A GPG Key ID: 0381878A -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQPkS4AAoJENnmXt7Lmc3KdAUH/0gVilpH52AxTP7EitUlk7I+ kYUE9pVAbldZJwClmZiZbfr0yWjXf0MxZFUWxqK2ai+jd1oaEKwpWsHkYYyXYtVg ZIRBhRS79zUcRhYNplF9ZLBGvVvX95DRcfcKHt8Hnafv1ZzEpqGiOL1GMBfOadha N4i7xeQ+fPFqeJQwnEcgJv+HPSJVhA+9auHTJ3uNWel6eV41umiZXKAKsLl68wbl eqEDUgw/U++TqwP7wYLTzF0IIHwPPzL/8oo+VklMcv1t/pJZs4r5SQpTSHt4+V5K Ztmj8gxLS1rdjf9aD2RZvwNDR672LXE7bvRDYLItsPCwp693YUghvVR3Iz5oyZM= =Xos6 -----END PGP SIGNATURE----- From rgerhards at hq.adiscon.com Wed Aug 29 19:13:31 2012 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Aug 2012 17:13:31 +0000 Subject: [Lognorm] liblognorm API changes In-Reply-To: <503E44B8.2060907@quadrantsec.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> <503CDA6B.7030503@quadrantsec.com> <82D3B6EBB984E84DACE6028384F75FB903615C@vmexch2.intern.adiscon.com> <503E44B8.2060907@quadrantsec.com> Message-ID: <82D3B6EBB984E84DACE6028384F75FB90375A2@vmexch2.intern.adiscon.com> > -----Original Message----- > From: lognorm-bounces at lists.adiscon.com [mailto:lognorm- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III > Sent: Wednesday, August 29, 2012 6:35 PM > To: lognorm at lists.adiscon.com > Subject: Re: [Lognorm] liblognorm API changes > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Okay. I think I understand now. So rather than libee handing back > the element, liblognorm will return JSON back. So, software like > Sagan, will need to use json-c to parse and extract the value(s) it > needs from the returned liblognorm JSON string. Correct? Not 100% ;). It would not return a JSON *string* but rather a json-c "struct json_object*" ptr, which you can access via the json-c API. So you do not need to re-parse it with json-c. > The only issue I have is that the current API makes it extremely > simple to pull the values needed. The json-c stuff will add some > extra overhead (granted, not very much at all). Anyway, I'll verify that I do not add extra complexity. I have not yet had a deep look at the changes. My impression was that the change would probably require one third *less* API calls to extract the data (that is what happens in rsyslog v7 when I access json-c objects vs. libee events). If that's not the case, I'll probably wrap the old API (assuming that the additional layer does not cause considerable overhead). That would also solve a couple of other issues I see within the whole subsystem (but I usually prefer to get along without additional wrapper layers). Thanks again for the good thoughts and pls let them continue to flow! Rainer > Think of it this way.. I'm using liblognorm to "parse" and "extract" > values out of a syslog line. Now, I need to "parse" and "extract" > values of that syslog line information from a JSON format :) > > However, it's really not a big deal either way. If it'll improve the > liblognorm and make it more useable for more people, then I'm all for > it. > > > - -- > - - Champ Clark III (cclark at quadrantsec.com) > Quadrant Information Security (http://quadrantsec.com) > Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A > GPG Key ID: 0381878A > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJQPkS4AAoJENnmXt7Lmc3KdAUH/0gVilpH52AxTP7EitUlk7I+ > kYUE9pVAbldZJwClmZiZbfr0yWjXf0MxZFUWxqK2ai+jd1oaEKwpWsHkYYyXYtVg > ZIRBhRS79zUcRhYNplF9ZLBGvVvX95DRcfcKHt8Hnafv1ZzEpqGiOL1GMBfOadha > N4i7xeQ+fPFqeJQwnEcgJv+HPSJVhA+9auHTJ3uNWel6eV41umiZXKAKsLl68wbl > eqEDUgw/U++TqwP7wYLTzF0IIHwPPzL/8oo+VklMcv1t/pJZs4r5SQpTSHt4+V5K > Ztmj8gxLS1rdjf9aD2RZvwNDR672LXE7bvRDYLItsPCwp693YUghvVR3Iz5oyZM= > =Xos6 > -----END PGP SIGNATURE----- > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm From cclark at quadrantsec.com Wed Aug 29 18:52:33 2012 From: cclark at quadrantsec.com (Champ Clark III) Date: Wed, 29 Aug 2012 12:52:33 -0400 Subject: [Lognorm] liblognorm API changes In-Reply-To: <82D3B6EBB984E84DACE6028384F75FB90375A2@vmexch2.intern.adiscon.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> <503CDA6B.7030503@quadrantsec.com> <82D3B6EBB984E84DACE6028384F75FB903615C@vmexch2.intern.adiscon.com> <503E44B8.2060907@quadrantsec.com> <82D3B6EBB984E84DACE6028384F75FB90375A2@vmexch2.intern.adiscon.com> Message-ID: <503E48D1.8030100@quadrantsec.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Okay. I see. I've had only limited interaction with json-c when working on a Sagan preprocessor that made external calls (TCP) to a database (IP/URL reputation database) that returned JSON. I used json-c to extract fields Sagan needed. If I'm incorrect and makes it _less_ complex to deal with, then I'm all for it. From a Sagan dependencies, it really doesn't add anything "extra". That is, it sounds like you're replacing libee with json-c. I guess from here it just requires some testing and proof of concept code :) - -- - - Champ Clark III (cclark at quadrantsec.com) Quadrant Information Security (http://quadrantsec.com) Key Fingerprint: 2E56 C2EB 1B25 C517 D5BA 2DCF 5E70 B2F8 0381 878A GPG Key ID: 0381878A -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQPkjRAAoJENnmXt7Lmc3Kc+gH/0K6UTYtBsUUAP4fWabpP3yk G7LJWE768AiYRF52gtOLzWBiV3D5CBLRN6xe5C1hzrEf7iqDTaHKb4QEFRctHird 4hABdrj6Oq3IVg70cHqbMesZGVLHwRVvhm2ukKjV5Nj9YfJiJH53kF21dSaiE58s Ahuygmws8Dkddl8JSwKfbbnD1uaq8GSmU63eBldrNb8xHv0xhyAEDn35tBdoGa/v LbuO9Z1G1JAGfU/lQ8ubw1J6n5+0DcUBsvruoZwBAjoNnZfZs9Ig3Of5JI2gbjMs vo8BQz+lYxUmpX8KjnUy4HlFThCnTH6A01n5OPGWB7sobIdJnKb6I0zPtiHXb9A= =Dqln -----END PGP SIGNATURE----- From david at lang.hm Wed Aug 29 22:32:38 2012 From: david at lang.hm (david at lang.hm) Date: Wed, 29 Aug 2012 13:32:38 -0700 (PDT) Subject: [Lognorm] liblognorm API changes In-Reply-To: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> References: <82D3B6EBB984E84DACE6028384F75FB9032A32@vmexch2.intern.adiscon.com> Message-ID: On Sun, 26 Aug 2012, Rainer Gerhards wrote: > Hi all, > > as you probably know, liblognorm uses libee as its "base data model > library". Libee was created based on a now quite old CEE spec, on that > has considerably changed in the mean time. > > Today, it looks like a simple JSON object model would take care of the > CEE needs - and many more. Rsyslog is also moving towards that model. So > I am tempted to modify the next major version of liblognorm to no longer > be based on libee but rather directly on the json-c (often packaged as > libjson0) library object model. That would of course mean that in order > to use new versions the caller apps needs to be modified. I wouldn't > expect that much code needs to be changed, but that of course depends. > > I have not made up my mind. But I'd appreciate feedback from liblognorm > users (Champ? ;)) as one of the decision factors. This changes liblognorm from a generic parsing library to a JSON parsing library. That may be a very common case for the expected use, but I think it's far from being the only case for this library. I think the key thing that you are looking to support going forward that was not supported before is hierarctical structure support. Other than the "assume it's JSON and parse accordingly", what sort of syntax are you thinking of using to define nesting in the pattern matching? What do will you do if the nesting doesn't match JSON (multiple things at the same level with the same name, arrays, etc) David Lang