From singh.janmejay at gmail.com  Mon Nov  3 10:26:37 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Mon, 3 Nov 2014 14:56:37 +0530
Subject: [Lognorm] Any reason we should not support regex-based field-type?
Message-ID: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>

I am thinking of it as a 2nd class field-type.

By that I mean, one gets best performance from 1st class supported
field-types, but if for some reason that is not sufficient for someone,
they can use a regex-field-type. It may be a little low on performance, but
then it unblocks people immediately.

I can do it, just need to know we are not ideologically against it.

Kind of construct im thinking of:

%foo:regex:[a-f0-9]+% to match hex-numbers for instance.

Thoughts?

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/4cca9a6d/attachment.html>

From rgerhards at hq.adiscon.com  Mon Nov  3 10:35:11 2014
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 3 Nov 2014 10:35:11 +0100
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
Message-ID: <CADk+mPAaj01GzKd_bg1NddedFKhXEHKMA_xvuvziv+kHiz16eQ@mail.gmail.com>

2014-11-03 10:26 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:

> I am thinking of it as a 2nd class field-type.
>
> By that I mean, one gets best performance from 1st class supported
> field-types, but if for some reason that is not sufficient for someone,
> they can use a regex-field-type. It may be a little low on performance, but
> then it unblocks people immediately.
>
> I can do it, just need to know we are not ideologically against it.
>
>
I really don't like it, because I fear that people will always use regexp,
and so this get's pretty low by default. But I think I've lost that war in
any case ;) So it's OK for me if you implement it - actually, I'd even be a
little bit happy ;-) .

Rainer

Kind of construct im thinking of:
>
> %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>
> Thoughts?
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/6d07bc15/attachment-0001.html>

From singh.janmejay at gmail.com  Mon Nov  3 10:59:37 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Mon, 3 Nov 2014 15:29:37 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <CADk+mPAaj01GzKd_bg1NddedFKhXEHKMA_xvuvziv+kHiz16eQ@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<CADk+mPAaj01GzKd_bg1NddedFKhXEHKMA_xvuvziv+kHiz16eQ@mail.gmail.com>
Message-ID: <CAGB1VvyzBdE-va5Nv-5CxO3zHNDq9ELpH9d43eH=npostSP+Xg@mail.gmail.com>

Cool, i'll do it.

On Mon, Nov 3, 2014 at 3:05 PM, Rainer Gerhards <rgerhards at hq.adiscon.com>
wrote:

> 2014-11-03 10:26 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>
>> I am thinking of it as a 2nd class field-type.
>>
>> By that I mean, one gets best performance from 1st class supported
>> field-types, but if for some reason that is not sufficient for someone,
>> they can use a regex-field-type. It may be a little low on performance, but
>> then it unblocks people immediately.
>>
>> I can do it, just need to know we are not ideologically against it.
>>
>>
> I really don't like it, because I fear that people will always use regexp,
> and so this get's pretty low by default. But I think I've lost that war in
> any case ;) So it's OK for me if you implement it - actually, I'd even be a
> little bit happy ;-) .
>
> Rainer
>
> Kind of construct im thinking of:
>>
>> %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>>
>> Thoughts?
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/136d3b23/attachment.html>

From david at lang.hm  Mon Nov  3 13:07:07 2014
From: david at lang.hm (David Lang)
Date: Mon, 3 Nov 2014 04:07:07 -0800 (PST)
Subject: [Lognorm] Any reason we should not support regex-based
 field-type?
In-Reply-To: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>

On Mon, 3 Nov 2014, singh.janmejay wrote:

> I am thinking of it as a 2nd class field-type.
>
> By that I mean, one gets best performance from 1st class supported
> field-types, but if for some reason that is not sufficient for someone,
> they can use a regex-field-type. It may be a little low on performance, but
> then it unblocks people immediately.
>
> I can do it, just need to know we are not ideologically against it.
>
> Kind of construct im thinking of:
>
> %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>
> Thoughts?

Since this will be such a performance pig compared to the existing parse tree, 
how about requiring a 'enable low performance types' flag or something like that 
to enable it?

There needs to be some good indicator that this is a performance problem.

David Lang
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From singh.janmejay at gmail.com  Mon Nov  3 13:15:46 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Mon, 3 Nov 2014 17:45:46 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
Message-ID: <CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>

We log a warning?

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm> wrote:

> On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>  I am thinking of it as a 2nd class field-type.
>>
>> By that I mean, one gets best performance from 1st class supported
>> field-types, but if for some reason that is not sufficient for someone,
>> they can use a regex-field-type. It may be a little low on performance,
>> but
>> then it unblocks people immediately.
>>
>> I can do it, just need to know we are not ideologically against it.
>>
>> Kind of construct im thinking of:
>>
>> %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>>
>> Thoughts?
>>
>
> Since this will be such a performance pig compared to the existing parse
> tree, how about requiring a 'enable low performance types' flag or
> something like that to enable it?
>
> There needs to be some good indicator that this is a performance problem.
>
> David Lang
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/972dfe9a/attachment.html>

From cclark at quadrantsec.com  Mon Nov  3 16:52:20 2014
From: cclark at quadrantsec.com (Champ Clark III)
Date: Mon, 03 Nov 2014 10:52:20 -0500
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
Message-ID: <5457A4B4.50306@quadrantsec.com>


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think David means something like:

./configure --enable-preformance-pigs

I'm not a huge fan of this for a couple of reasons.

In your example,  you want to find hex values.  I'd rather see a lognorm
"parser" created for this purpose.   I'm afraid if we start with regular
expressions,  we'll end up mixing rules (RE and non-RE) and it will make
things very confusing.  I really like the way lognorm's "masking"
works.   I'd rather not see that break.



On 11/03/2014 07:15 AM, singh.janmejay wrote:
>
> We log a warning?
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
>   
>
> On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
<mailto:david at lang.hm>> wrote:
>
>     On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>         I am thinking of it as a 2nd class field-type.
>
>         By that I mean, one gets best performance from 1st class supported
>         field-types, but if for some reason that is not sufficient for
someone,
>         they can use a regex-field-type. It may be a little low on
performance, but
>         then it unblocks people immediately.
>
>         I can do it, just need to know we are not ideologically
against it.
>
>         Kind of construct im thinking of:
>
>         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>
>         Thoughts?
>
>
>     Since this will be such a performance pig compared to the existing
parse tree, how about requiring a 'enable low performance types' flag or
something like that to enable it?
>
>     There needs to be some good indicator that this is a performance
problem.
>
>     David Lang
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
>
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm


- -- 
- - Quadrant Information Security
  Champ Clark III
  o: 800.538.9357 x 101
  c: 850.443.2440
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
=rGsk
-----END PGP SIGNATURE-----

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/be7aa8de/attachment.html>

From singh.janmejay at gmail.com  Mon Nov  3 18:22:25 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Mon, 3 Nov 2014 22:52:25 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <5457A4B4.50306@quadrantsec.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
	<5457A4B4.50306@quadrantsec.com>
Message-ID: <CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>

Hex was just an example. I meant when someone needs to roll out something
very quickly, and it's not supported in the build they have, it's useful to
be able to do it using regex in rulebase but still completely working in
terms of rules, rather than having to resort to a mix of rules and
exec-template calls with regexp based property extractors etc.

It's still the same amount of work, if someone is trying to do something
unsupported by lognorm. So performance effects will still show, just not in
lognorm.

This allows for flexibility and clean way of parsing logs at possibly even
slightly lesser cost then the former approach.

We can support a switch-on flag in conf file(module loading call, may be)
which can be used to enable performance sensitive features.

I personally think warning in logs would be enough though.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 3, 2014 9:22 PM, "Champ Clark III" <cclark at quadrantsec.com> wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I think David means something like:
>
> ./configure --enable-preformance-pigs
>
> I'm not a huge fan of this for a couple of reasons.
>
> In your example,  you want to find hex values.  I'd rather see a lognorm
> "parser" created for this purpose.   I'm afraid if we start with regular
> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
> things very confusing.  I really like the way lognorm's "masking" works.
> I'd rather not see that break.
>
>
>
> On 11/03/2014 07:15 AM, singh.janmejay wrote:
> >
> > We log a warning?
> >
> > --
> > Regards,
> > Janmejay
> >
> > PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
> >
> >
> > On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
> <mailto:david at lang.hm> <david at lang.hm>> wrote:
> >
> >     On Mon, 3 Nov 2014, singh.janmejay wrote:
> >
> >         I am thinking of it as a 2nd class field-type.
> >
> >         By that I mean, one gets best performance from 1st class
> supported
> >         field-types, but if for some reason that is not sufficient for
> someone,
> >         they can use a regex-field-type. It may be a little low on
> performance, but
> >         then it unblocks people immediately.
> >
> >         I can do it, just need to know we are not ideologically against
> it.
> >
> >         Kind of construct im thinking of:
> >
> >         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
> >
> >         Thoughts?
> >
> >
> >     Since this will be such a performance pig compared to the existing
> parse tree, how about requiring a 'enable low performance types' flag or
> something like that to enable it?
> >
> >     There needs to be some good indicator that this is a performance
> problem.
> >
> >     David Lang
> >     _______________________________________________
> >     Lognorm mailing list
> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>
> >     http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> >     _______________________________________________
> >     Lognorm mailing list
> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>
> >     http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> >
> >
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
> - --
> - - Quadrant Information Security
>   Champ Clark III
>   o: 800.538.9357 x 101
>   c: 850.443.2440
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
>
> iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
> U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
> Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
> 1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
> OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
> AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
> =rGsk
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/d3a09063/attachment-0001.html>

From singh.janmejay at gmail.com  Mon Nov  3 18:31:50 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Mon, 3 Nov 2014 23:01:50 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
	<5457A4B4.50306@quadrantsec.com>
	<CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>
Message-ID: <CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>

I think module-loading call having a flag to turn on/off would be better
than build-time flag, purely because it allows one to build a package and
use it on several boxes, while keeping the feature off when not required. A
more fine grained control, in that sense.

We can support both levels of enabling, but then a lot of people may be
confused as to why it doesn't work when they turned it on while building.

2 levels also seems a little over-protective to me. Kinda like Windows "do
you really want to delete this file" prompts.

Does module-loading time flag sound ok?

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 3, 2014 10:52 PM, "singh.janmejay" <singh.janmejay at gmail.com> wrote:

> Hex was just an example. I meant when someone needs to roll out something
> very quickly, and it's not supported in the build they have, it's useful to
> be able to do it using regex in rulebase but still completely working in
> terms of rules, rather than having to resort to a mix of rules and
> exec-template calls with regexp based property extractors etc.
>
> It's still the same amount of work, if someone is trying to do something
> unsupported by lognorm. So performance effects will still show, just not in
> lognorm.
>
> This allows for flexibility and clean way of parsing logs at possibly even
> slightly lesser cost then the former approach.
>
> We can support a switch-on flag in conf file(module loading call, may be)
> which can be used to enable performance sensitive features.
>
> I personally think warning in logs would be enough though.
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
>
> On Nov 3, 2014 9:22 PM, "Champ Clark III" <cclark at quadrantsec.com> wrote:
>
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I think David means something like:
>>
>> ./configure --enable-preformance-pigs
>>
>> I'm not a huge fan of this for a couple of reasons.
>>
>> In your example,  you want to find hex values.  I'd rather see a lognorm
>> "parser" created for this purpose.   I'm afraid if we start with regular
>> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
>> things very confusing.  I really like the way lognorm's "masking" works.
>> I'd rather not see that break.
>>
>>
>>
>> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>> >
>> > We log a warning?
>> >
>> > --
>> > Regards,
>> > Janmejay
>> >
>> > PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>> >
>> >
>> > On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
>> <mailto:david at lang.hm> <david at lang.hm>> wrote:
>> >
>> >     On Mon, 3 Nov 2014, singh.janmejay wrote:
>> >
>> >         I am thinking of it as a 2nd class field-type.
>> >
>> >         By that I mean, one gets best performance from 1st class
>> supported
>> >         field-types, but if for some reason that is not sufficient for
>> someone,
>> >         they can use a regex-field-type. It may be a little low on
>> performance, but
>> >         then it unblocks people immediately.
>> >
>> >         I can do it, just need to know we are not ideologically against
>> it.
>> >
>> >         Kind of construct im thinking of:
>> >
>> >         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>> >
>> >         Thoughts?
>> >
>> >
>> >     Since this will be such a performance pig compared to the existing
>> parse tree, how about requiring a 'enable low performance types' flag or
>> something like that to enable it?
>> >
>> >     There needs to be some good indicator that this is a performance
>> problem.
>> >
>> >     David Lang
>> >     _______________________________________________
>> >     Lognorm mailing list
>> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>> <Lognorm at lists.adiscon.com>
>> >     http://lists.adiscon.net/mailman/listinfo/lognorm
>> >
>> >     _______________________________________________
>> >     Lognorm mailing list
>> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>> <Lognorm at lists.adiscon.com>
>> >     http://lists.adiscon.net/mailman/listinfo/lognorm
>> >
>> >
>> >
>> > _______________________________________________
>> > Lognorm mailing list
>> > Lognorm at lists.adiscon.com
>> > http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>> - --
>> - - Quadrant Information Security
>>   Champ Clark III
>>   o: 800.538.9357 x 101
>>   c: 850.443.2440
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.11 (GNU/Linux)
>>
>> iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
>> U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
>> Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
>> 1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
>> OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
>> AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
>> =rGsk
>> -----END PGP SIGNATURE-----
>>
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/8a0e2d95/attachment.html>

From david at lang.hm  Mon Nov  3 23:12:09 2014
From: david at lang.hm (David Lang)
Date: Mon, 3 Nov 2014 14:12:09 -0800 (PST)
Subject: [Lognorm] Any reason we should not support regex-based
 field-type?
In-Reply-To: <CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
	<5457A4B4.50306@quadrantsec.com>
	<CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>
	<CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1411031410150.31001@nftneq.ynat.uz>

On Mon, 3 Nov 2014, singh.janmejay wrote:

> I think module-loading call having a flag to turn on/off would be better
> than build-time flag, purely because it allows one to build a package and
> use it on several boxes, while keeping the feature off when not required. A
> more fine grained control, in that sense.

I was thinking in terms of something in the config file. We need it to be 
something that the person configuring rsyslog will see so that they know they 
are doing something slow. Most users don't compile rsyslog so they won't ever 
see the compile time flag.

module loading time seems fine.

David Lang

> We can support both levels of enabling, but then a lot of people may be
> confused as to why it doesn't work when they turned it on while building.
>
> 2 levels also seems a little over-protective to me. Kinda like Windows "do
> you really want to delete this file" prompts.
>
> Does module-loading time flag sound ok?
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
>
> On Nov 3, 2014 10:52 PM, "singh.janmejay" <singh.janmejay at gmail.com> wrote:
>
>> Hex was just an example. I meant when someone needs to roll out something
>> very quickly, and it's not supported in the build they have, it's useful to
>> be able to do it using regex in rulebase but still completely working in
>> terms of rules, rather than having to resort to a mix of rules and
>> exec-template calls with regexp based property extractors etc.
>>
>> It's still the same amount of work, if someone is trying to do something
>> unsupported by lognorm. So performance effects will still show, just not in
>> lognorm.
>>
>> This allows for flexibility and clean way of parsing logs at possibly even
>> slightly lesser cost then the former approach.
>>
>> We can support a switch-on flag in conf file(module loading call, may be)
>> which can be used to enable performance sensitive features.
>>
>> I personally think warning in logs would be enough though.
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Nov 3, 2014 9:22 PM, "Champ Clark III" <cclark at quadrantsec.com> wrote:
>>
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> I think David means something like:
>>>
>>> ./configure --enable-preformance-pigs
>>>
>>> I'm not a huge fan of this for a couple of reasons.
>>>
>>> In your example,  you want to find hex values.  I'd rather see a lognorm
>>> "parser" created for this purpose.   I'm afraid if we start with regular
>>> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
>>> things very confusing.  I really like the way lognorm's "masking" works.
>>> I'd rather not see that break.
>>>
>>>
>>>
>>> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>>>>
>>>> We log a warning?
>>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>>
>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>>
>>>> On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
>>> <mailto:david at lang.hm> <david at lang.hm>> wrote:
>>>>
>>>>     On Mon, 3 Nov 2014, singh.janmejay wrote:
>>>>
>>>>         I am thinking of it as a 2nd class field-type.
>>>>
>>>>         By that I mean, one gets best performance from 1st class
>>> supported
>>>>         field-types, but if for some reason that is not sufficient for
>>> someone,
>>>>         they can use a regex-field-type. It may be a little low on
>>> performance, but
>>>>         then it unblocks people immediately.
>>>>
>>>>         I can do it, just need to know we are not ideologically against
>>> it.
>>>>
>>>>         Kind of construct im thinking of:
>>>>
>>>>         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>>>>
>>>>         Thoughts?
>>>>
>>>>
>>>>     Since this will be such a performance pig compared to the existing
>>> parse tree, how about requiring a 'enable low performance types' flag or
>>> something like that to enable it?
>>>>
>>>>     There needs to be some good indicator that this is a performance
>>> problem.
>>>>
>>>>     David Lang
>>>>     _______________________________________________
>>>>     Lognorm mailing list
>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>> <Lognorm at lists.adiscon.com>
>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>     _______________________________________________
>>>>     Lognorm mailing list
>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>> <Lognorm at lists.adiscon.com>
>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>> - --
>>> - - Quadrant Information Security
>>>   Champ Clark III
>>>   o: 800.538.9357 x 101
>>>   c: 850.443.2440
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.11 (GNU/Linux)
>>>
>>> iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
>>> U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
>>> Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
>>> 1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
>>> OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
>>> AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
>>> =rGsk
>>> -----END PGP SIGNATURE-----
>>>
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From singh.janmejay at gmail.com  Tue Nov  4 00:54:11 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Tue, 4 Nov 2014 05:24:11 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <alpine.DEB.2.02.1411031410150.31001@nftneq.ynat.uz>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
	<5457A4B4.50306@quadrantsec.com>
	<CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>
	<CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>
	<alpine.DEB.2.02.1411031410150.31001@nftneq.ynat.uz>
Message-ID: <CAGB1Vvybh2HdczP2PsxdtMrpUVLmrC_bZ0Oje0y+uQyn2YCKUA@mail.gmail.com>

Alright, module loading flag it is then.

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 4, 2014 3:42 AM, "David Lang" <david at lang.hm> wrote:

> On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>  I think module-loading call having a flag to turn on/off would be better
>> than build-time flag, purely because it allows one to build a package and
>> use it on several boxes, while keeping the feature off when not required.
>> A
>> more fine grained control, in that sense.
>>
>
> I was thinking in terms of something in the config file. We need it to be
> something that the person configuring rsyslog will see so that they know
> they are doing something slow. Most users don't compile rsyslog so they
> won't ever see the compile time flag.
>
> module loading time seems fine.
>
> David Lang
>
>  We can support both levels of enabling, but then a lot of people may be
>> confused as to why it doesn't work when they turned it on while building.
>>
>> 2 levels also seems a little over-protective to me. Kinda like Windows "do
>> you really want to delete this file" prompts.
>>
>> Does module-loading time flag sound ok?
>>
>> --
>> Regards,
>> Janmejay
>>
>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>> keyboard sporting it's not-so-smart-assist technology.
>>
>> On Nov 3, 2014 10:52 PM, "singh.janmejay" <singh.janmejay at gmail.com>
>> wrote:
>>
>>  Hex was just an example. I meant when someone needs to roll out something
>>> very quickly, and it's not supported in the build they have, it's useful
>>> to
>>> be able to do it using regex in rulebase but still completely working in
>>> terms of rules, rather than having to resort to a mix of rules and
>>> exec-template calls with regexp based property extractors etc.
>>>
>>> It's still the same amount of work, if someone is trying to do something
>>> unsupported by lognorm. So performance effects will still show, just not
>>> in
>>> lognorm.
>>>
>>> This allows for flexibility and clean way of parsing logs at possibly
>>> even
>>> slightly lesser cost then the former approach.
>>>
>>> We can support a switch-on flag in conf file(module loading call, may be)
>>> which can be used to enable performance sensitive features.
>>>
>>> I personally think warning in logs would be enough though.
>>>
>>> --
>>> Regards,
>>> Janmejay
>>>
>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>> keyboard sporting it's not-so-smart-assist technology.
>>>
>>> On Nov 3, 2014 9:22 PM, "Champ Clark III" <cclark at quadrantsec.com>
>>> wrote:
>>>
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> I think David means something like:
>>>>
>>>> ./configure --enable-preformance-pigs
>>>>
>>>> I'm not a huge fan of this for a couple of reasons.
>>>>
>>>> In your example,  you want to find hex values.  I'd rather see a lognorm
>>>> "parser" created for this purpose.   I'm afraid if we start with regular
>>>> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
>>>> things very confusing.  I really like the way lognorm's "masking" works.
>>>> I'd rather not see that break.
>>>>
>>>>
>>>>
>>>> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>>>>
>>>>>
>>>>> We log a warning?
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>>
>>>>> PS: Please blame the typos in this mail on my phone's uncivilized soft
>>>>>
>>>> keyboard sporting it's not-so-smart-assist technology.
>>>>
>>>>>
>>>>>
>>>>> On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
>>>>>
>>>> <mailto:david at lang.hm> <david at lang.hm>> wrote:
>>>>
>>>>>
>>>>>     On Mon, 3 Nov 2014, singh.janmejay wrote:
>>>>>
>>>>>         I am thinking of it as a 2nd class field-type.
>>>>>
>>>>>         By that I mean, one gets best performance from 1st class
>>>>>
>>>> supported
>>>>
>>>>>         field-types, but if for some reason that is not sufficient for
>>>>>
>>>> someone,
>>>>
>>>>>         they can use a regex-field-type. It may be a little low on
>>>>>
>>>> performance, but
>>>>
>>>>>         then it unblocks people immediately.
>>>>>
>>>>>         I can do it, just need to know we are not ideologically against
>>>>>
>>>> it.
>>>>
>>>>>
>>>>>         Kind of construct im thinking of:
>>>>>
>>>>>         %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>>>>>
>>>>>         Thoughts?
>>>>>
>>>>>
>>>>>     Since this will be such a performance pig compared to the existing
>>>>>
>>>> parse tree, how about requiring a 'enable low performance types' flag or
>>>> something like that to enable it?
>>>>
>>>>>
>>>>>     There needs to be some good indicator that this is a performance
>>>>>
>>>> problem.
>>>>
>>>>>
>>>>>     David Lang
>>>>>     _______________________________________________
>>>>>     Lognorm mailing list
>>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>>>>
>>>> <Lognorm at lists.adiscon.com>
>>>>
>>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>     _______________________________________________
>>>>>     Lognorm mailing list
>>>>>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>>>>>
>>>> <Lognorm at lists.adiscon.com>
>>>>
>>>>>     http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>
>>>>
>>>> - --
>>>> - - Quadrant Information Security
>>>>   Champ Clark III
>>>>   o: 800.538.9357 x 101
>>>>   c: 850.443.2440
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG v1.4.11 (GNU/Linux)
>>>>
>>>> iQEcBAEBAgAGBQJUV6S0AAoJENnmXt7Lmc3KAwEH/jiwhj/nhFRpvRm7DttrvYTE
>>>> U7kHpjToIMlCSiJqiS4bnvOTS4oTG6mQ5myYzAr16ITTIJQLnJSHmVWBgxlGdUlM
>>>> Kq33I+zYjfUK2Go01PSoLjoE3rgdGa1hptFHVUZREuRkzSP6THtXn8XhexmZzdpH
>>>> 1lC+ZXwNZ9k7FsWm3M027I8zGDKnvZLVdf2UJUElyrNxmWW04ieR3lqJ5qh3Uj8l
>>>> OvTzEDYObghwyS6hThNb1oMz2Sr1AD/mWu+sri1BDKqlPyMkQgYe8KNyI4sKBQHL
>>>> AdCi3TXMvYa8WMb5AOVL4tX0QrOgwNj5mbpeWb2vWrV/S2mkN39TZbE406W91T8=
>>>> =rGsk
>>>> -----END PGP SIGNATURE-----
>>>>
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141104/97b7e9fc/attachment-0001.html>

From cclark at quadrantsec.com  Tue Nov  4 01:36:22 2014
From: cclark at quadrantsec.com (Champ Clark III)
Date: Mon, 03 Nov 2014 19:36:22 -0500
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <CAGB1Vvybh2HdczP2PsxdtMrpUVLmrC_bZ0Oje0y+uQyn2YCKUA@mail.gmail.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>	<5457A4B4.50306@quadrantsec.com>	<CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>	<CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>	<alpine.DEB.2.02.1411031410150.31001@nftneq.ynat.uz>
	<CAGB1Vvybh2HdczP2PsxdtMrpUVLmrC_bZ0Oje0y+uQyn2YCKUA@mail.gmail.com>
Message-ID: <54581F86.2070508@quadrantsec.com>

It depends on how you plan on doing regular expressions.   If you plan
on using libpcre,  then I'd say make it a build flag.   Probably
disabled by default,   since
I doubt anyone wants liblognorm to be dependent on libpcre by default. 


On 11/3/14, 6:54 PM, singh.janmejay wrote:
>
> Alright, module loading flag it is then.
>
> --
> Regards,
> Janmejay
>
> PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
>   
>
> On Nov 4, 2014 3:42 AM, "David Lang" <david at lang.hm
<mailto:david at lang.hm>> wrote:
>
>     On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>         I think module-loading call having a flag to turn on/off would
be better
>         than build-time flag, purely because it allows one to build a
package and
>         use it on several boxes, while keeping the feature off when
not required. A
>         more fine grained control, in that sense.
>
>
>     I was thinking in terms of something in the config file. We need
it to be something that the person configuring rsyslog will see so that
they know they are doing something slow. Most users don't compile
rsyslog so they won't ever see the compile time flag.
>
>     module loading time seems fine.
>
>     David Lang
>
>         We can support both levels of enabling, but then a lot of
people may be
>         confused as to why it doesn't work when they turned it on
while building.
>
>         2 levels also seems a little over-protective to me. Kinda like
Windows "do
>         you really want to delete this file" prompts.
>
>         Does module-loading time flag sound ok?
>
>         --
>         Regards,
>         Janmejay
>
>         PS: Please blame the typos in this mail on my phone's
uncivilized soft
>         keyboard sporting it's not-so-smart-assist technology.
>
>         On Nov 3, 2014 10:52 PM, "singh.janmejay"
<singh.janmejay at gmail.com <mailto:singh.janmejay at gmail.com>> wrote:
>
>             Hex was just an example. I meant when someone needs to
roll out something
>             very quickly, and it's not supported in the build they
have, it's useful to
>             be able to do it using regex in rulebase but still
completely working in
>             terms of rules, rather than having to resort to a mix of
rules and
>             exec-template calls with regexp based property extractors etc.
>
>             It's still the same amount of work, if someone is trying
to do something
>             unsupported by lognorm. So performance effects will still
show, just not in
>             lognorm.
>
>             This allows for flexibility and clean way of parsing logs
at possibly even
>             slightly lesser cost then the former approach.
>
>             We can support a switch-on flag in conf file(module
loading call, may be)
>             which can be used to enable performance sensitive features.
>
>             I personally think warning in logs would be enough though.
>
>             --
>             Regards,
>             Janmejay
>
>             PS: Please blame the typos in this mail on my phone's
uncivilized soft
>             keyboard sporting it's not-so-smart-assist technology.
>
>             On Nov 3, 2014 9:22 PM, "Champ Clark III"
<cclark at quadrantsec.com <mailto:cclark at quadrantsec.com>> wrote:
>
>
> I think David means something like:
>
> ./configure --enable-preformance-pigs
>
> I'm not a huge fan of this for a couple of reasons.
>
> In your example,  you want to find hex values.  I'd rather see a lognorm
> "parser" created for this purpose.   I'm afraid if we start with regular
> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
> things very confusing.  I really like the way lognorm's "masking" works.
> I'd rather not see that break.
>
>
>
> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>
>
>     We log a warning?
>
>     --
>     Regards,
>     Janmejay
>
>     PS: Please blame the typos in this mail on my phone's uncivilized soft
>
> keyboard sporting it's not-so-smart-assist technology.
>
>
>
>     On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
> <mailto:david at lang.hm>
>
> <mailto:david at lang.hm <mailto:david at lang.hm>> <david at lang.hm
> <mailto:david at lang.hm>>> wrote:
>
>
>         On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>             I am thinking of it as a 2nd class field-type.
>
>             By that I mean, one gets best performance from 1st class
>
> supported
>
>             field-types, but if for some reason that is not sufficient for
>
> someone,
>
>             they can use a regex-field-type. It may be a little low on
>
> performance, but
>
>             then it unblocks people immediately.
>
>             I can do it, just need to know we are not ideologically
> against
>
> it.
>
>
>             Kind of construct im thinking of:
>
>             %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>
>             Thoughts?
>
>
>         Since this will be such a performance pig compared to the existing
>
> parse tree, how about requiring a 'enable low performance types' flag or
> something like that to enable it?
>
>
>         There needs to be some good indicator that this is a performance
>
> problem.
>
>
>         David Lang
>         _______________________________________________
>         Lognorm mailing list
>         Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <mailto:Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>>
>
> <Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>>
>
>         http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>         _______________________________________________
>         Lognorm mailing list
>         Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <mailto:Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>>
>
> <Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>>
>
>         http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>
>
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>
>
>
>
>                 _______________________________________________
>                 Lognorm mailing list
>                 Lognorm at lists.adiscon.com
<mailto:Lognorm at lists.adiscon.com>
>                 http://lists.adiscon.net/mailman/listinfo/lognorm
<http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>
>
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
>
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/9d3a13b5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 536 bytes
Desc: OpenPGP digital signature
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141103/9d3a13b5/attachment.pgp>

From singh.janmejay at gmail.com  Tue Nov  4 02:16:07 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Tue, 4 Nov 2014 06:46:07 +0530
Subject: [Lognorm] Any reason we should not support regex-based
	field-type?
In-Reply-To: <54581F86.2070508@quadrantsec.com>
References: <CAGB1VvwmdE+fXDxJ1XkkappmvghtmF60APUBbcVqSqmHaOBLUw@mail.gmail.com>
	<alpine.DEB.2.02.1411030405440.31001@nftneq.ynat.uz>
	<CAGB1Vvw1AK3i6ptbQytA7+k72AsoxZCWVXhe-RbK-RHC-+95LQ@mail.gmail.com>
	<5457A4B4.50306@quadrantsec.com>
	<CAGB1VvxFU6zRRiHk=i_D5C8RuRVf_R7Un_9vN1ET8uDMGGDThA@mail.gmail.com>
	<CAGB1VvxJDGuk6yjRBJm1-Y7dKY4Doru8nBEhUFO4qxBb9bx+XQ@mail.gmail.com>
	<alpine.DEB.2.02.1411031410150.31001@nftneq.ynat.uz>
	<CAGB1Vvybh2HdczP2PsxdtMrpUVLmrC_bZ0Oje0y+uQyn2YCKUA@mail.gmail.com>
	<54581F86.2070508@quadrantsec.com>
Message-ID: <CAGB1VvwPAU5CrNG1025j-C+juxPpZx0T6H8Att6_CdmhT+TLQQ@mail.gmail.com>

Yes, it should have optional libpcre dependency.

But shouldn't it be enabled or disabled based on availability of it as
opposed to requiring manual enabling regardless of libpcre being available?

--
Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Nov 4, 2014 6:06 AM, "Champ Clark III" <cclark at quadrantsec.com> wrote:

>  It depends on how you plan on doing regular expressions.   If you plan on
> using libpcre,  then I'd say make it a build flag.   Probably disabled by
> default,   since
> I doubt anyone wants liblognorm to be dependent on libpcre by default.
>
>
> On 11/3/14, 6:54 PM, singh.janmejay wrote:
> >
> > Alright, module loading flag it is then.
> >
> > --
> > Regards,
> > Janmejay
> >
> > PS: Please blame the typos in this mail on my phone's uncivilized soft
> keyboard sporting it's not-so-smart-assist technology.
> >
> >
> > On Nov 4, 2014 3:42 AM, "David Lang" <david at lang.hm
> <mailto:david at lang.hm> <david at lang.hm>> wrote:
> >
> >     On Mon, 3 Nov 2014, singh.janmejay wrote:
> >
> >         I think module-loading call having a flag to turn on/off would
> be better
> >         than build-time flag, purely because it allows one to build a
> package and
> >         use it on several boxes, while keeping the feature off when not
> required. A
> >         more fine grained control, in that sense.
> >
> >
> >     I was thinking in terms of something in the config file. We need it
> to be something that the person configuring rsyslog will see so that they
> know they are doing something slow. Most users don't compile rsyslog so
> they won't ever see the compile time flag.
> >
> >     module loading time seems fine.
> >
> >     David Lang
> >
> >         We can support both levels of enabling, but then a lot of people
> may be
> >         confused as to why it doesn't work when they turned it on while
> building.
> >
> >         2 levels also seems a little over-protective to me. Kinda like
> Windows "do
> >         you really want to delete this file" prompts.
> >
> >         Does module-loading time flag sound ok?
> >
> >         --
> >         Regards,
> >         Janmejay
> >
> >         PS: Please blame the typos in this mail on my phone's
> uncivilized soft
> >         keyboard sporting it's not-so-smart-assist technology.
> >
> >         On Nov 3, 2014 10:52 PM, "singh.janmejay" <
> singh.janmejay at gmail.com <mailto:singh.janmejay at gmail.com>
> <singh.janmejay at gmail.com>> wrote:
> >
> >             Hex was just an example. I meant when someone needs to roll
> out something
> >             very quickly, and it's not supported in the build they have,
> it's useful to
> >             be able to do it using regex in rulebase but still
> completely working in
> >             terms of rules, rather than having to resort to a mix of
> rules and
> >             exec-template calls with regexp based property extractors
> etc.
> >
> >             It's still the same amount of work, if someone is trying to
> do something
> >             unsupported by lognorm. So performance effects will still
> show, just not in
> >             lognorm.
> >
> >             This allows for flexibility and clean way of parsing logs at
> possibly even
> >             slightly lesser cost then the former approach.
> >
> >             We can support a switch-on flag in conf file(module loading
> call, may be)
> >             which can be used to enable performance sensitive features.
> >
> >             I personally think warning in logs would be enough though.
> >
> >             --
> >             Regards,
> >             Janmejay
> >
> >             PS: Please blame the typos in this mail on my phone's
> uncivilized soft
> >             keyboard sporting it's not-so-smart-assist technology.
> >
> >             On Nov 3, 2014 9:22 PM, "Champ Clark III" <
> cclark at quadrantsec.com <mailto:cclark at quadrantsec.com>
> <cclark at quadrantsec.com>> wrote:
> >
> >
>
> I think David means something like:
>
> ./configure --enable-preformance-pigs
>
> I'm not a huge fan of this for a couple of reasons.
>
> In your example,  you want to find hex values.  I'd rather see a lognorm
> "parser" created for this purpose.   I'm afraid if we start with regular
> expressions,  we'll end up mixing rules (RE and non-RE) and it will make
> things very confusing.  I really like the way lognorm's "masking" works.
> I'd rather not see that break.
>
>
>
> On 11/03/2014 07:15 AM, singh.janmejay wrote:
>
>
>     We log a warning?
>
>     --
>     Regards,
>     Janmejay
>
>     PS: Please blame the typos in this mail on my phone's uncivilized soft
>
> keyboard sporting it's not-so-smart-assist technology.
>
>
>
>     On Nov 3, 2014 5:37 PM, "David Lang" <david at lang.hm
> <mailto:david at lang.hm> <david at lang.hm>
>
> <mailto:david at lang.hm <david at lang.hm> <mailto:david at lang.hm>
> <david at lang.hm>> <david at lang.hm <mailto:david at lang.hm> <david at lang.hm>>>
> wrote:
>
>
>         On Mon, 3 Nov 2014, singh.janmejay wrote:
>
>             I am thinking of it as a 2nd class field-type.
>
>             By that I mean, one gets best performance from 1st class
>
> supported
>
>             field-types, but if for some reason that is not sufficient for
>
> someone,
>
>             they can use a regex-field-type. It may be a little low on
>
> performance, but
>
>             then it unblocks people immediately.
>
>             I can do it, just need to know we are not ideologically against
>
> it.
>
>
>             Kind of construct im thinking of:
>
>             %foo:regex:[a-f0-9]+% to match hex-numbers for instance.
>
>             Thoughts?
>
>
>         Since this will be such a performance pig compared to the existing
>
> parse tree, how about requiring a 'enable low performance types' flag or
> something like that to enable it?
>
>
>         There needs to be some good indicator that this is a performance
>
> problem.
>
>
>         David Lang
>         _______________________________________________
>         Lognorm mailing list
>         Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com> <mailto:Lognorm at lists.adiscon.com
> <Lognorm at lists.adiscon.com> <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>>
>
> <Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>>
>
>         http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>         _______________________________________________
>         Lognorm mailing list
>         Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com> <mailto:Lognorm at lists.adiscon.com
> <Lognorm at lists.adiscon.com> <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>>
>
> <Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>>
>
>         http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>
>
>     _______________________________________________
>     Lognorm mailing list
>     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>
>     http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
>
>
>
>  >
> >
> >                 _______________________________________________
> >                 Lognorm mailing list
> >                 Lognorm at lists.adiscon.com
> <mailto:Lognorm at lists.adiscon.com> <Lognorm at lists.adiscon.com>
> >                 http://lists.adiscon.net/mailman/listinfo/lognorm
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
> <http://lists.adiscon.net/mailman/listinfo/lognorm>
> >
> >
> >
> >     _______________________________________________
> >     Lognorm mailing list
> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>
> >     http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> >     _______________________________________________
> >     Lognorm mailing list
> >     Lognorm at lists.adiscon.com <mailto:Lognorm at lists.adiscon.com>
> <Lognorm at lists.adiscon.com>
> >     http://lists.adiscon.net/mailman/listinfo/lognorm
> >
> >
> >
> > _______________________________________________
> > Lognorm mailing list
> > Lognorm at lists.adiscon.com
> > http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141104/f8b2b488/attachment-0001.html>

From singh.janmejay at gmail.com  Tue Nov 11 05:54:54 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Tue, 11 Nov 2014 10:24:54 +0530
Subject: [Lognorm] confusion related to documentation for liblognorm (linked
 from rsyslog mmnormalize page)
Message-ID: <CAGB1VvwXDfsyMoK44hAq_Fnx6Dg1=GDRs=NVmkxoFw1Ewpe9pg@mail.gmail.com>

Hi,

I was working on liblognorm docs recently, it seems to have doc-fragments
for rest, char-seq etc, but those fragments don't show up on the
documentation published and linked from mmnormalize page. The published
version also looks very different from the html version that gets built in
working-copy.

Here is one of this pages linked from mmnormalize doc.
http://www.liblognorm.com/help/creating-a-rulebase/

It seems like a recent version needs to be published? Or may be im looking
at the wrong link.

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141111/2dd46be9/attachment.html>

From rgerhards at hq.adiscon.com  Tue Nov 11 07:37:32 2014
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Nov 2014 07:37:32 +0100
Subject: [Lognorm] confusion related to documentation for liblognorm
 (linked from rsyslog mmnormalize page)
In-Reply-To: <CAGB1VvwXDfsyMoK44hAq_Fnx6Dg1=GDRs=NVmkxoFw1Ewpe9pg@mail.gmail.com>
References: <CAGB1VvwXDfsyMoK44hAq_Fnx6Dg1=GDRs=NVmkxoFw1Ewpe9pg@mail.gmail.com>
Message-ID: <CADk+mPDvgNFSCgtx4uWJRJoO5YvkwGDpWjM+hGPk1vyj1JLafw@mail.gmail.com>

yeah, there is no automatted process for the lognorm doc. It's becoming too
much to do for a single guy. I'll see if I can take down that doc portion
of the site and link it to github within the next couple of days.

Rainer

2014-11-11 5:54 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:

> Hi,
>
> I was working on liblognorm docs recently, it seems to have doc-fragments
> for rest, char-seq etc, but those fragments don't show up on the
> documentation published and linked from mmnormalize page. The published
> version also looks very different from the html version that gets built in
> working-copy.
>
> Here is one of this pages linked from mmnormalize doc.
> http://www.liblognorm.com/help/creating-a-rulebase/
>
> It seems like a recent version needs to be published? Or may be im looking
> at the wrong link.
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141111/ed1cdeb2/attachment.html>

From singh.janmejay at gmail.com  Tue Nov 11 07:52:43 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Tue, 11 Nov 2014 12:22:43 +0530
Subject: [Lognorm] confusion related to documentation for liblognorm
 (linked from rsyslog mmnormalize page)
In-Reply-To: <CADk+mPDvgNFSCgtx4uWJRJoO5YvkwGDpWjM+hGPk1vyj1JLafw@mail.gmail.com>
References: <CAGB1VvwXDfsyMoK44hAq_Fnx6Dg1=GDRs=NVmkxoFw1Ewpe9pg@mail.gmail.com>
	<CADk+mPDvgNFSCgtx4uWJRJoO5YvkwGDpWjM+hGPk1vyj1JLafw@mail.gmail.com>
Message-ID: <CAGB1VvwVkHseoweym535xoKBvwwL3vAugQaonM6Fyj3P8KFQ8A@mail.gmail.com>

If you mean automated way of generating docs, that seems to work (I just
stepped into doc directory and executed make).

Or did you mean there is no automated way of publishing it to the website?

On Tue, Nov 11, 2014 at 12:07 PM, Rainer Gerhards <rgerhards at hq.adiscon.com>
wrote:

> yeah, there is no automatted process for the lognorm doc. It's becoming
> too much to do for a single guy. I'll see if I can take down that doc
> portion of the site and link it to github within the next couple of days.
>
> Rainer
>
> 2014-11-11 5:54 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>
>> Hi,
>>
>> I was working on liblognorm docs recently, it seems to have doc-fragments
>> for rest, char-seq etc, but those fragments don't show up on the
>> documentation published and linked from mmnormalize page. The published
>> version also looks very different from the html version that gets built in
>> working-copy.
>>
>> Here is one of this pages linked from mmnormalize doc.
>> http://www.liblognorm.com/help/creating-a-rulebase/
>>
>> It seems like a recent version needs to be published? Or may be im
>> looking at the wrong link.
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141111/33ebca1d/attachment.html>

From rgerhards at hq.adiscon.com  Tue Nov 11 07:55:56 2014
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 11 Nov 2014 07:55:56 +0100
Subject: [Lognorm] confusion related to documentation for liblognorm
 (linked from rsyslog mmnormalize page)
In-Reply-To: <CAGB1VvwVkHseoweym535xoKBvwwL3vAugQaonM6Fyj3P8KFQ8A@mail.gmail.com>
References: <CAGB1VvwXDfsyMoK44hAq_Fnx6Dg1=GDRs=NVmkxoFw1Ewpe9pg@mail.gmail.com>
	<CADk+mPDvgNFSCgtx4uWJRJoO5YvkwGDpWjM+hGPk1vyj1JLafw@mail.gmail.com>
	<CAGB1VvwVkHseoweym535xoKBvwwL3vAugQaonM6Fyj3P8KFQ8A@mail.gmail.com>
Message-ID: <CADk+mPDW37YSJb1oe+SD=MhtChvy5Oe0y6OWn0zm8t_+9FFBWQ@mail.gmail.com>

2014-11-11 7:52 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:

> If you mean automated way of generating docs, that seems to work (I just
> stepped into doc directory and executed make).
>
> Or did you mean there is no automated way of publishing it to the website?
>
>
yup, this I meant. For rsyslog, I had set this up some time ago. But with
all the work going on on testbench, git shuffeling, project policies I
really have no time to do this for lognorm now.

Rainer

> On Tue, Nov 11, 2014 at 12:07 PM, Rainer Gerhards <
> rgerhards at hq.adiscon.com> wrote:
>
>> yeah, there is no automatted process for the lognorm doc. It's becoming
>> too much to do for a single guy. I'll see if I can take down that doc
>> portion of the site and link it to github within the next couple of days.
>>
>> Rainer
>>
>> 2014-11-11 5:54 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>
>>> Hi,
>>>
>>> I was working on liblognorm docs recently, it seems to have
>>> doc-fragments for rest, char-seq etc, but those fragments don't show up on
>>> the documentation published and linked from mmnormalize page. The published
>>> version also looks very different from the html version that gets built in
>>> working-copy.
>>>
>>> Here is one of this pages linked from mmnormalize doc.
>>> http://www.liblognorm.com/help/creating-a-rulebase/
>>>
>>> It seems like a recent version needs to be published? Or may be im
>>> looking at the wrong link.
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141111/34d4cda5/attachment.html>

From singh.janmejay at gmail.com  Wed Nov 12 05:22:08 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Wed, 12 Nov 2014 09:52:08 +0530
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>
	<54532DA5.8030107@levshin.spb.ru>
	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>
	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>
	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>
	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>
	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>
	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>
	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>
	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>
	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
Message-ID: <CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>

How do we want foreach support to be?

Possible places are:
1. a foreach loop-construct along-side flow-control structures in rscript
2. a foreach value-producing templateElement in template

The tradeoff is, someone looking to do some useful re-structuring of
output, may want to call exec-template several times in approach-1, but
they can just emit out what they want by placing property-extractors in
foreach block in approch-2.

The counter argument is, templates can't assign variables, execute
functions, do arithmetic-operations etc, so usefulness of foreach in a
template will be limited to most basic of message transformation cases.
Additionally, string-template as of now has no clean way of implementing
foreach. It'll require some involved syntax.

To me approach-1 seems better suited for such a thing (purely because
if-else, the other flow control structure is implemented at this level, and
variable assignment, functions etc make arbitrary transformation easier).

Once we have decided where to place foreach, let us zero in on its syntax.
To seed the thought, how about this:

set $.grault = "";
foreach($!foo as $.bar) {
    set $.baz = $.bar!quux;
    foreach($.baz as $.corge) {
        if ( $.corge contains "#" ) then {
            set $.grault = $.grault & $.corge;
        }
    }
}

Alternatively:
foreach($.bar <- $!foo) {...}
foreach($.bar in $!foo) {...}
foreach($.bar : $!foo) {...}



On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> Sure, I'll fork on github.
>
> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <rgerhards at hq.adiscon.com
> > wrote:
>
>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>
>>> Cool, I'll implement $!foo!bar[0].
>>>
>>> +1
>>
>>
>>> Let us process this patch-set, because is kinda hard to keep track of
>>> old patches and re-send in one shot.
>>>
>>>
>> would you mind cloning on github and maintain a feature branch there?
>> That would make it much easier for me, as I could merge the branch when you
>> are done. If not, it's no problem and I'll maintain that branch.
>>
>>
>> i'll send the new patch once done(i'll now only get to work on it on
>>> monday).
>>>
>>>
>> I haven't had a chance to look as I am now busy building test
>> environments and looking at the testbench [yes, one guy!] ;) But I see
>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>> it is best to coordinate that part with him.
>>
>> Rainer
>>
>> Do existing patches look ok except for the indexed-addressing feature?
>>>
>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>
>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>
>>>>  Yes, I didn't have a need to address tokens individually, but you have
>>>>> a
>>>>> point.
>>>>>
>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>
>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>> backward
>>>>> compatibility. How about a function?
>>>>>
>>>>> I'll be happy to implement support for addressing it in $!... notation
>>>>> if
>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>
>>>>
>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>> that's likely to be a real problem. I don't think there's ever a really
>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>> syntax, it gets messy.
>>>>
>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>> function that takes the pattern and repeats it for each item in the array.
>>>>
>>>> David Lang
>>>>
>>>>
>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>> tests:
>>>>>>
>>>>>>>
>>>>>>> [ "15", "26", "15" ]
>>>>>>>
>>>>>>>
>>>>>> right, but how do you access it in rsyslog?
>>>>>>
>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>> and
>>>>>> get the result '10'
>>>>>>
>>>>>> what would you use to access the value '26' in your example?
>>>>>>
>>>>>> we also don't have anything like foreach() in our template language,
>>>>>> which
>>>>>> makes it hard to make use of these values as anything other than a
>>>>>> JSON
>>>>>> string.
>>>>>>
>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>> problems
>>>>>> that we will have using it.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>> add
>>>>>>
>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>> can
>>>>>>> do.
>>>>>>>
>>>>>>> The idea behind this is to generate structured content from
>>>>>>> semi-structured
>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>> way to
>>>>>>> represent that).
>>>>>>>
>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>> so that
>>>>>>> it can be pre-processed before its emitted. May be have a canned set
>>>>>>> of
>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>
>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>> this
>>>>>>> was
>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>> complex
>>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>>> fields
>>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>>
>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>> other
>>>>>>> (recurse) field-type.
>>>>>>>
>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>
>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>> too.
>>>>>>>> But
>>>>>>>>
>>>>>>>>  I
>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>> it
>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>> chance to
>>>>>>>>> work on it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>> mmnormalize
>>>>>>>> on
>>>>>>>> a variable to parse subvariables.
>>>>>>>>
>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>> addressing yet.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>> discuss
>>>>>>>>
>>>>>>>>  something else that I was thinking about.
>>>>>>>>>
>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>
>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>> recurse will
>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>> stateless, so
>>>>>>>>> I
>>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>>> picking
>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>> else, it
>>>>>>>>> should be considered no-match.
>>>>>>>>>
>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>> thinking
>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  _______________________________________________
>>>>>>>>>
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141112/abe94ebe/attachment-0001.html>

From singh.janmejay at gmail.com  Wed Nov 12 05:25:31 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Wed, 12 Nov 2014 09:55:31 +0530
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>
	<54532DA5.8030107@levshin.spb.ru>
	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>
	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>
	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>
	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>
	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>
	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>
	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>
	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>
	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
	<CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
Message-ID: <CAGB1VvxjJ_+F4WSvwRezP9oHjr3ckCxP_bJv5STrzQMmLbgN2w@mail.gmail.com>

The array-subscripting patches are ready, I'll send a pull request on
github.

It is of the form $!foo[1]!bar[2]!baz.

On Wed, Nov 12, 2014 at 9:52 AM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> How do we want foreach support to be?
>
> Possible places are:
> 1. a foreach loop-construct along-side flow-control structures in rscript
> 2. a foreach value-producing templateElement in template
>
> The tradeoff is, someone looking to do some useful re-structuring of
> output, may want to call exec-template several times in approach-1, but
> they can just emit out what they want by placing property-extractors in
> foreach block in approch-2.
>
> The counter argument is, templates can't assign variables, execute
> functions, do arithmetic-operations etc, so usefulness of foreach in a
> template will be limited to most basic of message transformation cases.
> Additionally, string-template as of now has no clean way of implementing
> foreach. It'll require some involved syntax.
>
> To me approach-1 seems better suited for such a thing (purely because
> if-else, the other flow control structure is implemented at this level, and
> variable assignment, functions etc make arbitrary transformation easier).
>
> Once we have decided where to place foreach, let us zero in on its syntax.
> To seed the thought, how about this:
>
> set $.grault = "";
> foreach($!foo as $.bar) {
>     set $.baz = $.bar!quux;
>     foreach($.baz as $.corge) {
>         if ( $.corge contains "#" ) then {
>             set $.grault = $.grault & $.corge;
>         }
>     }
> }
>
> Alternatively:
> foreach($.bar <- $!foo) {...}
> foreach($.bar in $!foo) {...}
> foreach($.bar : $!foo) {...}
>
>
>
> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com>
> wrote:
>
>> Sure, I'll fork on github.
>>
>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <
>> rgerhards at hq.adiscon.com> wrote:
>>
>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>>
>>>> Cool, I'll implement $!foo!bar[0].
>>>>
>>>> +1
>>>
>>>
>>>> Let us process this patch-set, because is kinda hard to keep track of
>>>> old patches and re-send in one shot.
>>>>
>>>>
>>> would you mind cloning on github and maintain a feature branch there?
>>> That would make it much easier for me, as I could merge the branch when you
>>> are done. If not, it's no problem and I'll maintain that branch.
>>>
>>>
>>> i'll send the new patch once done(i'll now only get to work on it on
>>>> monday).
>>>>
>>>>
>>> I haven't had a chance to look as I am now busy building test
>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>>> it is best to coordinate that part with him.
>>>
>>> Rainer
>>>
>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>
>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>>
>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>
>>>>>  Yes, I didn't have a need to address tokens individually, but you
>>>>>> have a
>>>>>> point.
>>>>>>
>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>
>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>> backward
>>>>>> compatibility. How about a function?
>>>>>>
>>>>>> I'll be happy to implement support for addressing it in $!...
>>>>>> notation if
>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>
>>>>>
>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>>> that's likely to be a real problem. I don't think there's ever a really
>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>> syntax, it gets messy.
>>>>>
>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>> function that takes the pattern and repeats it for each item in the array.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>>
>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>>> tests:
>>>>>>>
>>>>>>>>
>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>
>>>>>>>>
>>>>>>> right, but how do you access it in rsyslog?
>>>>>>>
>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>> and
>>>>>>> get the result '10'
>>>>>>>
>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>
>>>>>>> we also don't have anything like foreach() in our template language,
>>>>>>> which
>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>> JSON
>>>>>>> string.
>>>>>>>
>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>> problems
>>>>>>> that we will have using it.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>>> add
>>>>>>>
>>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>>> can
>>>>>>>> do.
>>>>>>>>
>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>> semi-structured
>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>> way to
>>>>>>>> represent that).
>>>>>>>>
>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>> so that
>>>>>>>> it can be pre-processed before its emitted. May be have a canned
>>>>>>>> set of
>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>
>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>> this
>>>>>>>> was
>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>> complex
>>>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>>>> fields
>>>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>>>
>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>> other
>>>>>>>> (recurse) field-type.
>>>>>>>>
>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>>
>>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>> too.
>>>>>>>>> But
>>>>>>>>>
>>>>>>>>>  I
>>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>>> it
>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>> chance to
>>>>>>>>>> work on it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>>> mmnormalize
>>>>>>>>> on
>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>
>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>> addressing yet.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>>> discuss
>>>>>>>>>
>>>>>>>>>  something else that I was thinking about.
>>>>>>>>>>
>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>
>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>> recurse will
>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>> stateless, so
>>>>>>>>>> I
>>>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>>>> picking
>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>> else, it
>>>>>>>>>> should be considered no-match.
>>>>>>>>>>
>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>> thinking
>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  _______________________________________________
>>>>>>>>>>
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>> http://codehunk.wordpress.com
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141112/4a0d879c/attachment.html>

From singh.janmejay at gmail.com  Wed Nov 12 05:41:34 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Wed, 12 Nov 2014 10:11:34 +0530
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <CAGB1VvxjJ_+F4WSvwRezP9oHjr3ckCxP_bJv5STrzQMmLbgN2w@mail.gmail.com>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>
	<54532DA5.8030107@levshin.spb.ru>
	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>
	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>
	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>
	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>
	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>
	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>
	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>
	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>
	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
	<CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
	<CAGB1VvxjJ_+F4WSvwRezP9oHjr3ckCxP_bJv5STrzQMmLbgN2w@mail.gmail.com>
Message-ID: <CAGB1Vvzs0V1WyH4=JDiiLzUH0NVCf3R5A08U6S-UhkMFt=iF5g@mail.gmail.com>

Sorry, this should be discussed on rsyslog mailing list, adding rsyslog as
well.

Here are the pull requests(in order):

https://github.com/rsyslog/libestr/pull/1
https://github.com/rsyslog/liblognorm/pull/8
https://github.com/rsyslog/rsyslog/pull/149


On Wed, Nov 12, 2014 at 9:55 AM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> The array-subscripting patches are ready, I'll send a pull request on
> github.
>
> It is of the form $!foo[1]!bar[2]!baz.
>
> On Wed, Nov 12, 2014 at 9:52 AM, singh.janmejay <singh.janmejay at gmail.com>
> wrote:
>
>> How do we want foreach support to be?
>>
>> Possible places are:
>> 1. a foreach loop-construct along-side flow-control structures in rscript
>> 2. a foreach value-producing templateElement in template
>>
>> The tradeoff is, someone looking to do some useful re-structuring of
>> output, may want to call exec-template several times in approach-1, but
>> they can just emit out what they want by placing property-extractors in
>> foreach block in approch-2.
>>
>> The counter argument is, templates can't assign variables, execute
>> functions, do arithmetic-operations etc, so usefulness of foreach in a
>> template will be limited to most basic of message transformation cases.
>> Additionally, string-template as of now has no clean way of implementing
>> foreach. It'll require some involved syntax.
>>
>> To me approach-1 seems better suited for such a thing (purely because
>> if-else, the other flow control structure is implemented at this level, and
>> variable assignment, functions etc make arbitrary transformation easier).
>>
>> Once we have decided where to place foreach, let us zero in on its
>> syntax. To seed the thought, how about this:
>>
>> set $.grault = "";
>> foreach($!foo as $.bar) {
>>     set $.baz = $.bar!quux;
>>     foreach($.baz as $.corge) {
>>         if ( $.corge contains "#" ) then {
>>             set $.grault = $.grault & $.corge;
>>         }
>>     }
>> }
>>
>> Alternatively:
>> foreach($.bar <- $!foo) {...}
>> foreach($.bar in $!foo) {...}
>> foreach($.bar : $!foo) {...}
>>
>>
>>
>> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com
>> > wrote:
>>
>>> Sure, I'll fork on github.
>>>
>>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <
>>> rgerhards at hq.adiscon.com> wrote:
>>>
>>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>>>
>>>>> Cool, I'll implement $!foo!bar[0].
>>>>>
>>>>> +1
>>>>
>>>>
>>>>> Let us process this patch-set, because is kinda hard to keep track of
>>>>> old patches and re-send in one shot.
>>>>>
>>>>>
>>>> would you mind cloning on github and maintain a feature branch there?
>>>> That would make it much easier for me, as I could merge the branch when you
>>>> are done. If not, it's no problem and I'll maintain that branch.
>>>>
>>>>
>>>> i'll send the new patch once done(i'll now only get to work on it on
>>>>> monday).
>>>>>
>>>>>
>>>> I haven't had a chance to look as I am now busy building test
>>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>>>> it is best to coordinate that part with him.
>>>>
>>>> Rainer
>>>>
>>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>>
>>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>>  Yes, I didn't have a need to address tokens individually, but you
>>>>>>> have a
>>>>>>> point.
>>>>>>>
>>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>>
>>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>>> backward
>>>>>>> compatibility. How about a function?
>>>>>>>
>>>>>>> I'll be happy to implement support for addressing it in $!...
>>>>>>> notation if
>>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>>
>>>>>>
>>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>>>> that's likely to be a real problem. I don't think there's ever a really
>>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>>> syntax, it gets messy.
>>>>>>
>>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>>> function that takes the pattern and repeats it for each item in the array.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>>>
>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>
>>>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>>>> tests:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>> right, but how do you access it in rsyslog?
>>>>>>>>
>>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>>> and
>>>>>>>> get the result '10'
>>>>>>>>
>>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>>
>>>>>>>> we also don't have anything like foreach() in our template
>>>>>>>> language, which
>>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>>> JSON
>>>>>>>> string.
>>>>>>>>
>>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>>> problems
>>>>>>>> that we will have using it.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:).
>>>>>>>> I'll add
>>>>>>>>
>>>>>>>>> tests for it soon, but until then pasting output here is the best
>>>>>>>>> I can
>>>>>>>>> do.
>>>>>>>>>
>>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>>> semi-structured
>>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>>> way to
>>>>>>>>> represent that).
>>>>>>>>>
>>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>>> so that
>>>>>>>>> it can be pre-processed before its emitted. May be have a canned
>>>>>>>>> set of
>>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>>
>>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>>> this
>>>>>>>>> was
>>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>>> complex
>>>>>>>>> config for an access log with 15 - 20 fields, especially given
>>>>>>>>> those
>>>>>>>>> fields
>>>>>>>>> can have colon separated entries inside comma separated entries
>>>>>>>>> etc.
>>>>>>>>>
>>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>>> other
>>>>>>>>> (recurse) field-type.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>>>
>>>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>>> too.
>>>>>>>>>> But
>>>>>>>>>>
>>>>>>>>>>  I
>>>>>>>>>>> promised myself that i'd write a test for that instead of
>>>>>>>>>>> testing it
>>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>>> chance to
>>>>>>>>>>> work on it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>>>> mmnormalize
>>>>>>>>>> on
>>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>>
>>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>>> addressing yet.
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>>>> discuss
>>>>>>>>>>
>>>>>>>>>>  something else that I was thinking about.
>>>>>>>>>>>
>>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>>
>>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>>> recurse will
>>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>>> stateless, so
>>>>>>>>>>> I
>>>>>>>>>>> don't see any problems with that. I also plan to support tag
>>>>>>>>>>> based
>>>>>>>>>>> picking
>>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>>> else, it
>>>>>>>>>>> should be considered no-match.
>>>>>>>>>>>
>>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>>> thinking
>>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>>
>>>>>>>>>> Lognorm mailing list
>>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Lognorm mailing list
>>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>> http://codehunk.wordpress.com
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141112/0e826c8b/attachment-0001.html>

From david at lang.hm  Wed Nov 12 06:37:43 2014
From: david at lang.hm (David Lang)
Date: Tue, 11 Nov 2014 21:37:43 -0800 (PST)
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>
	<54532DA5.8030107@levshin.spb.ru>
	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>
	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>
	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>
	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>
	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>
	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>
	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>
	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>
	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
	<CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1411112124330.4597@nftneq.ynat.uz>

On Wed, 12 Nov 2014, singh.janmejay wrote:

> How do we want foreach support to be?
>
> Possible places are:
> 1. a foreach loop-construct along-side flow-control structures in rscript

My understanding of how rscript works is that it's mostly parsed at startup 
time, not at message processing time, so I don't see how it can be doen in 
rscript.

> 2. a foreach value-producing templateElement in template
>
> The tradeoff is, someone looking to do some useful re-structuring of
> output, may want to call exec-template several times in approach-1, but
> they can just emit out what they want by placing property-extractors in
> foreach block in approch-2.
>
> The counter argument is, templates can't assign variables, execute
> functions, do arithmetic-operations etc, so usefulness of foreach in a
> template will be limited to most basic of message transformation cases.
> Additionally, string-template as of now has no clean way of implementing
> foreach. It'll require some involved syntax.

the result of templates can be assigned to variables.

> To me approach-1 seems better suited for such a thing (purely because
> if-else, the other flow control structure is implemented at this level, and
> variable assignment, functions etc make arbitrary transformation easier).
>
> Once we have decided where to place foreach, let us zero in on its syntax.
> To seed the thought, how about this:
>
> set $.grault = "";
> foreach($!foo as $.bar) {
>    set $.baz = $.bar!quux;
>    foreach($.baz as $.corge) {
>        if ( $.corge contains "#" ) then {
>            set $.grault = $.grault & $.corge;
>        }
>    }
> }

in spite of everything I said above, if this can be implemented reasonably 
(without killing performance when it's not used), this would be extremely 
powerful.


> Alternatively:
> foreach($.bar <- $!foo) {...}
> foreach($.bar in $!foo) {...}

I like this a little better than 'as', but they are almost equal and either is 
far better than the punctuation versions.

David Lang

> foreach($.bar : $!foo) {...}
>
>
> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com>
> wrote:
>
>> Sure, I'll fork on github.
>>
>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <rgerhards at hq.adiscon.com
>>> wrote:
>>
>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>>
>>>> Cool, I'll implement $!foo!bar[0].
>>>>
>>>> +1
>>>
>>>
>>>> Let us process this patch-set, because is kinda hard to keep track of
>>>> old patches and re-send in one shot.
>>>>
>>>>
>>> would you mind cloning on github and maintain a feature branch there?
>>> That would make it much easier for me, as I could merge the branch when you
>>> are done. If not, it's no problem and I'll maintain that branch.
>>>
>>>
>>> i'll send the new patch once done(i'll now only get to work on it on
>>>> monday).
>>>>
>>>>
>>> I haven't had a chance to look as I am now busy building test
>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>>> it is best to coordinate that part with him.
>>>
>>> Rainer
>>>
>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>
>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>>
>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>
>>>>>  Yes, I didn't have a need to address tokens individually, but you have
>>>>>> a
>>>>>> point.
>>>>>>
>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>
>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>> backward
>>>>>> compatibility. How about a function?
>>>>>>
>>>>>> I'll be happy to implement support for addressing it in $!... notation
>>>>>> if
>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>
>>>>>
>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>>> that's likely to be a real problem. I don't think there's ever a really
>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>> syntax, it gets messy.
>>>>>
>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>> function that takes the pattern and repeats it for each item in the array.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>>
>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>>> tests:
>>>>>>>
>>>>>>>>
>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>
>>>>>>>>
>>>>>>> right, but how do you access it in rsyslog?
>>>>>>>
>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>> and
>>>>>>> get the result '10'
>>>>>>>
>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>
>>>>>>> we also don't have anything like foreach() in our template language,
>>>>>>> which
>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>> JSON
>>>>>>> string.
>>>>>>>
>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>> problems
>>>>>>> that we will have using it.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>>> add
>>>>>>>
>>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>>> can
>>>>>>>> do.
>>>>>>>>
>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>> semi-structured
>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>> way to
>>>>>>>> represent that).
>>>>>>>>
>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>> so that
>>>>>>>> it can be pre-processed before its emitted. May be have a canned set
>>>>>>>> of
>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>
>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>> this
>>>>>>>> was
>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>> complex
>>>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>>>> fields
>>>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>>>
>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>> other
>>>>>>>> (recurse) field-type.
>>>>>>>>
>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>>
>>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>> too.
>>>>>>>>> But
>>>>>>>>>
>>>>>>>>>  I
>>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>>> it
>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>> chance to
>>>>>>>>>> work on it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>>> mmnormalize
>>>>>>>>> on
>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>
>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>> addressing yet.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>>> discuss
>>>>>>>>>
>>>>>>>>>  something else that I was thinking about.
>>>>>>>>>>
>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>
>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>> recurse will
>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>> stateless, so
>>>>>>>>>> I
>>>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>>>> picking
>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>> else, it
>>>>>>>>>> should be considered no-match.
>>>>>>>>>>
>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>> thinking
>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  _______________________________________________
>>>>>>>>>>
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>> http://codehunk.wordpress.com
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>
>
>
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From singh.janmejay at gmail.com  Thu Nov 13 08:07:34 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Thu, 13 Nov 2014 12:37:34 +0530
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <alpine.DEB.2.02.1411112124330.4597@nftneq.ynat.uz>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>
	<54532DA5.8030107@levshin.spb.ru>
	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>
	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>
	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>
	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>
	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>
	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>
	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>
	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>
	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>
	<CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>
	<alpine.DEB.2.02.1411112124330.4597@nftneq.ynat.uz>
Message-ID: <CAGB1Vvzg3nOTqm_cmF7mxU014sHQwd21ewN-_tB2RZ1xF4gxMw@mail.gmail.com>

Minor change, have made it:

foreach($.bar in $!foo) do {...}

(note the 'do' before body)

Added do because it kinda keeps syntax similar to if (expr) then {...}

Here is the change:
https://github.com/janmejay/rsyslog/commit/d4229ed42afb759adfc4d44661fda211e37c72a0

Need to clarify desired semantics of 'set' in rscript (started separate
mail thread for that). Will send a pull request for this once we close it
on that thread.

On Wed, Nov 12, 2014 at 11:07 AM, David Lang <david at lang.hm> wrote:

> On Wed, 12 Nov 2014, singh.janmejay wrote:
>
>  How do we want foreach support to be?
>>
>> Possible places are:
>> 1. a foreach loop-construct along-side flow-control structures in rscript
>>
>
> My understanding of how rscript works is that it's mostly parsed at
> startup time, not at message processing time, so I don't see how it can be
> doen in rscript.
>
>  2. a foreach value-producing templateElement in template
>>
>> The tradeoff is, someone looking to do some useful re-structuring of
>> output, may want to call exec-template several times in approach-1, but
>> they can just emit out what they want by placing property-extractors in
>> foreach block in approch-2.
>>
>> The counter argument is, templates can't assign variables, execute
>> functions, do arithmetic-operations etc, so usefulness of foreach in a
>> template will be limited to most basic of message transformation cases.
>> Additionally, string-template as of now has no clean way of implementing
>> foreach. It'll require some involved syntax.
>>
>
> the result of templates can be assigned to variables.
>
>  To me approach-1 seems better suited for such a thing (purely because
>> if-else, the other flow control structure is implemented at this level,
>> and
>> variable assignment, functions etc make arbitrary transformation easier).
>>
>> Once we have decided where to place foreach, let us zero in on its syntax.
>> To seed the thought, how about this:
>>
>> set $.grault = "";
>> foreach($!foo as $.bar) {
>>    set $.baz = $.bar!quux;
>>    foreach($.baz as $.corge) {
>>        if ( $.corge contains "#" ) then {
>>            set $.grault = $.grault & $.corge;
>>        }
>>    }
>> }
>>
>
> in spite of everything I said above, if this can be implemented reasonably
> (without killing performance when it's not used), this would be extremely
> powerful.
>
>
>  Alternatively:
>> foreach($.bar <- $!foo) {...}
>> foreach($.bar in $!foo) {...}
>>
>
> I like this a little better than 'as', but they are almost equal and
> either is far better than the punctuation versions.
>
> David Lang
>
>
>  foreach($.bar : $!foo) {...}
>>
>>
>> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com
>> >
>> wrote:
>>
>>  Sure, I'll fork on github.
>>>
>>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <
>>> rgerhards at hq.adiscon.com
>>>
>>>> wrote:
>>>>
>>>
>>>  2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>>>
>>>>  Cool, I'll implement $!foo!bar[0].
>>>>>
>>>>> +1
>>>>>
>>>>
>>>>
>>>>  Let us process this patch-set, because is kinda hard to keep track of
>>>>> old patches and re-send in one shot.
>>>>>
>>>>>
>>>>>  would you mind cloning on github and maintain a feature branch there?
>>>> That would make it much easier for me, as I could merge the branch when
>>>> you
>>>> are done. If not, it's no problem and I'll maintain that branch.
>>>>
>>>>
>>>> i'll send the new patch once done(i'll now only get to work on it on
>>>>
>>>>> monday).
>>>>>
>>>>>
>>>>>  I haven't had a chance to look as I am now busy building test
>>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>>> Pavel has asked some questions. He recently did a lot of work on the
>>>> lib,so
>>>> it is best to coordinate that part with him.
>>>>
>>>> Rainer
>>>>
>>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>
>>>>>
>>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>>  Yes, I didn't have a need to address tokens individually, but you
>>>>>> have
>>>>>>
>>>>>>> a
>>>>>>> point.
>>>>>>>
>>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>>
>>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>>> backward
>>>>>>> compatibility. How about a function?
>>>>>>>
>>>>>>> I'll be happy to implement support for addressing it in $!...
>>>>>>> notation
>>>>>>> if
>>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>>
>>>>>>>
>>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>>> does mean that you can't have '[' in a variable name, but I don't
>>>>>> think
>>>>>> that's likely to be a real problem. I don't think there's ever a
>>>>>> really
>>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>>> syntax, it gets messy.
>>>>>>
>>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>>> function that takes the pattern and repeats it for each item in the
>>>>>> array.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>>
>>>>>>>
>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>>>> tests:
>>>>>>>>
>>>>>>>>
>>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  right, but how do you access it in rsyslog?
>>>>>>>>
>>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>>> and
>>>>>>>> get the result '10'
>>>>>>>>
>>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>>
>>>>>>>> we also don't have anything like foreach() in our template language,
>>>>>>>> which
>>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>>> JSON
>>>>>>>> string.
>>>>>>>>
>>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>>> problems
>>>>>>>> that we will have using it.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>>>> add
>>>>>>>>
>>>>>>>>  tests for it soon, but until then pasting output here is the best I
>>>>>>>>> can
>>>>>>>>> do.
>>>>>>>>>
>>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>>> semi-structured
>>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>>> way to
>>>>>>>>> represent that).
>>>>>>>>>
>>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>>> so that
>>>>>>>>> it can be pre-processed before its emitted. May be have a canned
>>>>>>>>> set
>>>>>>>>> of
>>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>>
>>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>>> this
>>>>>>>>> was
>>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>>> complex
>>>>>>>>> config for an access log with 15 - 20 fields, especially given
>>>>>>>>> those
>>>>>>>>> fields
>>>>>>>>> can have colon separated entries inside comma separated entries
>>>>>>>>> etc.
>>>>>>>>>
>>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>>> other
>>>>>>>>> (recurse) field-type.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>>>
>>>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>>> too.
>>>>>>>>>> But
>>>>>>>>>>
>>>>>>>>>>  I
>>>>>>>>>>
>>>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>>>> it
>>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>>> chance to
>>>>>>>>>>> work on it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>>>>>
>>>>>>>>>> mmnormalize
>>>>>>>>>> on
>>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>>
>>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>>> addressing yet.
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>>>> discuss
>>>>>>>>>>
>>>>>>>>>>  something else that I was thinking about.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>>
>>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>>> recurse will
>>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>>> stateless, so
>>>>>>>>>>> I
>>>>>>>>>>> don't see any problems with that. I also plan to support tag
>>>>>>>>>>> based
>>>>>>>>>>> picking
>>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>>> else, it
>>>>>>>>>>> should be considered no-match.
>>>>>>>>>>>
>>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>>> thinking
>>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>>
>>>>>>>>>>>  Lognorm mailing list
>>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Lognorm mailing list
>>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>  _______________________________________________
>>>>>>>>>
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>> http://codehunk.wordpress.com
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>>
>>
>>
>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141113/8ec6d3e4/attachment-0001.html>

From pavel at levshin.spb.ru  Fri Nov 14 13:28:06 2014
From: pavel at levshin.spb.ru (Pavel Levshin)
Date: Fri, 14 Nov 2014 15:28:06 +0300
Subject: [Lognorm] Tokenized-multivalue field-type for liblognorm
In-Reply-To: <CAGB1Vvzs0V1WyH4=JDiiLzUH0NVCf3R5A08U6S-UhkMFt=iF5g@mail.gmail.com>
References: <CAGB1VvzYqFhaGPa0=BNZ0btcjqZRryMJe1HR9ZVytAMVKTS__w@mail.gmail.com>	<54532DA5.8030107@levshin.spb.ru>	<CAGB1VvyP+Xdqw2KgwsCmNhR+WgGc5KnBoh5nb9TO8zpZeU-qQg@mail.gmail.com>	<alpine.DEB.2.02.1410310251330.6113@nftneq.ynat.uz>	<CAGB1Vvx2+G30BVBqqKjk99fC4APN0wGE3oT=AzRpnVwuYSo1Og@mail.gmail.com>	<alpine.DEB.2.02.1410310310390.6113@nftneq.ynat.uz>	<CAGB1Vvyhf=DJefWhpS4zJb92BEy0f95mpd4482qu2EQHCRJVPw@mail.gmail.com>	<alpine.DEB.2.02.1410310341220.6113@nftneq.ynat.uz>	<CAGB1Vvws8svhkHXowX59TqW__oZFqERXgRDC2peCfv13k03wYQ@mail.gmail.com>	<CADk+mPAKD0tS54a=9WfeKeKAsbs9q8B4kXEOHub1PZWwtHZCbg@mail.gmail.com>	<CAGB1VvxK9HwuLQptb48i8i4ubEAUF7GFAuEuS9jOdcsaAJKBHg@mail.gmail.com>	<CAGB1Vvx+kztVtNYUQr2VdoM5cVODXXM3GTyUETo=fOUN96bRgA@mail.gmail.com>	<CAGB1VvxjJ_+F4WSvwRezP9oHjr3ckCxP_bJv5STrzQMmLbgN2w@mail.gmail.com>
	<CAGB1Vvzs0V1WyH4=JDiiLzUH0NVCf3R5A08U6S-UhkMFt=iF5g@mail.gmail.com>
Message-ID: <5465F556.3030108@levshin.spb.ru>

Hi.

I've just merged liblognorm part. It should work even without libestr 
patches, just emits some warnings in compile time.


--
Pavel


12.11.2014 7:41, singh.janmejay:
> Sorry, this should be discussed on rsyslog mailing list, adding 
> rsyslog as well.
>
> Here are the pull requests(in order):
>
> https://github.com/rsyslog/libestr/pull/1
> https://github.com/rsyslog/liblognorm/pull/8
> https://github.com/rsyslog/rsyslog/pull/149
>
>
> On Wed, Nov 12, 2014 at 9:5


From singh.janmejay at gmail.com  Tue Nov 25 13:20:19 2014
From: singh.janmejay at gmail.com (singh.janmejay)
Date: Tue, 25 Nov 2014 17:50:19 +0530
Subject: [Lognorm] Pull request for tokenized and regex field types
Message-ID: <CAGB1VvwELUz8QL2cj6A4hEPXZCmXtXjw73WNrizsGkzjaPPb+g@mail.gmail.com>

Hi,

These patch-sets complete tokenized field-type and regex field-type support
+ the rscript side features required for effective use of json-arrays in
rulesets.

Some other changes include fixes for memory-leak and invalid memory access
in liblognorm in non-happy-path flows + testing-setup for liblognorm (with
optional and transparent valgrind support).

Summary:
- tokenized field-type (integration-tests with rsyslog, documentation etc)
- regex support (tests, integration-tests, documentation)
- memory access/leak bug fixes
- testing env setup for liblognorm
- rscript support for json-array subscripting and 'foreach' loop
- rscript support for 'reset' statement, which as opposed to 'set' always
overwrites old value, regardless of the type)
- dedicated page for rscript control-structures
- detailed documentation around behaviour of rscript 'set', 'unset' and
'reset'

The patch-sets go in the following order:
https://github.com/rsyslog/liblognorm/pull/9
https://github.com/rsyslog/rsyslog/pull/149
https://github.com/rsyslog/rsyslog-doc/pull/98

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141125/57c2bced/attachment.html>