[Lognorm] Tokenized-multivalue field-type for liblognorm

singh.janmejay singh.janmejay at gmail.com
Wed Nov 12 05:25:31 CET 2014 

The array-subscripting patches are ready, I'll send a pull request on
github.

It is of the form $!foo[1]!bar[2]!baz.

On Wed, Nov 12, 2014 at 9:52 AM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> How do we want foreach support to be?
>
> Possible places are:
> 1. a foreach loop-construct along-side flow-control structures in rscript
> 2. a foreach value-producing templateElement in template
>
> The tradeoff is, someone looking to do some useful re-structuring of
> output, may want to call exec-template several times in approach-1, but
> they can just emit out what they want by placing property-extractors in
> foreach block in approch-2.
>
> The counter argument is, templates can't assign variables, execute
> functions, do arithmetic-operations etc, so usefulness of foreach in a
> template will be limited to most basic of message transformation cases.
> Additionally, string-template as of now has no clean way of implementing
> foreach. It'll require some involved syntax.
>
> To me approach-1 seems better suited for such a thing (purely because
> if-else, the other flow control structure is implemented at this level, and
> variable assignment, functions etc make arbitrary transformation easier).
>
> Once we have decided where to place foreach, let us zero in on its syntax.
> To seed the thought, how about this:
>
> set $.grault = "";
> foreach($!foo as $.bar) {
>     set $.baz = $.bar!quux;
>     foreach($.baz as $.corge) {
>         if ( $.corge contains "#" ) then {
>             set $.grault = $.grault & $.corge;
>         }
>     }
> }
>
> Alternatively:
> foreach($.bar <- $!foo) {...}
> foreach($.bar in $!foo) {...}
> foreach($.bar : $!foo) {...}
>
>
>
> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com>
> wrote:
>
>> Sure, I'll fork on github.
>>
>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <
>> rgerhards at hq.adiscon.com> wrote:
>>
>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>>
>>>> Cool, I'll implement $!foo!bar[0].
>>>>
>>>> +1
>>>
>>>
>>>> Let us process this patch-set, because is kinda hard to keep track of
>>>> old patches and re-send in one shot.
>>>>
>>>>
>>> would you mind cloning on github and maintain a feature branch there?
>>> That would make it much easier for me, as I could merge the branch when you
>>> are done. If not, it's no problem and I'll maintain that branch.
>>>
>>>
>>> i'll send the new patch once done(i'll now only get to work on it on
>>>> monday).
>>>>
>>>>
>>> I haven't had a chance to look as I am now busy building test
>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>>> it is best to coordinate that part with him.
>>>
>>> Rainer
>>>
>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>
>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>>
>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>
>>>>>  Yes, I didn't have a need to address tokens individually, but you
>>>>>> have a
>>>>>> point.
>>>>>>
>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>
>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>> backward
>>>>>> compatibility. How about a function?
>>>>>>
>>>>>> I'll be happy to implement support for addressing it in $!...
>>>>>> notation if
>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>
>>>>>
>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>>> that's likely to be a real problem. I don't think there's ever a really
>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>> syntax, it gets messy.
>>>>>
>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>> function that takes the pattern and repeats it for each item in the array.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>>
>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>>> tests:
>>>>>>>
>>>>>>>>
>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>
>>>>>>>>
>>>>>>> right, but how do you access it in rsyslog?
>>>>>>>
>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>> and
>>>>>>> get the result '10'
>>>>>>>
>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>
>>>>>>> we also don't have anything like foreach() in our template language,
>>>>>>> which
>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>> JSON
>>>>>>> string.
>>>>>>>
>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>> problems
>>>>>>> that we will have using it.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>>> add
>>>>>>>
>>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>>> can
>>>>>>>> do.
>>>>>>>>
>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>> semi-structured
>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>> way to
>>>>>>>> represent that).
>>>>>>>>
>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>> so that
>>>>>>>> it can be pre-processed before its emitted. May be have a canned
>>>>>>>> set of
>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>
>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>> this
>>>>>>>> was
>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>> complex
>>>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>>>> fields
>>>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>>>
>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>> other
>>>>>>>> (recurse) field-type.
>>>>>>>>
>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>>
>>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>> too.
>>>>>>>>> But
>>>>>>>>>
>>>>>>>>>  I
>>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>>> it
>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>> chance to
>>>>>>>>>> work on it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>>> mmnormalize
>>>>>>>>> on
>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>
>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>> addressing yet.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>>> discuss
>>>>>>>>>
>>>>>>>>>  something else that I was thinking about.
>>>>>>>>>>
>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>
>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>> recurse will
>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>> stateless, so
>>>>>>>>>> I
>>>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>>>> picking
>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>> else, it
>>>>>>>>>> should be considered no-match.
>>>>>>>>>>
>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>> thinking
>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  _______________________________________________
>>>>>>>>>>
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Lognorm mailing list
>>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Janmejay
>>>> http://codehunk.wordpress.com
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141112/4a0d879c/attachment.html>

More information about the Lognorm mailing list