[Lognorm] Tokenized-multivalue field-type for liblognorm

Wed Nov 12 05:22:08 CET 2014

How do we want foreach support to be?

Possible places are:
1. a foreach loop-construct along-side flow-control structures in rscript
2. a foreach value-producing templateElement in template

The tradeoff is, someone looking to do some useful re-structuring of
output, may want to call exec-template several times in approach-1, but
they can just emit out what they want by placing property-extractors in
foreach block in approch-2.

The counter argument is, templates can't assign variables, execute
functions, do arithmetic-operations etc, so usefulness of foreach in a
template will be limited to most basic of message transformation cases.
Additionally, string-template as of now has no clean way of implementing
foreach. It'll require some involved syntax.

To me approach-1 seems better suited for such a thing (purely because
if-else, the other flow control structure is implemented at this level, and
variable assignment, functions etc make arbitrary transformation easier).

Once we have decided where to place foreach, let us zero in on its syntax.
To seed the thought, how about this:

set $.grault = "";
foreach($!foo as $.bar) {
    set $.baz = $.bar!quux;
    foreach($.baz as $.corge) {
        if ( $.corge contains "#" ) then {
            set $.grault = $.grault & $.corge;
        }
    }
}

Alternatively:
foreach($.bar <- $!foo) {...}
foreach($.bar in $!foo) {...}
foreach($.bar : $!foo) {...}

On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> Sure, I'll fork on github.
>
> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <rgerhards at hq.adiscon.com
> > wrote:
>
>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>>
>>> Cool, I'll implement $!foo!bar[0].
>>>
>>> +1
>>
>>
>>> Let us process this patch-set, because is kinda hard to keep track of
>>> old patches and re-send in one shot.
>>>
>>>
>> would you mind cloning on github and maintain a feature branch there?
>> That would make it much easier for me, as I could merge the branch when you
>> are done. If not, it's no problem and I'll maintain that branch.
>>
>>
>> i'll send the new patch once done(i'll now only get to work on it on
>>> monday).
>>>
>>>
>> I haven't had a chance to look as I am now busy building test
>> environments and looking at the testbench [yes, one guy!] ;) But I see
>> Pavel has asked some questions. He recently did a lot of work on the lib,so
>> it is best to coordinate that part with him.
>>
>> Rainer
>>
>> Do existing patches look ok except for the indexed-addressing feature?
>>>
>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>>
>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>
>>>>  Yes, I didn't have a need to address tokens individually, but you have
>>>>> a
>>>>> point.
>>>>>
>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>
>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>> backward
>>>>> compatibility. How about a function?
>>>>>
>>>>> I'll be happy to implement support for addressing it in $!... notation
>>>>> if
>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>
>>>>
>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>> does mean that you can't have '[' in a variable name, but I don't think
>>>> that's likely to be a real problem. I don't think there's ever a really
>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>> syntax, it gets messy.
>>>>
>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>> function that takes the pattern and repeats it for each item in the array.
>>>>
>>>> David Lang
>>>>
>>>>
>>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>>  It writes it as a json array, here is a fragment from my manual
>>>>>> tests:
>>>>>>
>>>>>>>
>>>>>>> [ "15", "26", "15" ]
>>>>>>>
>>>>>>>
>>>>>> right, but how do you access it in rsyslog?
>>>>>>
>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>> and
>>>>>> get the result '10'
>>>>>>
>>>>>> what would you use to access the value '26' in your example?
>>>>>>
>>>>>> we also don't have anything like foreach() in our template language,
>>>>>> which
>>>>>> makes it hard to make use of these values as anything other than a
>>>>>> JSON
>>>>>> string.
>>>>>>
>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>> problems
>>>>>> that we will have using it.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>> add
>>>>>>
>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>> can
>>>>>>> do.
>>>>>>>
>>>>>>> The idea behind this is to generate structured content from
>>>>>>> semi-structured
>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>> way to
>>>>>>> represent that).
>>>>>>>
>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>> so that
>>>>>>> it can be pre-processed before its emitted. May be have a canned set
>>>>>>> of
>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>
>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>> this
>>>>>>> was
>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>> complex
>>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>>> fields
>>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>>
>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>> other
>>>>>>> (recurse) field-type.
>>>>>>>
>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>>
>>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>> too.
>>>>>>>> But
>>>>>>>>
>>>>>>>>  I
>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>> it
>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>> chance to
>>>>>>>>> work on it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>>> mmnormalize
>>>>>>>> on
>>>>>>>> a variable to parse subvariables.
>>>>>>>>
>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>> addressing yet.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>>> discuss
>>>>>>>>
>>>>>>>>  something else that I was thinking about.
>>>>>>>>>
>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>
>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>> recurse will
>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>> stateless, so
>>>>>>>>> I
>>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>>> picking
>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>> else, it
>>>>>>>>> should be considered no-match.
>>>>>>>>>
>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>> thinking
>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  _______________________________________________
>>>>>>>>>
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> Lognorm at lists.adiscon.com
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141112/abe94ebe/attachment-0001.html>