[Lognorm] Tokenized-multivalue field-type for liblognorm

Fri Oct 31 14:12:29 CET 2014

2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:

> Cool, I'll implement $!foo!bar[0].
>
> +1


> Let us process this patch-set, because is kinda hard to keep track of old
> patches and re-send in one shot.
>
>
would you mind cloning on github and maintain a feature branch there? That
would make it much easier for me, as I could merge the branch when you are
done. If not, it's no problem and I'll maintain that branch.


i'll send the new patch once done(i'll now only get to work on it on
> monday).
>
>
I haven't had a chance to look as I am now busy building test environments
and looking at the testbench [yes, one guy!] ;) But I see Pavel has asked
some questions. He recently did a lot of work on the lib,so it is best to
coordinate that part with him.

Rainer

Do existing patches look ok except for the indexed-addressing feature?
>
> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>
>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>
>>  Yes, I didn't have a need to address tokens individually, but you have a
>>> point.
>>>
>>> Any suggestions on what we want to do for addressing array elements?
>>>
>>> I wonder if its possible to do in $!... notation without breaking
>>> backward
>>> compatibility. How about a function?
>>>
>>> I'll be happy to implement support for addressing it in $!... notation if
>>> don't mind breaking a corner case in backward compatibility. Eg.
>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>
>>
>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>> does mean that you can't have '[' in a variable name, but I don't think
>> that's likely to be a real problem. I don't think there's ever a really
>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>> syntax, it gets messy.
>>
>> for templates, we probably need some sort of foreach(array, pattern)
>> function that takes the pattern and repeats it for each item in the array.
>>
>> David Lang
>>
>>
>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>
>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>
>>>>  It writes it as a json array, here is a fragment from my manual tests:
>>>>
>>>>>
>>>>> [ "15", "26", "15" ]
>>>>>
>>>>>
>>>> right, but how do you access it in rsyslog?
>>>>
>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar and
>>>> get the result '10'
>>>>
>>>> what would you use to access the value '26' in your example?
>>>>
>>>> we also don't have anything like foreach() in our template language,
>>>> which
>>>> makes it hard to make use of these values as anything other than a JSON
>>>> string.
>>>>
>>>> I'm not saying that it's not useful, but I am pointing out the problems
>>>> that we will have using it.
>>>>
>>>> David Lang
>>>>
>>>>
>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll add
>>>>
>>>>> tests for it soon, but until then pasting output here is the best I can
>>>>> do.
>>>>>
>>>>> The idea behind this is to generate structured content from
>>>>> semi-structured
>>>>> or unstructured log messages. So array is a good representation for
>>>>> tokenized-value (it is multi-valued by nature, and array is a good way
>>>>> to
>>>>> represent that).
>>>>>
>>>>> But eventually we should allow user to register value-transformers so
>>>>> that
>>>>> it can be pre-processed before its emitted. May be have a canned set of
>>>>> transformers, and allow user to plug in new ones.
>>>>>
>>>>> My first instinct was to utilize variable support for this, infact this
>>>>> was
>>>>> the motivator for variable support. But it still leads to a fairly
>>>>> complex
>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>> fields
>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>
>>>>> So I felt the need for a simpler way of doing it, hence this and other
>>>>> (recurse) field-type.
>>>>>
>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>
>>>>>>
>>>>>>  Tokenizer followed by tokenizer is something that I have in mind too.
>>>>>> But
>>>>>>
>>>>>>  I
>>>>>>> promised myself that i'd write a test for that instead of testing it
>>>>>>> manually :-). Will add that patch on this thread once I get a chance
>>>>>>> to
>>>>>>> work on it.
>>>>>>>
>>>>>>>
>>>>>>>  At least in the short term, you can use the ability to call
>>>>>> mmnormalize
>>>>>> on
>>>>>> a variable to parse subvariables.
>>>>>>
>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>> addressing yet.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>> discuss
>>>>>>
>>>>>>  something else that I was thinking about.
>>>>>>>
>>>>>>> The idea is to have another field type called recurse.
>>>>>>>
>>>>>>> Similar to how tokenized uses a ctx to parse matching text, recurse
>>>>>>> will
>>>>>>> parse it using the current context. AFAIK, the context is stateless,
>>>>>>> so
>>>>>>> I
>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>> picking
>>>>>>> of which rules the text may match, and if it matches something else,
>>>>>>> it
>>>>>>> should be considered no-match.
>>>>>>>
>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>> thinking
>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>>>
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> Lognorm at lists.adiscon.com
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>  _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141031/6d5cf2dd/attachment-0001.html>