[Lognorm] Tokenized-multivalue field-type for liblognorm

Fri Oct 31 14:16:53 CET 2014

Sure, I'll fork on github.

On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <rgerhards at hq.adiscon.com>
wrote:

> 2014-10-31 14:08 GMT+01:00 singh.janmejay <singh.janmejay at gmail.com>:
>
>> Cool, I'll implement $!foo!bar[0].
>>
>> +1
>
>
>> Let us process this patch-set, because is kinda hard to keep track of old
>> patches and re-send in one shot.
>>
>>
> would you mind cloning on github and maintain a feature branch there? That
> would make it much easier for me, as I could merge the branch when you are
> done. If not, it's no problem and I'll maintain that branch.
>
>
> i'll send the new patch once done(i'll now only get to work on it on
>> monday).
>>
>>
> I haven't had a chance to look as I am now busy building test environments
> and looking at the testbench [yes, one guy!] ;) But I see Pavel has asked
> some questions. He recently did a lot of work on the lib,so it is best to
> coordinate that part with him.
>
> Rainer
>
> Do existing patches look ok except for the indexed-addressing feature?
>>
>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:
>>
>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>
>>>  Yes, I didn't have a need to address tokens individually, but you have a
>>>> point.
>>>>
>>>> Any suggestions on what we want to do for addressing array elements?
>>>>
>>>> I wonder if its possible to do in $!... notation without breaking
>>>> backward
>>>> compatibility. How about a function?
>>>>
>>>> I'll be happy to implement support for addressing it in $!... notation
>>>> if
>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>
>>>
>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>> does mean that you can't have '[' in a variable name, but I don't think
>>> that's likely to be a real problem. I don't think there's ever a really
>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>> syntax, it gets messy.
>>>
>>> for templates, we probably need some sort of foreach(array, pattern)
>>> function that takes the pattern and repeats it for each item in the array.
>>>
>>> David Lang
>>>
>>>
>>>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>>>
>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>
>>>>>  It writes it as a json array, here is a fragment from my manual tests:
>>>>>
>>>>>>
>>>>>> [ "15", "26", "15" ]
>>>>>>
>>>>>>
>>>>> right, but how do you access it in rsyslog?
>>>>>
>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar and
>>>>> get the result '10'
>>>>>
>>>>> what would you use to access the value '26' in your example?
>>>>>
>>>>> we also don't have anything like foreach() in our template language,
>>>>> which
>>>>> makes it hard to make use of these values as anything other than a JSON
>>>>> string.
>>>>>
>>>>> I'm not saying that it's not useful, but I am pointing out the problems
>>>>> that we will have using it.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>> add
>>>>>
>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>> can
>>>>>> do.
>>>>>>
>>>>>> The idea behind this is to generate structured content from
>>>>>> semi-structured
>>>>>> or unstructured log messages. So array is a good representation for
>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>> way to
>>>>>> represent that).
>>>>>>
>>>>>> But eventually we should allow user to register value-transformers so
>>>>>> that
>>>>>> it can be pre-processed before its emitted. May be have a canned set
>>>>>> of
>>>>>> transformers, and allow user to plug in new ones.
>>>>>>
>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>> this
>>>>>> was
>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>> complex
>>>>>> config for an access log with 15 - 20 fields, especially given those
>>>>>> fields
>>>>>> can have colon separated entries inside comma separated entries etc.
>>>>>>
>>>>>> So I felt the need for a simpler way of doing it, hence this and other
>>>>>> (recurse) field-type.
>>>>>>
>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>>>
>>>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>>>
>>>>>>>  Tokenizer followed by tokenizer is something that I have in mind
>>>>>>> too.
>>>>>>> But
>>>>>>>
>>>>>>>  I
>>>>>>>> promised myself that i'd write a test for that instead of testing it
>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>> chance to
>>>>>>>> work on it.
>>>>>>>>
>>>>>>>>
>>>>>>>>  At least in the short term, you can use the ability to call
>>>>>>> mmnormalize
>>>>>>> on
>>>>>>> a variable to parse subvariables.
>>>>>>>
>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>> addressing yet.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>>
>>>>>>>  However, since you are asking about those kind of forms, let met
>>>>>>> discuss
>>>>>>>
>>>>>>>  something else that I was thinking about.
>>>>>>>>
>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>
>>>>>>>> Similar to how tokenized uses a ctx to parse matching text, recurse
>>>>>>>> will
>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>> stateless, so
>>>>>>>> I
>>>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>>>> picking
>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>> else, it
>>>>>>>> should be considered no-match.
>>>>>>>>
>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>> thinking
>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>>>>>
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Lognorm mailing list
>>>>>>> Lognorm at lists.adiscon.com
>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>

-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141031/429c6df7/attachment.html>