[Lognorm] Tokenized-multivalue field-type for liblognorm

[singh.janmejay]

Cool, I'll implement $!foo!bar[0].

Let us process this patch-set, because is kinda hard to keep track of old
patches and re-send in one shot.

i'll send the new patch once done(i'll now only get to work on it on
monday).

Do existing patches look ok except for the indexed-addressing feature?

On Fri, Oct 31, 2014 at 4:15 PM, David Lang <david at lang.hm> wrote:

> On Fri, 31 Oct 2014, singh.janmejay wrote:
>
>  Yes, I didn't have a need to address tokens individually, but you have a
>> point.
>>
>> Any suggestions on what we want to do for addressing array elements?
>>
>> I wonder if its possible to do in $!... notation without breaking backward
>> compatibility. How about a function?
>>
>> I'll be happy to implement support for addressing it in $!... notation if
>> don't mind breaking a corner case in backward compatibility. Eg.
>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>
>
> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It does
> mean that you can't have '[' in a variable name, but I don't think that's
> likely to be a real problem. I don't think there's ever a really clean way
> to do something like $!foo[2]!bar[2]!baz no matter what your syntax, it
> gets messy.
>
> for templates, we probably need some sort of foreach(array, pattern)
> function that takes the pattern and repeats it for each item in the array.
>
> David Lang
>
>
>  On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>>
>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>
>>>  It writes it as a json array, here is a fragment from my manual tests:
>>>
>>>>
>>>> [ "15", "26", "15" ]
>>>>
>>>>
>>> right, but how do you access it in rsyslog?
>>>
>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar and
>>> get the result '10'
>>>
>>> what would you use to access the value '26' in your example?
>>>
>>> we also don't have anything like foreach() in our template language,
>>> which
>>> makes it hard to make use of these values as anything other than a JSON
>>> string.
>>>
>>> I'm not saying that it's not useful, but I am pointing out the problems
>>> that we will have using it.
>>>
>>> David Lang
>>>
>>>
>>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll add
>>>
>>>> tests for it soon, but until then pasting output here is the best I can
>>>> do.
>>>>
>>>> The idea behind this is to generate structured content from
>>>> semi-structured
>>>> or unstructured log messages. So array is a good representation for
>>>> tokenized-value (it is multi-valued by nature, and array is a good way
>>>> to
>>>> represent that).
>>>>
>>>> But eventually we should allow user to register value-transformers so
>>>> that
>>>> it can be pre-processed before its emitted. May be have a canned set of
>>>> transformers, and allow user to plug in new ones.
>>>>
>>>> My first instinct was to utilize variable support for this, infact this
>>>> was
>>>> the motivator for variable support. But it still leads to a fairly
>>>> complex
>>>> config for an access log with 15 - 20 fields, especially given those
>>>> fields
>>>> can have colon separated entries inside comma separated entries etc.
>>>>
>>>> So I felt the need for a simpler way of doing it, hence this and other
>>>> (recurse) field-type.
>>>>
>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>>
>>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>
>>>>>
>>>>>  Tokenizer followed by tokenizer is something that I have in mind too.
>>>>> But
>>>>>
>>>>>  I
>>>>>> promised myself that i'd write a test for that instead of testing it
>>>>>> manually :-). Will add that patch on this thread once I get a chance
>>>>>> to
>>>>>> work on it.
>>>>>>
>>>>>>
>>>>>>  At least in the short term, you can use the ability to call
>>>>> mmnormalize
>>>>> on
>>>>> a variable to parse subvariables.
>>>>>
>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>> addressing yet.
>>>>>
>>>>> David Lang
>>>>>
>>>>>
>>>>>  However, since you are asking about those kind of forms, let met
>>>>> discuss
>>>>>
>>>>>  something else that I was thinking about.
>>>>>>
>>>>>> The idea is to have another field type called recurse.
>>>>>>
>>>>>> Similar to how tokenized uses a ctx to parse matching text, recurse
>>>>>> will
>>>>>> parse it using the current context. AFAIK, the context is stateless,
>>>>>> so
>>>>>> I
>>>>>> don't see any problems with that. I also plan to support tag based
>>>>>> picking
>>>>>> of which rules the text may match, and if it matches something else,
>>>>>> it
>>>>>> should be considered no-match.
>>>>>>
>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>> thinking
>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>>>
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> Lognorm at lists.adiscon.com
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>  _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>>
>>
>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141031/c3fd0f77/attachment.html>