[Lognorm] Tokenized-multivalue field-type for liblognorm

[David Lang]

On Fri, 31 Oct 2014, singh.janmejay wrote:

> Yes, I didn't have a need to address tokens individually, but you have a
> point.
>
> Any suggestions on what we want to do for addressing array elements?
>
> I wonder if its possible to do in $!... notation without breaking backward
> compatibility. How about a function?
>
> I'll be happy to implement support for addressing it in $!... notation if
> don't mind breaking a corner case in backward compatibility. Eg.
> $!foo!bar![0] ? Its kinda ugly though, or so I think.

I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It does mean 
that you can't have '[' in a variable name, but I don't think that's likely to 
be a real problem. I don't think there's ever a really clean way to do something 
like $!foo[2]!bar[2]!baz no matter what your syntax, it gets messy.

for templates, we probably need some sort of foreach(array, pattern) function 
that takes the pattern and repeats it for each item in the array.

David Lang

> On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:
>
>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>
>>  It writes it as a json array, here is a fragment from my manual tests:
>>>
>>> [ "15", "26", "15" ]
>>>
>>
>> right, but how do you access it in rsyslog?
>>
>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar and
>> get the result '10'
>>
>> what would you use to access the value '26' in your example?
>>
>> we also don't have anything like foreach() in our template language, which
>> makes it hard to make use of these values as anything other than a JSON
>> string.
>>
>> I'm not saying that it's not useful, but I am pointing out the problems
>> that we will have using it.
>>
>> David Lang
>>
>>
>>  It was using time in hh:mm:ss format and tokening by colon(:). I'll add
>>> tests for it soon, but until then pasting output here is the best I can
>>> do.
>>>
>>> The idea behind this is to generate structured content from
>>> semi-structured
>>> or unstructured log messages. So array is a good representation for
>>> tokenized-value (it is multi-valued by nature, and array is a good way to
>>> represent that).
>>>
>>> But eventually we should allow user to register value-transformers so that
>>> it can be pre-processed before its emitted. May be have a canned set of
>>> transformers, and allow user to plug in new ones.
>>>
>>> My first instinct was to utilize variable support for this, infact this
>>> was
>>> the motivator for variable support. But it still leads to a fairly complex
>>> config for an access log with 15 - 20 fields, especially given those
>>> fields
>>> can have colon separated entries inside comma separated entries etc.
>>>
>>> So I felt the need for a simpler way of doing it, hence this and other
>>> (recurse) field-type.
>>>
>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>>
>>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>
>>>>  Tokenizer followed by tokenizer is something that I have in mind too.
>>>> But
>>>>
>>>>> I
>>>>> promised myself that i'd write a test for that instead of testing it
>>>>> manually :-). Will add that patch on this thread once I get a chance to
>>>>> work on it.
>>>>>
>>>>>
>>>> At least in the short term, you can use the ability to call mmnormalize
>>>> on
>>>> a variable to parse subvariables.
>>>>
>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>> addressing yet.
>>>>
>>>> David Lang
>>>>
>>>>
>>>>  However, since you are asking about those kind of forms, let met discuss
>>>>
>>>>> something else that I was thinking about.
>>>>>
>>>>> The idea is to have another field type called recurse.
>>>>>
>>>>> Similar to how tokenized uses a ctx to parse matching text, recurse will
>>>>> parse it using the current context. AFAIK, the context is stateless, so
>>>>> I
>>>>> don't see any problems with that. I also plan to support tag based
>>>>> picking
>>>>> of which rules the text may match, and if it matches something else, it
>>>>> should be considered no-match.
>>>>>
>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>> thinking
>>>>> through it briefly(i'll attach it to the next mail).
>>>>>
>>>>>
>>>>>  _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> Lognorm at lists.adiscon.com
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>> _______________________________________________
>> Lognorm mailing list
>> Lognorm at lists.adiscon.com
>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>
>>
>
>
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm