[Lognorm] Tokenized-multivalue field-type for liblognorm

[singh.janmejay]

Yes, I didn't have a need to address tokens individually, but you have a
point.

Any suggestions on what we want to do for addressing array elements?

I wonder if its possible to do in $!... notation without breaking backward
compatibility. How about a function?

I'll be happy to implement support for addressing it in $!... notation if
don't mind breaking a corner case in backward compatibility. Eg.
$!foo!bar![0] ? Its kinda ugly though, or so I think.

On Fri, Oct 31, 2014 at 3:44 PM, David Lang <david at lang.hm> wrote:

> On Fri, 31 Oct 2014, singh.janmejay wrote:
>
>  It writes it as a json array, here is a fragment from my manual tests:
>>
>> [ "15", "26", "15" ]
>>
>
> right, but how do you access it in rsyslog?
>
> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar and
> get the result '10'
>
> what would you use to access the value '26' in your example?
>
> we also don't have anything like foreach() in our template language, which
> makes it hard to make use of these values as anything other than a JSON
> string.
>
> I'm not saying that it's not useful, but I am pointing out the problems
> that we will have using it.
>
> David Lang
>
>
>  It was using time in hh:mm:ss format and tokening by colon(:). I'll add
>> tests for it soon, but until then pasting output here is the best I can
>> do.
>>
>> The idea behind this is to generate structured content from
>> semi-structured
>> or unstructured log messages. So array is a good representation for
>> tokenized-value (it is multi-valued by nature, and array is a good way to
>> represent that).
>>
>> But eventually we should allow user to register value-transformers so that
>> it can be pre-processed before its emitted. May be have a canned set of
>> transformers, and allow user to plug in new ones.
>>
>> My first instinct was to utilize variable support for this, infact this
>> was
>> the motivator for variable support. But it still leads to a fairly complex
>> config for an access log with 15 - 20 fields, especially given those
>> fields
>> can have colon separated entries inside comma separated entries etc.
>>
>> So I felt the need for a simpler way of doing it, hence this and other
>> (recurse) field-type.
>>
>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:
>>
>>  On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>
>>>  Tokenizer followed by tokenizer is something that I have in mind too.
>>> But
>>>
>>>> I
>>>> promised myself that i'd write a test for that instead of testing it
>>>> manually :-). Will add that patch on this thread once I get a chance to
>>>> work on it.
>>>>
>>>>
>>> At least in the short term, you can use the ability to call mmnormalize
>>> on
>>> a variable to parse subvariables.
>>>
>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>> addressing yet.
>>>
>>> David Lang
>>>
>>>
>>>  However, since you are asking about those kind of forms, let met discuss
>>>
>>>> something else that I was thinking about.
>>>>
>>>> The idea is to have another field type called recurse.
>>>>
>>>> Similar to how tokenized uses a ctx to parse matching text, recurse will
>>>> parse it using the current context. AFAIK, the context is stateless, so
>>>> I
>>>> don't see any problems with that. I also plan to support tag based
>>>> picking
>>>> of which rules the text may match, and if it matches something else, it
>>>> should be considered no-match.
>>>>
>>>> Instead of typing it out here, i'll attach a picture I took after
>>>> thinking
>>>> through it briefly(i'll attach it to the next mail).
>>>>
>>>>
>>>>  _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>> _______________________________________________
>>> Lognorm mailing list
>>> Lognorm at lists.adiscon.com
>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>
>>>
>>>
>>
>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141031/21274db3/attachment-0001.html>