[Lognorm] Tokenized-multivalue field-type for liblognorm

[singh.janmejay]

It writes it as a json array, here is a fragment from my manual tests:

[ "15", "26", "15" ]

It was using time in hh:mm:ss format and tokening by colon(:). I'll add
tests for it soon, but until then pasting output here is the best I can do.

The idea behind this is to generate structured content from semi-structured
or unstructured log messages. So array is a good representation for
tokenized-value (it is multi-valued by nature, and array is a good way to
represent that).

But eventually we should allow user to register value-transformers so that
it can be pre-processed before its emitted. May be have a canned set of
transformers, and allow user to plug in new ones.

My first instinct was to utilize variable support for this, infact this was
the motivator for variable support. But it still leads to a fairly complex
config for an access log with 15 - 20 fields, especially given those fields
can have colon separated entries inside comma separated entries etc.

So I felt the need for a simpler way of doing it, hence this and other
(recurse) field-type.

On Fri, Oct 31, 2014 at 3:23 PM, David Lang <david at lang.hm> wrote:

> On Fri, 31 Oct 2014, singh.janmejay wrote:
>
>  Tokenizer followed by tokenizer is something that I have in mind too. But
>> I
>> promised myself that i'd write a test for that instead of testing it
>> manually :-). Will add that patch on this thread once I get a chance to
>> work on it.
>>
>
> At least in the short term, you can use the ability to call mmnormalize on
> a variable to parse subvariables.
>
> How are the resulting fields addressed? Rsyslog hasn't had array
> addressing yet.
>
> David Lang
>
>
>  However, since you are asking about those kind of forms, let met discuss
>> something else that I was thinking about.
>>
>> The idea is to have another field type called recurse.
>>
>> Similar to how tokenized uses a ctx to parse matching text, recurse will
>> parse it using the current context. AFAIK, the context is stateless, so I
>> don't see any problems with that. I also plan to support tag based picking
>> of which rules the text may match, and if it matches something else, it
>> should be considered no-match.
>>
>> Instead of typing it out here, i'll attach a picture I took after thinking
>> through it briefly(i'll attach it to the next mail).
>>
>>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>


-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141031/ce965713/attachment.html>