[Lognorm] Tokenized-multivalue field-type for liblognorm

[singh.janmejay]

The token-string can be escaped using the same mechanism as char-to. Eg.
\x3a for colon(:) etc.

Also, the tokenized field-type allows user to pick the field-type of each
field on tokenized-fragment and it produces a multi-valued variable(its a
json-array), similar to event.tags.


On Thu, Oct 30, 2014 at 4:33 PM, singh.janmejay <singh.janmejay at gmail.com>
wrote:

> Hi,
>
> This patch-set introduces a log-norm field-type called tokenized, which
> allows parsing of token-separated values.
>
> A lot of applications such as nginx write fields in logs that are
> comma+space separated etc. For instance, nginx upstream_addrs field writes
> comma-separated ip+port combinations to access logs.
>
> Parsing such logs takes significant amount of regex and exec-template work
> and leads to rather ugly solution for something as simple as tokenized
> string.
>
> With this patch, parsing a list of ip-addresses separated by ', '(comma +
> space) for instance, would require a rule similar to:
>
> rule=ips:%my_ips:tokenized:, :ipv4%
>
> This requires a small patch to libestr as well, so this mail has 3 patches
> attached.
>
> libestr patch:
>
> 0001-Changed-some-functions-that-don-t-modify-their-arg-t.patch
>
> liblognorm patch:
>
> 0001-Moved-from-parser-receving-data-as-escaped-string-to.patch
> 0002-added-support-for-field_type-tokenized-which-parses-.patch
>
> Patches go in order of prefix-number.
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
>



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141030/5a254ee7/attachment.html>