[Lognorm] Tokenized-multivalue field-type for liblognorm

[David Lang]

On Thu, 30 Oct 2014, singh.janmejay wrote:

> Hi,
>
> This patch-set introduces a log-norm field-type called tokenized, which
> allows parsing of token-separated values.
>
> A lot of applications such as nginx write fields in logs that are
> comma+space separated etc. For instance, nginx upstream_addrs field writes
> comma-separated ip+port combinations to access logs.
>
> Parsing such logs takes significant amount of regex and exec-template work
> and leads to rather ugly solution for something as simple as tokenized
> string.
>
> With this patch, parsing a list of ip-addresses separated by ', '(comma +
> space) for instance, would require a rule similar to:
>
> rule=ips:%my_ips:tokenized:, :ipv4%

What terminates the list?

It looks like this allows multi-character tokens, is that correct?

David Lang

> This requires a small patch to libestr as well, so this mail has 3 patches
> attached.
>
> libestr patch:
>
> 0001-Changed-some-functions-that-don-t-modify-their-arg-t.patch
>
> liblognorm patch:
>
> 0001-Moved-from-parser-receving-data-as-escaped-string-to.patch
> 0002-added-support-for-field_type-tokenized-which-parses-.patch
>
> Patches go in order of prefix-number.
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-added-support-for-field_type-tokenized-which-parses-.patch
Type: text/x-patch
Size: 9316 bytes
Desc: 
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141030/60f5986d/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Changed-some-functions-that-don-t-modify-their-arg-t.patch
Type: text/x-patch
Size: 3202 bytes
Desc: 
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141030/60f5986d/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Moved-from-parser-receving-data-as-escaped-string-to.patch
Type: text/x-patch
Size: 8336 bytes
Desc: 
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20141030/60f5986d/attachment-0005.bin>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm