[rsyslog] working with list templates

Brian Knox briank at talksum.com
Mon Jan 7 19:09:43 CET 2013


Hm. Interesting question.  Let me think...

I definitely will not always include one field only - say I wanted to do 
something in redis like create counts of how many messages I received, 
by hostname, for each programname.  The redis command would be

HINCR <programname> <hostname>

so

HINCR rsyslog myhost.mydomain.org

Would create a hash at key "rsyslog" with a field of 
"myhost.mydomain.org" and increment a counter at that field by 1.

The other thing I need to be able to do is make sure I can concatenate 
fields into a single value...

Like if I wanted to make make a combined key of programname and hostname 
with a delimiter...

"INCR rsyslog|myhost.mydomain.org"

That case is pretty easy to handle with a string template though... 
"INCR %programname%|%hostname%" works fine in that case.

Brian

On 1/7/2013 12:34 PM, Rainer Gerhards wrote:
> Just to make sure I did get this straight: This means you always include one field only?
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Brian Knox
>> Sent: Monday, January 07, 2013 5:18 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] working with list templates
>>
>> So - in my case I do not believe I need field names.  Redis is more of
>> a
>> k/v store than a document store like mongo (although it supports more
>> data
>> structures than simply key / value, like sets, hashes, and lists).
>>
>> With the hiredis api there are basically two ways of passing a command
>> to
>> redis.  For the sake of example, let's take the command I played in the
>> previous example, with would push the content of the msg property into
>> a
>> list in redis.
>>
>> Assuming "msg" contains the rsyslog message itself ("my message")
>>
>> 1) Using sprintf like formatting
>>
>> redisCommand(context, "LPUSH mylist %s", msg);
>>
>> 2) Passing an array of char arrays
>>
>> something like this:
>>
>> const char *argv[3];
>> argv[0] = "LPUSH";
>> argv[1] = "mylist";
>> argv[2] = msg;
>> size_t lens[3] = { 5, 6, 10 };
>> int argc = 3;
>>
>> redisFormatCommandArgv(&cmd,argc,argv,lens);
>>
>> The way my current omhiredis module works is to simply call
>>
>> rc = redisAppendCommand(pData->conn, (char*)message);
>>
>> This works fine for doing things like incrementing counters
>>
>> template(name="messages_processed" type="string" string="INCR processed
>> 1")
>>
>> However, if I include the actual log line, the spaces in the log line
>> break
>> things as then hiredis things the spaces are command delimiters.
>>
>> So, if I construct the command using a list template:
>>
>> template(name="redisPushQueue" type="list") {
>>    constant(value="LPUSH")
>>    constant(value="queue_key")
>>    property(name="msg")
>> }
>>
>> So that from the plugin side, I can get "LPUSH", "queue_key", and the
>> message from the msg property, and their lengths - then I could easily
>> use the ArgV version of the hiredis command runners and formatters.
>>
>> Make sense?
>>
>> Brian
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Jan 7, 2013 at 10:59 AM, Rainer Gerhards
>> <rgerhards at hq.adiscon.com>wrote:
>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Brian Knox
>>>> Sent: Monday, January 07, 2013 2:11 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] working with list templates
>>>>
>>>> Happy Monday - getting ready to revisit the improvements to the
>>>> template
>>>> handling I'd like to make to omhiredis.  To sum up the thread so
>> far:
>>>> 1) I can register from the plugin how I would like to handle the
>>>> template
>>> For an example, look here:
>>>
>>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommong
>> odb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504
>>> Note the OMSR_TPL_AS_... (JSON) in this case.
>>>
>>>> 2) I should probably use the JSON method as array will be
>> deprecated
>>> This is actually a good question. The original plan is to deprecate
>> them,
>>> HOWEVER, this is under review. Array mode *may* (may!) be a good
>>> alternative for more traditional (SQL) like output sources. However,
>> array
>>> mode lacks field names, so it all comes down to counting positions -
>> what
>>> in fact IMHO makes them unsuitable. My personal guess is that we will
>>> probably deprecate them. You'll see a related request for comment
>> here on
>>> this list in the not so distant future (I need to clearly describe
>> the need
>>> first).
>>>
>>>> So my question is now - if I use a "list" style template
>> declaration,
>>>> what
>>>> sort of JSON object will I end up with?
>>>>
>>>> For a concrete example:
>>>>
>>>> template(name="redisPushQueue" type="list") {
>>>>    constant(value="LPUSH")
>>>>    constant(value="queue_key")
>>>>    property(name="msg")
>>>> }
>>>>
>>>> And here's the function I'm passing to:
>>>> void redisAppendCommandArgv(redisContext *c, int argc, const char
>>>> **argv, const size_t *argvlen);
>>>>
>>>> So if I could pass this as essentially a JSON array ["LPUSH",
>>>> "queue_key",
>>>> "msg"] that would make life easy.
>>> Well, not sure about redis... don't you need field names? That's
>> actually
>>> one of the big questions in regard to relational databases. I think
>> they
>>> need field names in any case, so if they are not inside the template,
>> I
>>> need to pass them in via a different mechanism. Thus I want to have
>> them
>>> inside the template.
>>>
>>> With JSON passing what you guess is actually just one field, namely
>> "msg"
>>> with the given content. That is because the constants have no "name"
>>> parameter, and without a name, they cannot be part of a json object
>> (this
>>> is intentional, and can be useful to use the same template with
>> plugins
>>> that expect JSON and plugins that expect text).
>>>
>>> Rainer
>>>> Brian
>>>>
>>>>
>>>> On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <briank at talksum.com>
>> wrote:
>>>>> Aha!  Thanks for the info David - I wouldn't want to build on top
>> of
>>>> a
>>>>> feature that may be deprecated in the near future.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> On Fri, Jan 4, 2013 at 8:30 PM, David Lang <david at lang.hm> wrote:
>>>>>
>>>>>> It's probably better to use JSON instead of array. Array is a
>> very
>>>> early
>>>>>> thing and there is some talk of phasing it out rather than
>> needing
>>>> to
>>>>>> support multiple ways of passing groups of messages.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>>   On Fri, 4 Jan 2013, Brian Knox wrote:
>>>>>>
>>>>>>   Aha!  I think in this case array passing may be what I need.
>>>> Thanks!
>>>>>>> Brian
>>>>>>> On 1/4/2013 4:00 AM, Rainer Gerhards wrote:
>>>>>>>
>>>>>>>> Quick hint: you need to set the param passing mode inside the
>>>> plugin.
>>>>>>>> Not the template type specifies what you get, but the plugin
>> asks
>>>> the
>>>>>>>> engine. What you look for is JSON passing mode (or maybe array
>>>> passing).
>>>>>>>> Ommongodb should help you, else ping me early next week.
>>>>>>>>
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>>   -----Original Message-----
>>>>>>>>> From: rsyslog-bounces at lists.adiscon.**com<rsyslog-
>>>> bounces at lists.adiscon.com>[mailto:
>>>>>>>>> rsyslog-
>>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Brian Knox
>>>>>>>>> Sent: Thursday, January 03, 2013 2:22 PM
>>>>>>>>> To: rsyslog at lists.adiscon.com
>>>>>>>>> Subject: Re: [rsyslog] working with list templates
>>>>>>>>>
>>>>>>>>> Hi Radu!  Thanks for the link, but I already know how to work
>>>> with list
>>>>>>>>> templates from a conf perspective.  I'm looking for good
>> examples
>>>> of
>>>>>>>>> accessing that data from code in an output plugin.  Currently
>> my
>>>> plugin
>>>>>>>>> simply does this in CODESTARTdoAction:
>>>>>>>>>
>>>>>>>>> CHKiRet(writeHiredis(ppString[**0], pData));
>>>>>>>>>
>>>>>>>>> This passes the entire output from the template as a string
>> to
>>>> redis.
>>>>>>>>> This works ok when I want to construct redis commands from
>>>> templates
>>>>>>>>> using individual properties straight to hiredis's
>>>> redisAppendCommand
>>>>>>>>> function, such as:
>>>>>>>>>
>>>>>>>>> "HINCR progcount %programname% 1"
>>>>>>>>>
>>>>>>>>> But it does not work when I want to send whole messages to
>> redis
>>>>>>>>> (whitespace in the syslog message be interpreted as
>> delimiters in
>>>> the
>>>>>>>>> command, causing errors).
>>>>>>>>>
>>>>>>>>> hiredis has another function however, redisAppendCommandArgv,
>>>> that is
>>>>>>>>> variatic.  In order to use this I would like to construct a
>> list
>>>> using
>>>>>>>>> the list templates, and then access each individual member of
>> the
>>>> list
>>>>>>>>> from the output plugin code so I can construct the array to
>> pass
>>>> to
>>>>>>>>> redisAppendCommandArgv.  So, let's say I construct the
>> following
>>>>>>>>> template:
>>>>>>>>>
>>>>>>>>> template(name="redisPushQueue" type="list") {
>>>>>>>>>        constant(value="LPUSH")
>>>>>>>>>        constant(value="queue_key")
>>>>>>>>>        property(name="msg")
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>    From the output plugin code, how do I then access each
>> member
>>>> of this
>>>>>>>>> list individually with the new template code?
>>>>>>>>>
>>>>>>>>> Make sense?
>>>>>>>>>
>>>>>>>>> Brian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1) Can
>>>>>>>>> On 1/3/2013 7:59 AM, Radu Gheorghe wrote:
>>>>>>>>>
>>>>>>>>>> Hello Brian,
>>>>>>>>>>
>>>>>>>>>> Just to make sure we're not missing the obvious:
>>>>>>>>>>
>> http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy
>>>> slog.com/doc/rsyslog_conf_templates.html>
>>>>>>>>>> Besides that, there are a couple of examples here:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/1/3 Brian Knox <taotetek at gmail.com>
>>>>>>>>>>
>>>>>>>>>>   I've been working on some improvements for the omhiredis
>> output
>>>>>>>>>> plugin this
>>>>>>>>>> week.  I think I've come up with a better way of combining
>>>> templates
>>>>>>>>>> with
>>>>>>>>>> redis command formatting.  In order to do this, I need to,
>> given
>>>> a
>>>>>>>>>> list
>>>>>>>>>> style template, get the number of elements in the list and
>> the
>>>>>>>>>> length of
>>>>>>>>>> each element.
>>>>>>>>>>> I've only worked with string templates (from a plugin
>>>> perspective)
>>>>>>>>>> with my
>>>>>>>>>> output plugins so far.  Is there a good example somewhere
>> for
>>>>>>>>>> dealing with
>>>>>>>>>> list templates?
>>>>>>>>>>> On other fronts, I now omhiredis working with batch dequeue
>> in
>>>>>>>>>> conjunction
>>>>>>>>>> with hiredis' pipeline commands.  In local tests this let me
>>>> push
>>>>>>>>>> around
>>>>>>>>>> 250,000 redis commands per second to a redis instance (I was
>>>> getting
>>>>>>>>>> around
>>>>>>>>>> 50k commands of the same type with the same test before the
>>>>>>>>>> changes).
>>>>>>>>>> Rainer - I'm hoping to wrap up this next round of
>> improvements
>>>> in a
>>>>>>>>>> few
>>>>>>>>>> days and get a patch your way!
>>>>>>>>>>> Brian
>>>>>>>>>>> ______________________________**_________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>>>>>>> What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED by
>>>> a
>>>>>>>>>> myriad
>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>>>> if
>>>>>>>>>> you
>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>   ______________________________**_________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>>>> a
>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT
>>>> POST
>>>>>>>>> if you DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>>> ______________________________**_________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a
>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT
>>>> POST
>>>>>>>>> if you DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>> ______________________________**_________________
>>>>>>>> rsyslog mailing list
>>>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a
>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT
>>>> POST if
>>>>>>>> you DON'T LIKE THAT.
>>>>>>>>
>>>>>>> ______________________________**_________________
>>>>>>> rsyslog mailing list
>>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a
>>>> myriad
>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>>>> you
>>>>>>> DON'T LIKE THAT.
>>>>>>>
>>>>>>>   ______________________________**_________________
>>>>>> rsyslog mailing list
>>>>>>
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
>>>> n.net/mailman/listinfo/rsyslog>
>>>>>> http://www.rsyslog.com/**professional-
>>>> services/<http://www.rsyslog.com/professional-services/>
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a
>>>> myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>>>> you
>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>>>> if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>>> DON'T LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.



More information about the rsyslog mailing list