[rsyslog] working with list templates

Brian Knox briank at talksum.com
Mon Jan 7 17:18:22 CET 2013


So - in my case I do not believe I need field names.  Redis is more of a
k/v store than a document store like mongo (although it supports more data
structures than simply key / value, like sets, hashes, and lists).

With the hiredis api there are basically two ways of passing a command to
redis.  For the sake of example, let's take the command I played in the
previous example, with would push the content of the msg property into a
list in redis.

Assuming "msg" contains the rsyslog message itself ("my message")

1) Using sprintf like formatting

redisCommand(context, "LPUSH mylist %s", msg);

2) Passing an array of char arrays

something like this:

const char *argv[3];
argv[0] = "LPUSH";
argv[1] = "mylist";
argv[2] = msg;
size_t lens[3] = { 5, 6, 10 };
int argc = 3;

redisFormatCommandArgv(&cmd,argc,argv,lens);

The way my current omhiredis module works is to simply call

rc = redisAppendCommand(pData->conn, (char*)message);

This works fine for doing things like incrementing counters

template(name="messages_processed" type="string" string="INCR processed 1")

However, if I include the actual log line, the spaces in the log line break
things as then hiredis things the spaces are command delimiters.

So, if I construct the command using a list template:

template(name="redisPushQueue" type="list") {
  constant(value="LPUSH")
  constant(value="queue_key")
  property(name="msg")
}

So that from the plugin side, I can get "LPUSH", "queue_key", and the
message from the msg property, and their lengths - then I could easily
use the ArgV version of the hiredis command runners and formatters.

Make sense?

Brian







On Mon, Jan 7, 2013 at 10:59 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Brian Knox
> > Sent: Monday, January 07, 2013 2:11 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] working with list templates
> >
> > Happy Monday - getting ready to revisit the improvements to the
> > template
> > handling I'd like to make to omhiredis.  To sum up the thread so far:
> >
> > 1) I can register from the plugin how I would like to handle the
> > template
>
> For an example, look here:
>
> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommongodb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504
>
> Note the OMSR_TPL_AS_... (JSON) in this case.
>
> > 2) I should probably use the JSON method as array will be deprecated
>
> This is actually a good question. The original plan is to deprecate them,
> HOWEVER, this is under review. Array mode *may* (may!) be a good
> alternative for more traditional (SQL) like output sources. However, array
> mode lacks field names, so it all comes down to counting positions - what
> in fact IMHO makes them unsuitable. My personal guess is that we will
> probably deprecate them. You'll see a related request for comment here on
> this list in the not so distant future (I need to clearly describe the need
> first).
>
> >
> > So my question is now - if I use a "list" style template declaration,
> > what
> > sort of JSON object will I end up with?
> >
> > For a concrete example:
> >
> > template(name="redisPushQueue" type="list") {
> >   constant(value="LPUSH")
> >   constant(value="queue_key")
> >   property(name="msg")
> > }
> >
> > And here's the function I'm passing to:
> > void redisAppendCommandArgv(redisContext *c, int argc, const char
> > **argv, const size_t *argvlen);
> >
> > So if I could pass this as essentially a JSON array ["LPUSH",
> > "queue_key",
> > "msg"] that would make life easy.
>
> Well, not sure about redis... don't you need field names? That's actually
> one of the big questions in regard to relational databases. I think they
> need field names in any case, so if they are not inside the template, I
> need to pass them in via a different mechanism. Thus I want to have them
> inside the template.
>
> With JSON passing what you guess is actually just one field, namely "msg"
> with the given content. That is because the constants have no "name"
> parameter, and without a name, they cannot be part of a json object (this
> is intentional, and can be useful to use the same template with plugins
> that expect JSON and plugins that expect text).
>
> Rainer
> >
> > Brian
> >
> >
> > On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <briank at talksum.com> wrote:
> >
> > > Aha!  Thanks for the info David - I wouldn't want to build on top of
> > a
> > > feature that may be deprecated in the near future.
> > >
> > > Brian
> > >
> > >
> > > On Fri, Jan 4, 2013 at 8:30 PM, David Lang <david at lang.hm> wrote:
> > >
> > >> It's probably better to use JSON instead of array. Array is a very
> > early
> > >> thing and there is some talk of phasing it out rather than needing
> > to
> > >> support multiple ways of passing groups of messages.
> > >>
> > >> David Lang
> > >>
> > >>
> > >>  On Fri, 4 Jan 2013, Brian Knox wrote:
> > >>
> > >>  Aha!  I think in this case array passing may be what I need.
> > Thanks!
> > >>>
> > >>> Brian
> > >>> On 1/4/2013 4:00 AM, Rainer Gerhards wrote:
> > >>>
> > >>>> Quick hint: you need to set the param passing mode inside the
> > plugin.
> > >>>> Not the template type specifies what you get, but the plugin asks
> > the
> > >>>> engine. What you look for is JSON passing mode (or maybe array
> > passing).
> > >>>> Ommongodb should help you, else ping me early next week.
> > >>>>
> > >>>> Rainer
> > >>>>
> > >>>>  -----Original Message-----
> > >>>>> From: rsyslog-bounces at lists.adiscon.**com<rsyslog-
> > bounces at lists.adiscon.com>[mailto:
> > >>>>> rsyslog-
> > >>>>> bounces at lists.adiscon.com] On Behalf Of Brian Knox
> > >>>>> Sent: Thursday, January 03, 2013 2:22 PM
> > >>>>> To: rsyslog at lists.adiscon.com
> > >>>>> Subject: Re: [rsyslog] working with list templates
> > >>>>>
> > >>>>> Hi Radu!  Thanks for the link, but I already know how to work
> > with list
> > >>>>> templates from a conf perspective.  I'm looking for good examples
> > of
> > >>>>> accessing that data from code in an output plugin.  Currently my
> > plugin
> > >>>>> simply does this in CODESTARTdoAction:
> > >>>>>
> > >>>>> CHKiRet(writeHiredis(ppString[**0], pData));
> > >>>>>
> > >>>>> This passes the entire output from the template as a string to
> > redis.
> > >>>>> This works ok when I want to construct redis commands from
> > templates
> > >>>>> using individual properties straight to hiredis's
> > redisAppendCommand
> > >>>>> function, such as:
> > >>>>>
> > >>>>> "HINCR progcount %programname% 1"
> > >>>>>
> > >>>>> But it does not work when I want to send whole messages to redis
> > >>>>> (whitespace in the syslog message be interpreted as delimiters in
> > the
> > >>>>> command, causing errors).
> > >>>>>
> > >>>>> hiredis has another function however, redisAppendCommandArgv,
> > that is
> > >>>>> variatic.  In order to use this I would like to construct a list
> > using
> > >>>>> the list templates, and then access each individual member of the
> > list
> > >>>>> from the output plugin code so I can construct the array to pass
> > to
> > >>>>> redisAppendCommandArgv.  So, let's say I construct the following
> > >>>>> template:
> > >>>>>
> > >>>>> template(name="redisPushQueue" type="list") {
> > >>>>>       constant(value="LPUSH")
> > >>>>>       constant(value="queue_key")
> > >>>>>       property(name="msg")
> > >>>>> }
> > >>>>>
> > >>>>>   From the output plugin code, how do I then access each member
> > of this
> > >>>>> list individually with the new template code?
> > >>>>>
> > >>>>> Make sense?
> > >>>>>
> > >>>>> Brian
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> 1) Can
> > >>>>> On 1/3/2013 7:59 AM, Radu Gheorghe wrote:
> > >>>>>
> > >>>>>> Hello Brian,
> > >>>>>>
> > >>>>>> Just to make sure we're not missing the obvious:
> > >>>>>>
> > http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy
> > slog.com/doc/rsyslog_conf_templates.html>
> > >>>>>>
> > >>>>>> Besides that, there are a couple of examples here:
> > >>>>>>
> > >>>>>>
> > >>>>>> 2013/1/3 Brian Knox <taotetek at gmail.com>
> > >>>>>>
> > >>>>>>  I've been working on some improvements for the omhiredis output
> > >>>>>>>
> > >>>>>> plugin this
> > >>>>>
> > >>>>>> week.  I think I've come up with a better way of combining
> > templates
> > >>>>>>>
> > >>>>>> with
> > >>>>>
> > >>>>>> redis command formatting.  In order to do this, I need to, given
> > a
> > >>>>>>>
> > >>>>>> list
> > >>>>>
> > >>>>>> style template, get the number of elements in the list and the
> > >>>>>>>
> > >>>>>> length of
> > >>>>>
> > >>>>>> each element.
> > >>>>>>>
> > >>>>>>> I've only worked with string templates (from a plugin
> > perspective)
> > >>>>>>>
> > >>>>>> with my
> > >>>>>
> > >>>>>> output plugins so far.  Is there a good example somewhere for
> > >>>>>>>
> > >>>>>> dealing with
> > >>>>>
> > >>>>>> list templates?
> > >>>>>>>
> > >>>>>>> On other fronts, I now omhiredis working with batch dequeue in
> > >>>>>>>
> > >>>>>> conjunction
> > >>>>>
> > >>>>>> with hiredis' pipeline commands.  In local tests this let me
> > push
> > >>>>>>>
> > >>>>>> around
> > >>>>>
> > >>>>>> 250,000 redis commands per second to a redis instance (I was
> > getting
> > >>>>>>>
> > >>>>>> around
> > >>>>>
> > >>>>>> 50k commands of the same type with the same test before the
> > >>>>>>>
> > >>>>>> changes).
> > >>>>>
> > >>>>>> Rainer - I'm hoping to wrap up this next round of improvements
> > in a
> > >>>>>>>
> > >>>>>> few
> > >>>>>
> > >>>>>> days and get a patch your way!
> > >>>>>>>
> > >>>>>>> Brian
> > >>>>>>> ______________________________**_________________
> > >>>>>>> rsyslog mailing list
> > >>>>>>>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >>>>>>> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> > a
> > >>>>>>>
> > >>>>>> myriad
> > >>>>>
> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if
> > >>>>>>>
> > >>>>>> you
> > >>>>>
> > >>>>>> DON'T LIKE THAT.
> > >>>>>>>
> > >>>>>>>  ______________________________**_________________
> > >>>>>> rsyslog mailing list
> > >>>>>>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >>>>>> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> > a
> > >>>>>>
> > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>>>> if you DON'T LIKE THAT.
> > >>>>>
> > >>>>> ______________________________**_________________
> > >>>>> rsyslog mailing list
> > >>>>>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >>>>> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>>>> if you DON'T LIKE THAT.
> > >>>>>
> > >>>> ______________________________**_________________
> > >>>> rsyslog mailing list
> > >>>>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >>>> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST if
> > >>>> you DON'T LIKE THAT.
> > >>>>
> > >>>
> > >>> ______________________________**_________________
> > >>> rsyslog mailing list
> > >>>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >>> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > >>> DON'T LIKE THAT.
> > >>>
> > >>>  ______________________________**_________________
> > >> rsyslog mailing list
> > >>
> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> > n.net/mailman/listinfo/rsyslog>
> > >> http://www.rsyslog.com/**professional-
> > services/<http://www.rsyslog.com/professional-services/>
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > >> DON'T LIKE THAT.
> > >>
> > >
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>


More information about the rsyslog mailing list