[rsyslog] Some list templates with mmjsonparse try to get 2GB of RAM or give OOM

Radu Gheorghe radu0gheorghe at gmail.com
Tue Jan 8 12:41:54 CET 2013


2013/1/7 Rainer Gerhards <rgerhards at hq.adiscon.com>

>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Radu Gheorghe
> > Sent: Monday, January 07, 2013 3:47 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Some list templates with mmjsonparse try to get
> > 2GB of RAM or give OOM
> >
> > Wow, that was fast! Thanks, Rainer!
> >
> > Regarding the related questions:
> > - please let me know if there's any issue with using $!all-json. I
> > would
> > need to use it to handle with CEE-enhanced and "regular" logs with the
> > same
> > template - and output a JSON. Something like:
> >
> > template(name="testTemplate"
> >          type="list") {
> >            constant(value="{")
> >            constant(value="\"@timestamp\":\"")
> >            property(name="timereported" dateFormat="rfc3339")
> >            constant(value="\",")
> >            property(name="$!all-json" position.from="2")
> >          }
> >
> > I can work around that with 2 templates and %parsesuccess%, but I find
> > the
> > solution above much nicer.
>
> No, I think $!all-json will stay, at least for a long while. During my
> testing, though, I saw something that looked strange. Had not time to
> investigate, yet. So be careful. The JSON may be sometimes incorrect (I'll
> verify later this week).
>

With you saying I should be careful, I thought it's a good moment for a
little stress test. I didn't need much to get into trouble:

tail -1000 /var/log/messages | logger

And for most of the messages, the $!all-json thing becomes an empty string.
Which leads to an invalid JSON. If I replace that with %msg% I don't lose
any log out of 100K (which is as far as I went for now).

I'm looking now at the alternative of using %parsesuccess%. But it's a bit
ugly, because:
- each action needs its own queue. That means I'd have to change some
settings on the main message queue as well if I want to make good use of my
RAM. Otherwise, if I put big queues for each action and they're used
unevenly, some RAM is left unused and the busiest queue fills up faster
- I need 3 conditions that I didn't use before, plus 3 templates instead of
one:
  - if %parsesuccess% is false, I'd have to use a template for non-CEE
syslog
  - else, if the CEE cookie has a space, trim the cookie and insert the
JSON message in the big JSON that I want to log
    - else, trim the non-space CEE cookie (pun intended :D) with another
template and insert the JSON


> >
> > As for contributions to the documentation: how can I actually do that?
> > Let's say I would want to contribute some changes to this page:
> > http://www.rsyslog.com/doc/property_replacer.html
> >
> > I mean, I only know how to do pull requests on github. In this case I
> > should simply download the html, change it and send the back the diff?
>
> That would be fine with me. If you know how to create your own branch on
> github and commit to it, that also works for me. Just as you like :-)
>
>
OK, thanks. I will send diffs when I can find time to contribute
documentation.

I have no idea how I can clone the site to a github repo and contribute to
it. I didn't see it on github or on the Adiscon git.

Best regards,
Radu


More information about the rsyslog mailing list