[rsyslog] Any good guides for generating JSON formatted log messages on rsyslog client nodes for processing on rsyslog receiver node?

David Lang david at lang.hm
Mon Sep 18 19:02:27 CEST 2017


to send in JSON you need to create a custom template

I use something like (typed from memory, may be errors)
$template structured,"<%pri%>%timestamp% %hostname% %syslogtag% %$!%\n"

getting things into different variables so that the result looks reasonable 
takes a little more effort.

If what you are getting is json in the msg field to start with, you can use 
mmjsonparse (but you may need to set the cee cookie to "" to parse things 
correctly)

If you are getting anything else, then you need to use mmnormalize to parse the 
message.

on the sending system, log locally using RSYSLOG_DebugFormat and you will be 
able to see what is in $! (and what else is known about the message)

David Lang

  On Mon, 18 Sep 2017, deoren wrote:

> Date: Mon, 18 Sep 2017 10:52:32 -0500
> From: deoren <rsyslog-users-lists.adiscon.net at whyaskwhy.org>
> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> To: rsyslog-users <rsyslog at lists.adiscon.com>
> Subject: [rsyslog] Any good guides for generating JSON formatted log messages
>     on rsyslog client nodes for processing on rsyslog receiver node?
> 
> Most of what I'm coming across is geared towards sending into another product 
> like mongodb or elasticsearch.
>
> I'm really new to this aspect, so the more newbie friendly the better. In 
> particular, I'd like to find a barebones template for replicating forwarding 
> of content using RSYSLOG_ForwardFormat or RSYSLOG_SyslogProtocol23Format via 
> RELP (which I'm already doing with good results).
>
> Some points that I've gotten hung up on:
>
> * Do I use mmjsonparse before or after forwarding the message (or both?). I 
> assumed after receiving the message was when I needed to use the 
> 'action(type="mmjsonparse" cookie="")' entry, but I've tried it both ways.
>
> * Do I use a custom template for forwarding, or craft the message into JSON 
> format and then forward using an existing template (this doesn't sound like 
> the right approach)?
>
> * In my testing, what I expected to see as separate JSON keys are embedded in 
> the 'msg' value when saved to a flat-file on the receiver. Does this sound 
> like a common mistake?
>
> Some of the resources I've looked at thus far:
>
> * http://www.rsyslog.com/doc/v8-stable/configuration/templates.html
>
> * 
> https://www.digitalocean.com/community/tutorials/how-to-centralize-logs-with-rsyslog-logstash-and-elasticsearch-on-ubuntu-14-04
>
> * 
> http://www.rsyslog.com/using-rsyslog-and-elasticsearch-to-handle-different-types-of-json-logs/
>
> * https://techpunch.co.uk/development/how-to-shop-json-logs-via-rsyslog
>
> * 
> https://sematext.com/blog/2013/05/28/structured-logging-with-rsyslog-and-elasticsearch/
>
> My goal is to (at some future date) get all rsyslog clients configured to 
> send exclusively with JSON and then perform all conversions on the receiver 
> node (RSYSLOG_FileFormat for local flat-file storage, GELF for Graylog, etc).
>
> As I've seen mentioned elsewhere, I'm hoping to use JSON format to include 
> additional metadata with log messages.
>
> Thank you in advance for your help.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>


More information about the rsyslog mailing list