[rsyslog-notify] Forum Thread: mmpstrucdata,omelastic module memory leak? - (Mode 'post')

[noreply at adiscon.com]

User: cybermedi 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=25007#p25007

Message: 
----------
Hi,
I use rsyslog with structured data parser and sender to omelastic. The idea
convert structured data to json format acceptable by elastic search.
The problem is that rsyslog process consume all available memory. Can
anybody help me how with troubleshooting? I use version on debian wheeze:
Maybe there is another way how to convert structured data to json elastic
with less memory footprint but I think even it is not very elegant
configuration rsyslog should not consume all available memory or Am I
wrong?

[code:ypd8e3s2]rsyslogd 8.4.2, compiled with:
	FEATURE_REGEXP:				Yes
	GSSAPI Kerberos 5 support:		No
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	memory allocator:			system default
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes
	Number of Bits in RainerScript integers: 32 (due to too-old json-c
lib)

See http://www.rsyslog.com for more
information.[/code:ypd8e3s2]


I have created such configuration:

[code:ypd8e3s2]module(load="mmpstrucdata”)	#for structured data parsing
module(load="omelasticsearch") 	# for outputting to Elasticsearch
action(type="mmpstrucdata")

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
  type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}

#this template is for log line containing structured data
template(name="structured" type="list")
{
	constant(value="{")
    constant(value="\"@timestamp\":\"")    
property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")        property(name="hostname")
    constant(value="\",\"severity\":\"")   
property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")   
property(name="syslogfacility-text")
    constant(value="\",\"tag\":\"")   property(name="syslogtag"
format="json")
    constant(value="\",\"message\":\"")    property(name="msg"
format="json")
	constant(value="\",")
	property(name="$!rfc5424-sd!mdc at 18060" position.From="3")
}

#this template is for line without structured data
template(name="pure" type="list")
{
    constant(value="{")
    constant(value="\"@timestamp\":\"")    
property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")        property(name="hostname")
    constant(value="\",\"severity\":\"")   
property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")   
property(name="syslogfacility-text")
    constant(value="\",\"tag\":\"")   property(name="syslogtag"
format="json")
    constant(value="\",\"message\":\"")    property(name="msg"
format="json")
    constant(value="\"}")
}

#make decision what template use
if $!rfc5424-sd == "" then {

*.* action(type="omelasticsearch"
           server="192.168.62.109"
           serverport="9200"
			template="pure"
			searchIndex="logstash-index"
			dynSearchIndex="on"
#           searchType="kibana"
           bulkmode="off"
           queue.type="linkedlist"
           queue.size="100"
           queue.dequeuebatchsize="300"
           action.resumeretrycount="-1")
} else {

*.* action(type="omelasticsearch"
           server="192.168.62.109"
           serverport="9200"
			template="structured"
			searchIndex="logstash-index"
			dynSearchIndex="on"
#           searchType="kibana"
           bulkmode="off"
           queue.type="linkedlist"
           queue.size="100"
           queue.dequeuebatchsize="300"
           action.resumeretrycount="-1")
}[/code:ypd8e3s2]