[rsyslog-notify] Forum Thread: mmfields broken - (Mode 'post')

[noreply at adiscon.com]

User: goodleaf 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=24246#p24246

Message: 
----------
Hi,

I am using Rsyslog with mmfields to receive CEF formated events from
Arcsight over syslog udp and tcp. In addition to a local configuration I
have the following rsyslog configuration to receive remote CEF formated
messages:

module(load="imtcp")
module(load="imudp")
module(load="mmfields")
template(name="cef" type="string" string="%$!%\n")
ruleset(name="remote") {
action(type="mmfields")
action(type="omfile" file="/var/log/remote.log")
action(type="omfile" file="/var/log/remote_ parsed.log" template="cef")
}
input(type="imtcp" port="514" ruleset="remote")
input(type="imudp" port="514" ruleset="remote")


Receiving the events and logging them to remote.log is no problem. Parsing
them, however, is a completely different story:

Output of remote.log (unparsed CEF), properly received and stored as
expected:
2014-02-18T11:44:37.292916+01:00 somehost.somedomain.com CEF:
0|ArcSight|Logger|5.3.1.6838.0|eps:103|Individual Forwarder EPS|1|
cat=/Monitor/Forwarder/EPS/Individual cn1=0 cn1Label=EPS cs1=UDP
cs1Label=Forwarder Type cs2=SinceLastMonitorEvent cs2Label=timeframe
cs6=log_udp cs6Label=Forwarder name dst=10.0.0.100 dvc=10.0.0.10
end=1392716400079 rt=1392716400079


Output of remote_parsed.log (parsed CEF), obviously parsed incorrectly:
{ "f1": "0|ArcSight|Logger|5.3.1.6838.0|eps:103|Individual Forwarder EPS|1|
cat=\/Monitor\/Forwarder\/EPS\/Individual cn1=0 cn1Label=EPS cs1=UDP
cs1Label=Forwarder Type cs2=SinceLastMonitorEvent cs2Label=timeframe
cs6=log_udp cs6Label=Forwarder name dst=10.0.0.100 dvc=10.0.0.10
end=1392716400079 rt=1392716400079" }


For sake of completeness, I have included the output of rsyslogd -N1 below:
rsyslogd: version 8.1.5, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.


I have also tried the rsyslog 7 dev branch with identical results. Have
anyone successfully made mmfields properly parse CEF formated logs?


Just a quick note, the original data sent from ArcSight is (according to
netcat):

CEF:0|ArcSight|Logger|5.3.1.6838.0|eps:103|Individual Forwarder EPS|1|
cat=/Monitor/Forwarder/EPS/Individual cn1=0 cn1Label=EPS cs1=UDP
cs1Label=Forwarder Type cs2=SinceLastMonitorEvent cs2Label=timeframe
cs6=log_udp cs6Label=Forwarder name dst=10.0.0.100 dvc=10.0.0.10
end=1392716400079 rt=1392716400079


i.e. no syslog header, just the message.


Also; If you can provide me with a guide on writing message modification
plugins from scratch, I already have a working CEF parser in C I could
implement myself (something I wrote before I I realized I perhaps could use
rsyslog for the purpose). I tried looking at existing plugins, but I could
not quite figure it out on my own.