[rsyslog] rsyslog performance as receiver, heavily using regex in templates

Rainer Gerhards rgerhards at hq.adiscon.com
Thu Jan 31 17:58:36 CET 2013


> On a slightly related note. With regular expressions is there a way to
> extract the sub-matches into separate positions within the template?
> 
> For example, here's my test template to use with omelasticsearch
> (broken onto new lines for readability)...
> $template ApacheAccessElasticSearch,"{
> \"msg\":\"%msg:::json%\",
> \"sysloghost\":\"%HOSTNAME:::json%\",
> \"syslogip\":\"%fromhost-ip%\",
> \"syslogfacility\":\"%syslogfacility-text%\",
> \"syslogpri\":\"%pri%\",
> \"syslogseverity\":\"%syslogseverity-text%\",
> \"program\":\"%programname%\",
> \"syslogtime\":\"%timereported:1:19:date-
> rfc3339%.%timereported:1:3:date-subseconds%\",
> \"syslogtag\":\"%syslogtag:::json%\",
> \"http.usec\":\"%msg:R,ERE,1,BLANK:([0-9]+)$--end%\"
> \"http.vhost\":\"%msg:R,ERE,1,BLANK:([a-z0-9\-\.]+) [0-9]+$--end%\"
> }"
> 
> 
> If you look at the http.usec and http.vhost fields in the template, is
> there a way I can have a single regex with submatch 2 going in to
> http.usec and submatch 1 going in to http.vhost?
> And submatch 3,4,5,6 etc going into their own fields in the JSON output
> of the template?

Unfortunately, this is currently not possible. It is also surprisingly hard to add this quickly. The reason is that the property replacer works on a property-by-property basis, so it is unable to see any previous regex. Of course, that can be changed, but it's not trivial.

I also plan - but med-term - to add regex extraction functions to the script language. Then these can do that task.

Sorry I have no better answer.

Rainer


More information about the rsyslog mailing list