[rsyslog-notify] Forum Thread: Re: Send single database to its own file - re_extract - (Mode 'edit_last_post')

[noreply at adiscon.com]

User: dlang 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26327#p26327

Message: 
----------
so my first suggestion is to stop using regex matching. It's slow, and as
you are finding, it's hard :-)

instead use mmnormalize

for your example line:



I would probably do a match against the entire thing, something like:

in rsyslog.conf

# create a log in a known format that doesn't care if it arrrived as
rfc3184 or rfc5424
set $.stdmsg = exec_template("RSYSLOG_TraditionalFileFormat");
# parse the message with the ruleset
action(type="mmnormalize" path="$.extracted" variable="$.stdmsg"
ruleBase="/etc/rsyslog.rulebase" name="mmnorm")

Then in the ruleset, I would start with something like:

version=2
#use the new rule engine with all it's nifty options
prefix=%timestamp:date-rfc3164% %hostname:word% postgres[%pid:char-to:]%]:
# It's a good practice to put a sample of the log as a comment in the file
for future reference. You may want to trim the prefix part off, I'm leaving
it in place for this example.
# Feb 5 10:25:08 odcdb postgres[16898]: [4-1] [16898-2] postgres at sessions
LOG: connection authorized: user=postgres database=sessions
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
%user:char-to:@%@%database:word% LOG: connection authorized:
user=%user2:word% database=%database2:word%
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
[unknown]@[unknown] LOG: connection authorized: authorized:
user=%user2:word% database=%database2:word%

this would create the json output {'user': 'postgres',
'database':'session', 'user2':'postgres', 'database2':'session'}

now, if you have a lot of rules with this pattern, you can create your own
'type'

up at the top
type=@connection:%user:char-to:@%@%database:word%
type=@connection:[unknown]@[unknown]

then the rule would change to be
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
%conn:@connection% LOG: connection auhorized: authorized: user=%user2:word%
database=%database2:word%

this would create the json output {'conn':{'user': 'postgres',
'database':'session'}, 'user2':'postgres', 'database2':'session'}
note that the connection definition is a sub-element, by doing
%.:@connection% instead you would get the same thing as the earlier
example.

a bit more straightforward to translate logs into rules, and it scales
really well (effectlively O(1) as it scales on the length of the message
rather with the number of rules having almost no effect, unlike regexes
that tend to go O(N) to O(N^2) on the number of rules

Also, I tagged this rule with the tag 'goodauth', tags are a comma
separated list, so you can have many per log message if appropriate. the
tag is then attached to the JSON of the message and you can make logic or
use in dynafile templates to sort the logs by topic without a million if
rules. it's really handy to get all goodauth logs in one file and all
badauth logs in a different one when you are looking for things going
wrong. And if you are consistant in your element names in the template, no
matter what the original log looked like, you will have the userid easily
referenced in the JSON for reporting and alerting