[rsyslog-notify] Forum Thread: Re: Send single database to its own file - re_extract - (Mode 'reply')

[noreply at adiscon.com]

User: dlang 
Forumlink: http://kb.monitorware.com/viewtopic.php?p=26327#p26327

Message: 
----------
so my first suggestion is to stop using regex matching. It's slow, and as
you are finding, it's hard :-)

instead use mmnormalize

for your example line:



I would probably do a match against the entire thing, something like:

in rsyslog.conf

# create a log in a known format that doesn't care if it arrrived as
rfc3184 or rfc5424
set $.stdmsg = exec_template("RSYSLOG_TraditionalFileFormat");
# parse the message with the ruleset
action(type="mmnormalize" path="$.extracted" variable="$.stdmsg"
ruleBase="/etc/rsyslog.rulebase" name="mmnorm")

Then in the ruleset, I would start with something like:

version=2
#use the new rule engine with all it's nifty options
prefix=%timestamp:date-rfc3164% %hostname:word% postgres[%pid:char-to:]%]:
# It's a good practice to put a sample of the log as a comment in the file
for future reference. You may want to trim the prefix part off, I'm leaving
it in place for this example.
# Feb 5 10:25:08 odcdb postgres[16898]: [4-1] [16898-2] postgres at sessions
LOG: connection authorized: user=postgres database=sessions
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
%user:char-to:@%@%database:word% LOG: connection authorized:
user=%user2:word% database=%database2:word%
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
[unknown]@[unknown] LOG: connection authorized: authorized:
user=%user2:word% database=%database2:word%

this would create the json output {'user': 'postgres',
'database':'session', 'user2':'postgres', 'database2':'session'}

now, if you have a lot of rules with this pattern, you can create your own
'type'

up at the top
type=@connection:%user:char-to:@%@%database:word%
type=@connection:[unknown]@[unknown]

then the rule would change to be
rule=goodauth: [%something:char-to:]%] [%else:char-to:]%]
%conn:@connection% LOG: connection auhorized: authorized: user=%user2:word%
database=%database2:word%

this would create the json output {'conn':{'user': 'postgres',
'database':'session'}, 'user2':'postgres', 'database2':'session'}
note that the connection definition is a sub-element, by doing
%.:@connection% instead you would get the same thing as the earlier
example.

a bit more straightforward to translate logs into rules, and it scales
really well (effectlively O(1) as it scales on the length of the message
rather with the number of rules having almost no effect, unlike regexes
that tend to go O(N) to O(N^2) on the number of rules