From Christopher.Racky at web.de Wed Jul 1 10:42:39 2015 From: Christopher.Racky at web.de (Christopher.Racky at web.de) Date: Wed, 1 Jul 2015 10:42:39 +0200 Subject: [Lognorm] normalization of vmware date format References: Message-ID: Hello together, I wonder, that I did not get any response after one month. Is my question so bad, is there no solution available yet or is this mailing-list dead? Best regards Chris > Gesendet: Montag, 01. Juni 2015 um 21:21 Uhr > Von: Christopher.Racky at web.de > An: lognorm at lists.adiscon.com > Betreff: normalization of vmware date format > > Hi list, > > VMWare ESXi Systems have raw log messages like this: > <35>2015-05-18T07:56:48Z hostname.fqdn.de DCUI: Authentication of user root failed > The date-format does not match to the "standard" Format of any RFC (as far as I see). > > How can I add this date into _one_ field via log-normalization? > A rule like this > rule=:<%prio:number%>%time:char-to:Z%Z %host:word% ... > is to generic, but this "variation" of RFC5424 is not supported by this kind of rule: > rule=:<%prio:number%>%time:date-rfc5424 %host:word% ... > as you can see here: > [root at bug log]# echo "<35>2015-05-18T07:56:48Z host DCUI: Authentication of user root failed" | lognormalizer -r /etc/csiem.rb > [cee at 115 originalmsg="<35>2015-05-18T07:56:48Z host DCUI: Authentication of user root failed" unparsed-data="56:48Z host DCUI: Authentication of user root failed"] > > Because I later want to insert this date via rsyslog ommysql I guess there is also missing converting support. Right? > > > > By the way: > When will the next version of liblognorm be released? Beginning of May seems to over ;) > There are a couple of important bugfixes of the new version... > > best regards > Chris From friedl at adiscon.com Mon Jul 20 17:52:39 2015 From: friedl at adiscon.com (Florian Riedl) Date: Mon, 20 Jul 2015 17:52:39 +0200 Subject: [Lognorm] liblognorm 1.1.2 released Message-ID: Hi all, we have just released liblognorm 1.1.2. This release contains new parser syntaxes and some fixes. Changes: Version 1.1.2, 2015-07-20 - permit newline inside parser definition - new parser ?cisco-interface-spec? - new parser ?json? to process json parts of the message - new parser ?mac48? to process mac layer addresses - new parser ?name-value-list? (currently inofficial, experimental) - some parsers did incorrectly report success when an error occurred this was caused by inconsistencies between various macros. We have changed the parser-generation macros to match the semantics of the broader CHKN/CHKR macros and also restructured/simplified the parser generation macros. closes https://github.com/rsyslog/liblognorm/issues/41 - call ?rest? parser only if nothing else matches. Versions prior to 1.1.2 did execute ?rest? during regular parser processing, and thus parser matches have been more or less random. With 1.1.2 this is now always the last parser called. This may cause problems with existing rulesets, HOWEVER, adding any other rule or changing the load order would also have caused problems, so there really is no compatibility to preserve. see also: http://blog.gerhards.net/2015/04/liblognorms-rest-parser-now-more-useful.html - new API to support error callbacks This permits callers to forward messages in regard to e.g. wrong rule bases to their users, which is very useful and actually missing in the previous code base. So far, we only have few error messages. However, we will review the code and add more. The important part is that callers can begin to use the new API and thus will benefit when we add more error messages. - testbench is now enabled by default - bugfix: misadressing on some constant values see also https://github.com/rsyslog/liblognorm/pull/67 Thanks to github user ontholerian for the patch - bugfix: add missing function prototypes This could potentially lead to problems on some platforms, especially those with 64 bit pointers. Download: http://www.liblognorm.com/download/liblognorm-1-1-2/ As always, feedback is appreciated. Best regards, Florian Riedl -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgerhards at hq.adiscon.com Tue Jul 21 09:09:56 2015 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 21 Jul 2015 09:09:56 +0200 Subject: [Lognorm] [rsyslog] liblognorm 1.1.2 released In-Reply-To: <038FA10A-4C8D-486D-951D-32A49E900D47@opencsw.org> References: <038FA10A-4C8D-486D-951D-32A49E900D47@opencsw.org> Message-ID: 2015-07-21 9:08 GMT+02:00 Dagobert Michelsen : > Hi Florian, > > Am 20.07.2015 um 17:52 schrieb Florian Riedl : >> we have just released liblognorm 1.1.2. This release contains new parser >> syntaxes and some fixes. > > Is there a specific reason why this has not been fixed? > http://bugzilla.adiscon.com/show_bug.cgi?id=539 No, just general lack of time. It's on the list and we hope to get this done later. Rainer > > > Best regards > > ? Dago > > -- > "You don't become great by trying to be great, you become great by wanting to do something, > and then doing it so hard that you become great in the process." - xkcd #896 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. From Christopher.Racky at web.de Wed Jul 22 09:41:37 2015 From: Christopher.Racky at web.de (Christopher.Racky at web.de) Date: Wed, 22 Jul 2015 09:41:37 +0200 Subject: [Lognorm] [rsyslog] liblognorm 1.1.2 released - rest parser Message-ID: Hello, I'm going to become mad with the rest parser, even in the latest liblognorm release. I just want to normalize this very basic message using the "rest" field-type: echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/name.rb This is my ruleset: --------------- prefix=<%prio:number%>%time:date-rfc3164% %host:word% rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest% rule=:sshd%-:rest% --------------- I use this alternatively: --------------- rule=:<%prio:number%>%time:date-rfc3164% %host:word% sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word% %-:rest% --------------- None of them work. Result is always: [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] I installed the latest release, see here: [root at server ~]# yum info liblognorm1.x86_64 Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile * base: ftp.wrz.de * extras: centos.bio.lmu.de * updates: centos.bio.lmu.de Installed Packages Name : liblognorm1 Arch : x86_64 Version : 1.1.2 Release : 1.el6 Size : 103 k Repo : installed >From repo : rsyslog-v8-stable Summary : Fast samples-based log normalization library URL : http://www.liblognorm.com License : LGPLv2+ Description : Briefly described, liblognorm is a tool to normalize log data. : : People who need to take a look at logs often have a common problem. Logs from : different machines (from different vendors) usually have different formats for : their logs. Even if it is the same type of log (e.g. from firewalls), the log : entries are so different, that it is pretty hard to read these. This is where : liblognorm comes into the game. With this tool you can normalize all your logs. : All you need is liblognorm and its dependencies and a sample database that fits : the logs you want to normalize. Debug-Output: [root at bug ~]# echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/test3.rb -v liblognorm: read sample line: 'prefix=<%prio:number%>%time:date-rfc3164% %host:word% ' liblognorm: read sample line: 'rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' liblognorm: sample line to add: ':sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' liblognorm: addSampToTree 0 of 167 liblognorm: parsed literal: '<' liblognorm: buildPTree: begin at 0x8b0060, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 1 of 167 liblognorm: parsed field: 'prio' liblognorm: got new subtree 0x8b0df0 liblognorm: prev subtree 0x8b0060 liblognorm: new subtree 0x8b0df0 liblognorm: addSampToTree 14 of 167 liblognorm: parsed literal: '>' liblognorm: buildPTree: begin at 0x8b0df0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 15 of 167 liblognorm: parsed field: 'time' liblognorm: got new subtree 0x8b16a0 liblognorm: prev subtree 0x8b0df0 liblognorm: new subtree 0x8b16a0 liblognorm: addSampToTree 34 of 167 liblognorm: parsed literal: ' ' liblognorm: buildPTree: begin at 0x8b16a0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 1, offs 0 liblognorm: addSampToTree 35 of 167 liblognorm: parsed field: 'host' liblognorm: got new subtree 0x8b1f50 liblognorm: prev subtree 0x8b16a0 liblognorm: new subtree 0x8b1f50 liblognorm: addSampToTree 46 of 167 liblognorm: parsed literal: ' sshd[' liblognorm: buildPTree: begin at 0x8b1f50, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 52 of 167 liblognorm: parsed field: 'procid' liblognorm: got new subtree 0x8b2800 liblognorm: prev subtree 0x8b1f50 liblognorm: new subtree 0x8b2800 liblognorm: addSampToTree 67 of 167 liblognorm: parsed literal: ']: Failed ' liblognorm: buildPTree: begin at 0x8b2800, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 10, offs 0 liblognorm: addSampToTree 77 of 167 liblognorm: parsed field: 'user_authmethod' liblognorm: got new subtree 0x8b30b0 liblognorm: prev subtree 0x8b2800 liblognorm: new subtree 0x8b30b0 liblognorm: addSampToTree 99 of 167 liblognorm: parsed literal: ' for ' liblognorm: buildPTree: begin at 0x8b30b0, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 5, offs 0 liblognorm: addSampToTree 104 of 167 liblognorm: parsed field: 'username' liblognorm: got new subtree 0x8b3960 liblognorm: prev subtree 0x8b30b0 liblognorm: new subtree 0x8b3960 liblognorm: addSampToTree 119 of 167 liblognorm: parsed literal: ' from ' liblognorm: buildPTree: begin at 0x8b3960, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 125 of 167 liblognorm: parsed field: 'src_ip' liblognorm: got new subtree 0x8b4210 liblognorm: prev subtree 0x8b3960 liblognorm: new subtree 0x8b4210 liblognorm: addSampToTree 138 of 167 liblognorm: parsed literal: ' port ' liblognorm: buildPTree: begin at 0x8b4210, offs 0 liblognorm: case 3.1 liblognorm: addPTree: offs 0 liblognorm: setPrefix lenBuf 6, offs 0 liblognorm: addSampToTree 144 of 167 liblognorm: parsed field: 'src_port' liblognorm: got new subtree 0x8b4ac0 liblognorm: prev subtree 0x8b4210 liblognorm: new subtree 0x8b4ac0 liblognorm: addSampToTree 159 of 167 liblognorm: parsed field: '-' liblognorm: ERROR: invalid field type 'rest' liblognorm: read sample line: 'rule=:sshd%-:rest%' liblognorm: sample line to add: ':sshd%-:rest%' liblognorm: addSampToTree 0 of 59 liblognorm: parsed literal: '<' liblognorm: buildPTree: begin at 0x8b0060, offs 0 liblognorm: buildPTree: tree 0x8b0060, i 0, char '<' liblognorm: case 1.1 liblognorm: addSampToTree 1 of 59 liblognorm: parsed field: 'prio' liblognorm: got new subtree 0x8b5390 liblognorm: merging with tree 0x8b0df0 liblognorm: addSampToTree 14 of 59 liblognorm: parsed literal: '>' liblognorm: buildPTree: begin at 0x8b0df0, offs 0 liblognorm: buildPTree: tree 0x8b0df0, i 0, char '>' liblognorm: case 1.1 liblognorm: addSampToTree 15 of 59 liblognorm: parsed field: 'time' liblognorm: got new subtree 0x8b5c40 liblognorm: merging with tree 0x8b16a0 liblognorm: addSampToTree 34 of 59 liblognorm: parsed literal: ' ' liblognorm: buildPTree: begin at 0x8b16a0, offs 0 liblognorm: buildPTree: tree 0x8b16a0, i 0, char ' ' liblognorm: case 1.1 liblognorm: addSampToTree 35 of 59 liblognorm: parsed field: 'host' liblognorm: got new subtree 0x8b64f0 liblognorm: merging with tree 0x8b1f50 liblognorm: addSampToTree 46 of 59 liblognorm: parsed literal: ' sshd' liblognorm: buildPTree: begin at 0x8b1f50, offs 0 liblognorm: buildPTree: tree 0x8b1f50, i 0, char ' ' liblognorm: buildPTree: tree 0x8b1f50, i 1, char 's' liblognorm: buildPTree: tree 0x8b1f50, i 2, char 's' liblognorm: buildPTree: tree 0x8b1f50, i 3, char 'h' liblognorm: buildPTree: tree 0x8b1f50, i 4, char 'd' liblognorm: case 1.2 liblognorm: splitTree 0x8b1f50 at offs 5 liblognorm: setPrefix lenBuf 5, offs 0 liblognorm: splitTree new tree 0x8b6d40 lenPrefix=5, char ' ' liblognorm: splitTree new case two bb, offs=5, newlen 0 liblognorm: addSampToTree 51 of 59 liblognorm: parsed field: '-' liblognorm: ERROR: invalid field type 'rest' liblognorm: read sample line: 'annotate=passwd:+proc_name="passwd"' liblognorm: sample annotation to add: 'passwd:+proc_name="passwd"' liblognorm: read sample line: 'annotate=netbackup:+proc_name="netbackup"' liblognorm: sample annotation to add: 'netbackup:+proc_name="netbackup"' liblognorm: read sample line: 'annotate=httpd:+proc_name="httpd"' liblognorm: sample annotation to add: 'httpd:+proc_name="httpd"' liblognorm: read sample line: 'annotate=crond:+proc_name="crond"' liblognorm: sample annotation to add: 'crond:+proc_name="crond"' liblognorm: read sample line: 'annotate=atd:+proc_name="atd"' liblognorm: sample annotation to add: 'atd:+proc_name="atd"' liblognorm: read sample line: 'annotate=sudo:+proc_name="sudo"' liblognorm: sample annotation to add: 'sudo:+proc_name="sudo"' liblognorm: read sample line: 'annotate=su:+proc_name="su"' liblognorm: sample annotation to add: 'su:+proc_name="su"' liblognorm: read sample line: 'annotate=sshd:+proc_name="sshd"' liblognorm: sample annotation to add: 'sshd:+proc_name="sshd"' liblognorm: read sample line: 'annotate=dhcpd:+proc_name="dhcpd"' liblognorm: sample annotation to add: 'dhcpd:+proc_name="dhcpd"' liblognorm: read sample line: 'annotate=esx-vpxa:+proc_name="esx-vpxa"' liblognorm: sample annotation to add: 'esx-vpxa:+proc_name="esx-vpxa"' liblognorm: invalid tag field in annotation, line is 'annotate=esx-vpxa:+proc_name="esx-vpxa"' liblognorm: read sample line: 'annotate=vmware:+proc_name="vmware"' liblognorm: sample annotation to add: 'vmware:+proc_name="vmware"' liblognorm: read sample line: 'annotate=vmware_dcui:+proc_name="vmware-dcui"' liblognorm: sample annotation to add: 'vmware_dcui:+proc_name="vmware-dcui"' liblognorm: read sample line: 'annotate=vmware_vobd:+proc_name="vmware-vobd"' liblognorm: sample annotation to add: 'vmware_vobd:+proc_name="vmware-vobd"' liblognorm: read sample line: 'annotate=vmware_hostd:+proc_name="vmware-hostd"' liblognorm: sample annotation to add: 'vmware_hostd:+proc_name="vmware-hostd"' liblognorm: read sample line: 'annotate=vmware_vmauthd:+proc_name="vmware-vmauthd"' liblognorm: sample annotation to add: 'vmware_vmauthd:+proc_name="vmware-vmauthd"' liblognorm: read sample line: 'annotate=rsyslogd:+proc_name="rsyslogd"' liblognorm: sample annotation to add: 'rsyslogd:+proc_name="rsyslogd"' liblognorm: read sample line: 'annotate=autofail:+action="denied"' liblognorm: sample annotation to add: 'autofail:+action="denied"' liblognorm: read sample line: 'annotate=autosucc:+action="authorized"' liblognorm: sample annotation to add: 'autosucc:+action="authorized"' liblognorm: read sample line: 'annotate=authfail:+action="auth-fail"' liblognorm: sample annotation to add: 'authfail:+action="auth-fail"' liblognorm: read sample line: 'annotate=authsucc:+action="auth"' liblognorm: sample annotation to add: 'authsucc:+action="auth"' liblognorm: read sample line: 'annotate=sesopen:+action="ses-opened"' liblognorm: sample annotation to add: 'sesopen:+action="ses-opened"' liblognorm: read sample line: 'annotate=sesclose:+action="ses-closed"' liblognorm: sample annotation to add: 'sesclose:+action="ses-closed"' liblognorm: read sample line: 'annotate=invuser:+details="user invalid"' liblognorm: sample annotation to add: 'invuser:+details="user invalid"' liblognorm: read sample line: 'annotate=exec:+action="exec"' liblognorm: sample annotation to add: 'exec:+action="exec"' liblognorm: read sample line: 'annotate=pwch:+action="password change"' liblognorm: sample annotation to add: 'pwch:+action="password change"' liblognorm: read sample line: 'annotate=open:+action="conn-open"' liblognorm: sample annotation to add: 'open:+action="conn-open"' liblognorm: read sample line: 'annotate=close:+action="conn-close"' liblognorm: sample annotation to add: 'close:+action="conn-close"' liblognorm: read sample line: 'annotate=discon:+action="disconn"' liblognorm: sample annotation to add: 'discon:+action="disconn"' liblognorm: read sample line: 'annotate=discover:+action="discovery"' liblognorm: sample annotation to add: 'discover:+action="discovery"' liblognorm: read sample line: 'annotate=proterr:+action="prot-error"' liblognorm: sample annotation to add: 'proterr:+action="prot-error"' liblognorm: read sample line: 'annotate=hb:+action="heartbeat"' liblognorm: sample annotation to add: 'hb:+action="heartbeat"' number of tree nodes: 13 To normalize: '<86>Jun 1 12:37:41 felix sshd[23795]: Failed password for root from ::1 port 54849 ssh2' liblognorm: 0: prefix compare '<', '<' liblognorm: 1: prefix compare succeeded, still valid liblognorm: 1:trying parser for field 'prio': 0x7fcfc09bb4d0 liblognorm: potential hit, trying subtree liblognorm: 3: prefix compare '>', '>' liblognorm: 4: prefix compare succeeded, still valid liblognorm: 4:trying parser for field 'time': 0x7fcfc09baf70 liblognorm: potential hit, trying subtree liblognorm: 19: prefix compare ' ', ' ' liblognorm: 20: prefix compare succeeded, still valid liblognorm: 20:trying parser for field 'host': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 25: prefix compare ' ', ' ' liblognorm: 26: prefix compare 's', 's' liblognorm: 27: prefix compare 's', 's' liblognorm: 28: prefix compare 'h', 'h' liblognorm: 29: prefix compare 'd', 'd' liblognorm: 30: prefix compare succeeded, still valid liblognorm: 30 no field, trying subtree char '[': 0x8b1f50 liblognorm: 31: prefix compare succeeded, still valid liblognorm: 31:trying parser for field 'procid': 0x7fcfc09bb4d0 liblognorm: potential hit, trying subtree liblognorm: 36: prefix compare ']', ']' liblognorm: 37: prefix compare ':', ':' liblognorm: 38: prefix compare ' ', ' ' liblognorm: 39: prefix compare 'F', 'F' liblognorm: 40: prefix compare 'a', 'a' liblognorm: 41: prefix compare 'i', 'i' liblognorm: 42: prefix compare 'l', 'l' liblognorm: 43: prefix compare 'e', 'e' liblognorm: 44: prefix compare 'd', 'd' liblognorm: 45: prefix compare ' ', ' ' liblognorm: 46: prefix compare succeeded, still valid liblognorm: 46:trying parser for field 'user_authmethod': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 54: prefix compare ' ', ' ' liblognorm: 55: prefix compare 'f', 'f' liblognorm: 56: prefix compare 'o', 'o' liblognorm: 57: prefix compare 'r', 'r' liblognorm: 58: prefix compare ' ', ' ' liblognorm: 59: prefix compare succeeded, still valid liblognorm: 59:trying parser for field 'username': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 63: prefix compare ' ', ' ' liblognorm: 64: prefix compare 'f', 'f' liblognorm: 65: prefix compare 'r', 'r' liblognorm: 66: prefix compare 'o', 'o' liblognorm: 67: prefix compare 'm', 'm' liblognorm: 68: prefix compare ' ', ' ' liblognorm: 69: prefix compare succeeded, still valid liblognorm: 69:trying parser for field 'src_ip': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 72: prefix compare ' ', ' ' liblognorm: 73: prefix compare 'p', 'p' liblognorm: 74: prefix compare 'o', 'o' liblognorm: 75: prefix compare 'r', 'r' liblognorm: 76: prefix compare 't', 't' liblognorm: 77: prefix compare ' ', ' ' liblognorm: 78: prefix compare succeeded, still valid liblognorm: 78:trying parser for field 'src_port': 0x7fcfc09ba790 liblognorm: potential hit, trying subtree liblognorm: 83: prefix compare succeeded, still valid liblognorm: 83 no field, trying subtree char ' ': (nil) liblognorm: 83 returns 5 liblognorm: 78 nonmatch, backtracking required, left=5 liblognorm: 78 no field, trying subtree char '5': (nil) liblognorm: 78 returns 5 liblognorm: 69 nonmatch, backtracking required, left=5 liblognorm: 69 no field, trying subtree char ':': (nil) liblognorm: 69 returns 5 liblognorm: 59 nonmatch, backtracking required, left=5 liblognorm: 59 no field, trying subtree char 'r': (nil) liblognorm: 59 returns 5 liblognorm: 46 nonmatch, backtracking required, left=5 liblognorm: 46 no field, trying subtree char 'p': (nil) liblognorm: 46 returns 5 liblognorm: 31 nonmatch, backtracking required, left=5 liblognorm: 31 no field, trying subtree char '2': (nil) liblognorm: 31 returns 5 liblognorm: 30 returns 5 liblognorm: 20 nonmatch, backtracking required, left=5 liblognorm: 20 no field, trying subtree char 'f': (nil) liblognorm: 20 returns 5 liblognorm: 4 nonmatch, backtracking required, left=5 liblognorm: 4 no field, trying subtree char 'J': (nil) liblognorm: 4 returns 5 liblognorm: 1 nonmatch, backtracking required, left=5 liblognorm: 1 no field, trying subtree char '8': (nil) liblognorm: 1 returns 5 liblognorm: final result for normalizer: left 5, endNode 0x34352074726f7020 normalized: '[cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]' [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] From david at lang.hm Wed Jul 22 18:10:35 2015 From: david at lang.hm (David Lang) Date: Wed, 22 Jul 2015 09:10:35 -0700 (PDT) Subject: [Lognorm] [rsyslog] liblognorm 1.1.2 released - rest parser In-Reply-To: References: Message-ID: I don't know why you are getting the error, but in your debug output there is this line: liblognorm: ERROR: invalid field type 'rest' so for some reason it's not happy with rest. try it with a fieldname instead of - just to eliminate that possibility. David Lang On Wed, 22 Jul 2015, Christopher.Racky at web.de wrote: > Date: Wed, 22 Jul 2015 09:41:37 +0200 > From: Christopher.Racky at web.de > Reply-To: lognorm > To: lognorm at lists.adiscon.com > Cc: rsyslog at lists.adiscon.com > Subject: [Lognorm] [rsyslog] liblognorm 1.1.2 released - rest parser > > Hello, > I'm going to become mad with the rest parser, even in the latest liblognorm release. > I just want to normalize this very basic message using the "rest" field-type: > > echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/name.rb > > > This is my ruleset: > --------------- > prefix=<%prio:number%>%time:date-rfc3164% %host:word% > rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest% > rule=:sshd%-:rest% > --------------- > > I use this alternatively: > --------------- > rule=:<%prio:number%>%time:date-rfc3164% %host:word% sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word% %-:rest% > --------------- > > None of them work. > Result is always: > [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] > > > > > I installed the latest release, see here: > > [root at server ~]# yum info liblognorm1.x86_64 > Loaded plugins: fastestmirror, presto > Loading mirror speeds from cached hostfile > * base: ftp.wrz.de > * extras: centos.bio.lmu.de > * updates: centos.bio.lmu.de > Installed Packages > Name : liblognorm1 > Arch : x86_64 > Version : 1.1.2 > Release : 1.el6 > Size : 103 k > Repo : installed > From repo : rsyslog-v8-stable > Summary : Fast samples-based log normalization library > URL : http://www.liblognorm.com > License : LGPLv2+ > Description : Briefly described, liblognorm is a tool to normalize log data. > : > : People who need to take a look at logs often have a common problem. Logs from > : different machines (from different vendors) usually have different formats for > : their logs. Even if it is the same type of log (e.g. from firewalls), the log > : entries are so different, that it is pretty hard to read these. This is where > : liblognorm comes into the game. With this tool you can normalize all your logs. > : All you need is liblognorm and its dependencies and a sample database that fits > : the logs you want to normalize. > > > > Debug-Output: > > [root at bug ~]# echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/test3.rb -v > liblognorm: read sample line: 'prefix=<%prio:number%>%time:date-rfc3164% %host:word% ' > liblognorm: read sample line: 'rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' > liblognorm: sample line to add: ':sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%' > > liblognorm: addSampToTree 0 of 167 > liblognorm: parsed literal: '<' > liblognorm: buildPTree: begin at 0x8b0060, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 1 of 167 > liblognorm: parsed field: 'prio' > liblognorm: got new subtree 0x8b0df0 > liblognorm: prev subtree 0x8b0060 > liblognorm: new subtree 0x8b0df0 > liblognorm: addSampToTree 14 of 167 > liblognorm: parsed literal: '>' > liblognorm: buildPTree: begin at 0x8b0df0, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 15 of 167 > liblognorm: parsed field: 'time' > liblognorm: got new subtree 0x8b16a0 > liblognorm: prev subtree 0x8b0df0 > liblognorm: new subtree 0x8b16a0 > liblognorm: addSampToTree 34 of 167 > liblognorm: parsed literal: ' ' > liblognorm: buildPTree: begin at 0x8b16a0, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 1, offs 0 > liblognorm: addSampToTree 35 of 167 > liblognorm: parsed field: 'host' > liblognorm: got new subtree 0x8b1f50 > liblognorm: prev subtree 0x8b16a0 > liblognorm: new subtree 0x8b1f50 > liblognorm: addSampToTree 46 of 167 > liblognorm: parsed literal: ' sshd[' > liblognorm: buildPTree: begin at 0x8b1f50, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 6, offs 0 > liblognorm: addSampToTree 52 of 167 > liblognorm: parsed field: 'procid' > liblognorm: got new subtree 0x8b2800 > liblognorm: prev subtree 0x8b1f50 > liblognorm: new subtree 0x8b2800 > liblognorm: addSampToTree 67 of 167 > liblognorm: parsed literal: ']: Failed ' > liblognorm: buildPTree: begin at 0x8b2800, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 10, offs 0 > liblognorm: addSampToTree 77 of 167 > liblognorm: parsed field: 'user_authmethod' > liblognorm: got new subtree 0x8b30b0 > liblognorm: prev subtree 0x8b2800 > liblognorm: new subtree 0x8b30b0 > liblognorm: addSampToTree 99 of 167 > liblognorm: parsed literal: ' for ' > liblognorm: buildPTree: begin at 0x8b30b0, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 5, offs 0 > liblognorm: addSampToTree 104 of 167 > liblognorm: parsed field: 'username' > liblognorm: got new subtree 0x8b3960 > liblognorm: prev subtree 0x8b30b0 > liblognorm: new subtree 0x8b3960 > liblognorm: addSampToTree 119 of 167 > liblognorm: parsed literal: ' from ' > liblognorm: buildPTree: begin at 0x8b3960, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 6, offs 0 > liblognorm: addSampToTree 125 of 167 > liblognorm: parsed field: 'src_ip' > liblognorm: got new subtree 0x8b4210 > liblognorm: prev subtree 0x8b3960 > liblognorm: new subtree 0x8b4210 > liblognorm: addSampToTree 138 of 167 > liblognorm: parsed literal: ' port ' > liblognorm: buildPTree: begin at 0x8b4210, offs 0 > liblognorm: case 3.1 > liblognorm: addPTree: offs 0 > liblognorm: setPrefix lenBuf 6, offs 0 > liblognorm: addSampToTree 144 of 167 > liblognorm: parsed field: 'src_port' > liblognorm: got new subtree 0x8b4ac0 > liblognorm: prev subtree 0x8b4210 > liblognorm: new subtree 0x8b4ac0 > liblognorm: addSampToTree 159 of 167 > liblognorm: parsed field: '-' > liblognorm: ERROR: invalid field type 'rest' > liblognorm: read sample line: 'rule=:sshd%-:rest%' > liblognorm: sample line to add: ':sshd%-:rest%' > > liblognorm: addSampToTree 0 of 59 > liblognorm: parsed literal: '<' > liblognorm: buildPTree: begin at 0x8b0060, offs 0 > liblognorm: buildPTree: tree 0x8b0060, i 0, char '<' > liblognorm: case 1.1 > liblognorm: addSampToTree 1 of 59 > liblognorm: parsed field: 'prio' > liblognorm: got new subtree 0x8b5390 > liblognorm: merging with tree 0x8b0df0 > > liblognorm: addSampToTree 14 of 59 > liblognorm: parsed literal: '>' > liblognorm: buildPTree: begin at 0x8b0df0, offs 0 > liblognorm: buildPTree: tree 0x8b0df0, i 0, char '>' > liblognorm: case 1.1 > liblognorm: addSampToTree 15 of 59 > liblognorm: parsed field: 'time' > liblognorm: got new subtree 0x8b5c40 > liblognorm: merging with tree 0x8b16a0 > > liblognorm: addSampToTree 34 of 59 > liblognorm: parsed literal: ' ' > liblognorm: buildPTree: begin at 0x8b16a0, offs 0 > liblognorm: buildPTree: tree 0x8b16a0, i 0, char ' ' > liblognorm: case 1.1 > liblognorm: addSampToTree 35 of 59 > liblognorm: parsed field: 'host' > liblognorm: got new subtree 0x8b64f0 > liblognorm: merging with tree 0x8b1f50 > > liblognorm: addSampToTree 46 of 59 > liblognorm: parsed literal: ' sshd' > liblognorm: buildPTree: begin at 0x8b1f50, offs 0 > liblognorm: buildPTree: tree 0x8b1f50, i 0, char ' ' > liblognorm: buildPTree: tree 0x8b1f50, i 1, char 's' > liblognorm: buildPTree: tree 0x8b1f50, i 2, char 's' > liblognorm: buildPTree: tree 0x8b1f50, i 3, char 'h' > liblognorm: buildPTree: tree 0x8b1f50, i 4, char 'd' > liblognorm: case 1.2 > liblognorm: splitTree 0x8b1f50 at offs 5 > liblognorm: setPrefix lenBuf 5, offs 0 > liblognorm: splitTree new tree 0x8b6d40 lenPrefix=5, char ' ' > liblognorm: splitTree new case two bb, offs=5, newlen 0 > liblognorm: addSampToTree 51 of 59 > liblognorm: parsed field: '-' > liblognorm: ERROR: invalid field type 'rest' > liblognorm: read sample line: 'annotate=passwd:+proc_name="passwd"' > liblognorm: sample annotation to add: 'passwd:+proc_name="passwd"' > liblognorm: read sample line: 'annotate=netbackup:+proc_name="netbackup"' > liblognorm: sample annotation to add: 'netbackup:+proc_name="netbackup"' > liblognorm: read sample line: 'annotate=httpd:+proc_name="httpd"' > liblognorm: sample annotation to add: 'httpd:+proc_name="httpd"' > liblognorm: read sample line: 'annotate=crond:+proc_name="crond"' > liblognorm: sample annotation to add: 'crond:+proc_name="crond"' > liblognorm: read sample line: 'annotate=atd:+proc_name="atd"' > liblognorm: sample annotation to add: 'atd:+proc_name="atd"' > liblognorm: read sample line: 'annotate=sudo:+proc_name="sudo"' > liblognorm: sample annotation to add: 'sudo:+proc_name="sudo"' > liblognorm: read sample line: 'annotate=su:+proc_name="su"' > liblognorm: sample annotation to add: 'su:+proc_name="su"' > liblognorm: read sample line: 'annotate=sshd:+proc_name="sshd"' > liblognorm: sample annotation to add: 'sshd:+proc_name="sshd"' > liblognorm: read sample line: 'annotate=dhcpd:+proc_name="dhcpd"' > liblognorm: sample annotation to add: 'dhcpd:+proc_name="dhcpd"' > liblognorm: read sample line: 'annotate=esx-vpxa:+proc_name="esx-vpxa"' > liblognorm: sample annotation to add: 'esx-vpxa:+proc_name="esx-vpxa"' > liblognorm: invalid tag field in annotation, line is 'annotate=esx-vpxa:+proc_name="esx-vpxa"' > liblognorm: read sample line: 'annotate=vmware:+proc_name="vmware"' > liblognorm: sample annotation to add: 'vmware:+proc_name="vmware"' > liblognorm: read sample line: 'annotate=vmware_dcui:+proc_name="vmware-dcui"' > liblognorm: sample annotation to add: 'vmware_dcui:+proc_name="vmware-dcui"' > liblognorm: read sample line: 'annotate=vmware_vobd:+proc_name="vmware-vobd"' > liblognorm: sample annotation to add: 'vmware_vobd:+proc_name="vmware-vobd"' > liblognorm: read sample line: 'annotate=vmware_hostd:+proc_name="vmware-hostd"' > liblognorm: sample annotation to add: 'vmware_hostd:+proc_name="vmware-hostd"' > liblognorm: read sample line: 'annotate=vmware_vmauthd:+proc_name="vmware-vmauthd"' > liblognorm: sample annotation to add: 'vmware_vmauthd:+proc_name="vmware-vmauthd"' > liblognorm: read sample line: 'annotate=rsyslogd:+proc_name="rsyslogd"' > liblognorm: sample annotation to add: 'rsyslogd:+proc_name="rsyslogd"' > liblognorm: read sample line: 'annotate=autofail:+action="denied"' > liblognorm: sample annotation to add: 'autofail:+action="denied"' > liblognorm: read sample line: 'annotate=autosucc:+action="authorized"' > liblognorm: sample annotation to add: 'autosucc:+action="authorized"' > liblognorm: read sample line: 'annotate=authfail:+action="auth-fail"' > liblognorm: sample annotation to add: 'authfail:+action="auth-fail"' > liblognorm: read sample line: 'annotate=authsucc:+action="auth"' > liblognorm: sample annotation to add: 'authsucc:+action="auth"' > liblognorm: read sample line: 'annotate=sesopen:+action="ses-opened"' > liblognorm: sample annotation to add: 'sesopen:+action="ses-opened"' > liblognorm: read sample line: 'annotate=sesclose:+action="ses-closed"' > liblognorm: sample annotation to add: 'sesclose:+action="ses-closed"' > liblognorm: read sample line: 'annotate=invuser:+details="user invalid"' > liblognorm: sample annotation to add: 'invuser:+details="user invalid"' > liblognorm: read sample line: 'annotate=exec:+action="exec"' > liblognorm: sample annotation to add: 'exec:+action="exec"' > liblognorm: read sample line: 'annotate=pwch:+action="password change"' > liblognorm: sample annotation to add: 'pwch:+action="password change"' > liblognorm: read sample line: 'annotate=open:+action="conn-open"' > liblognorm: sample annotation to add: 'open:+action="conn-open"' > liblognorm: read sample line: 'annotate=close:+action="conn-close"' > liblognorm: sample annotation to add: 'close:+action="conn-close"' > liblognorm: read sample line: 'annotate=discon:+action="disconn"' > liblognorm: sample annotation to add: 'discon:+action="disconn"' > liblognorm: read sample line: 'annotate=discover:+action="discovery"' > liblognorm: sample annotation to add: 'discover:+action="discovery"' > liblognorm: read sample line: 'annotate=proterr:+action="prot-error"' > liblognorm: sample annotation to add: 'proterr:+action="prot-error"' > liblognorm: read sample line: 'annotate=hb:+action="heartbeat"' > liblognorm: sample annotation to add: 'hb:+action="heartbeat"' > number of tree nodes: 13 > To normalize: '<86>Jun 1 12:37:41 felix sshd[23795]: Failed password for root from ::1 port 54849 ssh2' > liblognorm: 0: prefix compare '<', '<' > liblognorm: 1: prefix compare succeeded, still valid > liblognorm: 1:trying parser for field 'prio': 0x7fcfc09bb4d0 > liblognorm: potential hit, trying subtree > liblognorm: 3: prefix compare '>', '>' > liblognorm: 4: prefix compare succeeded, still valid > liblognorm: 4:trying parser for field 'time': 0x7fcfc09baf70 > liblognorm: potential hit, trying subtree > liblognorm: 19: prefix compare ' ', ' ' > liblognorm: 20: prefix compare succeeded, still valid > liblognorm: 20:trying parser for field 'host': 0x7fcfc09ba790 > liblognorm: potential hit, trying subtree > liblognorm: 25: prefix compare ' ', ' ' > liblognorm: 26: prefix compare 's', 's' > liblognorm: 27: prefix compare 's', 's' > liblognorm: 28: prefix compare 'h', 'h' > liblognorm: 29: prefix compare 'd', 'd' > liblognorm: 30: prefix compare succeeded, still valid > liblognorm: 30 no field, trying subtree char '[': 0x8b1f50 > liblognorm: 31: prefix compare succeeded, still valid > liblognorm: 31:trying parser for field 'procid': 0x7fcfc09bb4d0 > liblognorm: potential hit, trying subtree > liblognorm: 36: prefix compare ']', ']' > liblognorm: 37: prefix compare ':', ':' > liblognorm: 38: prefix compare ' ', ' ' > liblognorm: 39: prefix compare 'F', 'F' > liblognorm: 40: prefix compare 'a', 'a' > liblognorm: 41: prefix compare 'i', 'i' > liblognorm: 42: prefix compare 'l', 'l' > liblognorm: 43: prefix compare 'e', 'e' > liblognorm: 44: prefix compare 'd', 'd' > liblognorm: 45: prefix compare ' ', ' ' > liblognorm: 46: prefix compare succeeded, still valid > liblognorm: 46:trying parser for field 'user_authmethod': 0x7fcfc09ba790 > liblognorm: potential hit, trying subtree > liblognorm: 54: prefix compare ' ', ' ' > liblognorm: 55: prefix compare 'f', 'f' > liblognorm: 56: prefix compare 'o', 'o' > liblognorm: 57: prefix compare 'r', 'r' > liblognorm: 58: prefix compare ' ', ' ' > liblognorm: 59: prefix compare succeeded, still valid > liblognorm: 59:trying parser for field 'username': 0x7fcfc09ba790 > liblognorm: potential hit, trying subtree > liblognorm: 63: prefix compare ' ', ' ' > liblognorm: 64: prefix compare 'f', 'f' > liblognorm: 65: prefix compare 'r', 'r' > liblognorm: 66: prefix compare 'o', 'o' > liblognorm: 67: prefix compare 'm', 'm' > liblognorm: 68: prefix compare ' ', ' ' > liblognorm: 69: prefix compare succeeded, still valid > liblognorm: 69:trying parser for field 'src_ip': 0x7fcfc09ba790 > liblognorm: potential hit, trying subtree > liblognorm: 72: prefix compare ' ', ' ' > liblognorm: 73: prefix compare 'p', 'p' > liblognorm: 74: prefix compare 'o', 'o' > liblognorm: 75: prefix compare 'r', 'r' > liblognorm: 76: prefix compare 't', 't' > liblognorm: 77: prefix compare ' ', ' ' > liblognorm: 78: prefix compare succeeded, still valid > liblognorm: 78:trying parser for field 'src_port': 0x7fcfc09ba790 > liblognorm: potential hit, trying subtree > liblognorm: 83: prefix compare succeeded, still valid > liblognorm: 83 no field, trying subtree char ' ': (nil) > liblognorm: 83 returns 5 > liblognorm: 78 nonmatch, backtracking required, left=5 > liblognorm: 78 no field, trying subtree char '5': (nil) > liblognorm: 78 returns 5 > liblognorm: 69 nonmatch, backtracking required, left=5 > liblognorm: 69 no field, trying subtree char ':': (nil) > liblognorm: 69 returns 5 > liblognorm: 59 nonmatch, backtracking required, left=5 > liblognorm: 59 no field, trying subtree char 'r': (nil) > liblognorm: 59 returns 5 > liblognorm: 46 nonmatch, backtracking required, left=5 > liblognorm: 46 no field, trying subtree char 'p': (nil) > liblognorm: 46 returns 5 > liblognorm: 31 nonmatch, backtracking required, left=5 > liblognorm: 31 no field, trying subtree char '2': (nil) > liblognorm: 31 returns 5 > liblognorm: 30 returns 5 > liblognorm: 20 nonmatch, backtracking required, left=5 > liblognorm: 20 no field, trying subtree char 'f': (nil) > liblognorm: 20 returns 5 > liblognorm: 4 nonmatch, backtracking required, left=5 > liblognorm: 4 no field, trying subtree char 'J': (nil) > liblognorm: 4 returns 5 > liblognorm: 1 nonmatch, backtracking required, left=5 > liblognorm: 1 no field, trying subtree char '8': (nil) > liblognorm: 1 returns 5 > liblognorm: final result for normalizer: left 5, endNode 0x34352074726f7020 > normalized: '[cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]' > [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"] > _______________________________________________ > Lognorm mailing list > Lognorm at lists.adiscon.com > http://lists.adiscon.net/mailman/listinfo/lognorm >