[Lognorm] [rsyslog] liblognorm 1.1.2 released - rest parser
David Lang
david at lang.hm
Wed Jul 22 18:10:35 CEST 2015
I don't know why you are getting the error, but in your debug output there is
this line:
liblognorm: ERROR: invalid field type 'rest'
so for some reason it's not happy with rest. try it with a fieldname instead of
- just to eliminate that possibility.
David Lang
On Wed, 22 Jul 2015, Christopher.Racky at web.de wrote:
> Date: Wed, 22 Jul 2015 09:41:37 +0200
> From: Christopher.Racky at web.de
> Reply-To: lognorm <lognorm at lists.adiscon.com>
> To: lognorm at lists.adiscon.com
> Cc: rsyslog at lists.adiscon.com
> Subject: [Lognorm] [rsyslog] liblognorm 1.1.2 released - rest parser
>
> Hello,
> I'm going to become mad with the rest parser, even in the latest liblognorm release.
> I just want to normalize this very basic message using the "rest" field-type:
>
> echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/name.rb
>
>
> This is my ruleset:
> ---------------
> prefix=<%prio:number%>%time:date-rfc3164% %host:word%
> rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%
> rule=:sshd%-:rest%
> ---------------
>
> I use this alternatively:
> ---------------
> rule=:<%prio:number%>%time:date-rfc3164% %host:word% sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word% %-:rest%
> ---------------
>
> None of them work.
> Result is always:
> [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]
>
>
>
>
> I installed the latest release, see here:
>
> [root at server ~]# yum info liblognorm1.x86_64
> Loaded plugins: fastestmirror, presto
> Loading mirror speeds from cached hostfile
> * base: ftp.wrz.de
> * extras: centos.bio.lmu.de
> * updates: centos.bio.lmu.de
> Installed Packages
> Name : liblognorm1
> Arch : x86_64
> Version : 1.1.2
> Release : 1.el6
> Size : 103 k
> Repo : installed
> From repo : rsyslog-v8-stable
> Summary : Fast samples-based log normalization library
> URL : http://www.liblognorm.com
> License : LGPLv2+
> Description : Briefly described, liblognorm is a tool to normalize log data.
> :
> : People who need to take a look at logs often have a common problem. Logs from
> : different machines (from different vendors) usually have different formats for
> : their logs. Even if it is the same type of log (e.g. from firewalls), the log
> : entries are so different, that it is pretty hard to read these. This is where
> : liblognorm comes into the game. With this tool you can normalize all your logs.
> : All you need is liblognorm and its dependencies and a sample database that fits
> : the logs you want to normalize.
>
>
>
> Debug-Output:
>
> [root at bug ~]# echo "<86>Jun 1 12:37:41 server sshd[23795]: Failed password for root from ::1 port 54849 ssh2" | lognormalizer -r /etc/test3.rb -v
> liblognorm: read sample line: 'prefix=<%prio:number%>%time:date-rfc3164% %host:word% '
> liblognorm: read sample line: 'rule=:sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%'
> liblognorm: sample line to add: ':sshd[%procid:number%]: Failed %user_authmethod:word% for %username:word% from %src_ip:word% port %src_port:word%%-:rest%'
>
> liblognorm: addSampToTree 0 of 167
> liblognorm: parsed literal: '<'
> liblognorm: buildPTree: begin at 0x8b0060, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 1, offs 0
> liblognorm: addSampToTree 1 of 167
> liblognorm: parsed field: 'prio'
> liblognorm: got new subtree 0x8b0df0
> liblognorm: prev subtree 0x8b0060
> liblognorm: new subtree 0x8b0df0
> liblognorm: addSampToTree 14 of 167
> liblognorm: parsed literal: '>'
> liblognorm: buildPTree: begin at 0x8b0df0, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 1, offs 0
> liblognorm: addSampToTree 15 of 167
> liblognorm: parsed field: 'time'
> liblognorm: got new subtree 0x8b16a0
> liblognorm: prev subtree 0x8b0df0
> liblognorm: new subtree 0x8b16a0
> liblognorm: addSampToTree 34 of 167
> liblognorm: parsed literal: ' '
> liblognorm: buildPTree: begin at 0x8b16a0, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 1, offs 0
> liblognorm: addSampToTree 35 of 167
> liblognorm: parsed field: 'host'
> liblognorm: got new subtree 0x8b1f50
> liblognorm: prev subtree 0x8b16a0
> liblognorm: new subtree 0x8b1f50
> liblognorm: addSampToTree 46 of 167
> liblognorm: parsed literal: ' sshd['
> liblognorm: buildPTree: begin at 0x8b1f50, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 6, offs 0
> liblognorm: addSampToTree 52 of 167
> liblognorm: parsed field: 'procid'
> liblognorm: got new subtree 0x8b2800
> liblognorm: prev subtree 0x8b1f50
> liblognorm: new subtree 0x8b2800
> liblognorm: addSampToTree 67 of 167
> liblognorm: parsed literal: ']: Failed '
> liblognorm: buildPTree: begin at 0x8b2800, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 10, offs 0
> liblognorm: addSampToTree 77 of 167
> liblognorm: parsed field: 'user_authmethod'
> liblognorm: got new subtree 0x8b30b0
> liblognorm: prev subtree 0x8b2800
> liblognorm: new subtree 0x8b30b0
> liblognorm: addSampToTree 99 of 167
> liblognorm: parsed literal: ' for '
> liblognorm: buildPTree: begin at 0x8b30b0, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 5, offs 0
> liblognorm: addSampToTree 104 of 167
> liblognorm: parsed field: 'username'
> liblognorm: got new subtree 0x8b3960
> liblognorm: prev subtree 0x8b30b0
> liblognorm: new subtree 0x8b3960
> liblognorm: addSampToTree 119 of 167
> liblognorm: parsed literal: ' from '
> liblognorm: buildPTree: begin at 0x8b3960, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 6, offs 0
> liblognorm: addSampToTree 125 of 167
> liblognorm: parsed field: 'src_ip'
> liblognorm: got new subtree 0x8b4210
> liblognorm: prev subtree 0x8b3960
> liblognorm: new subtree 0x8b4210
> liblognorm: addSampToTree 138 of 167
> liblognorm: parsed literal: ' port '
> liblognorm: buildPTree: begin at 0x8b4210, offs 0
> liblognorm: case 3.1
> liblognorm: addPTree: offs 0
> liblognorm: setPrefix lenBuf 6, offs 0
> liblognorm: addSampToTree 144 of 167
> liblognorm: parsed field: 'src_port'
> liblognorm: got new subtree 0x8b4ac0
> liblognorm: prev subtree 0x8b4210
> liblognorm: new subtree 0x8b4ac0
> liblognorm: addSampToTree 159 of 167
> liblognorm: parsed field: '-'
> liblognorm: ERROR: invalid field type 'rest'
> liblognorm: read sample line: 'rule=:sshd%-:rest%'
> liblognorm: sample line to add: ':sshd%-:rest%'
>
> liblognorm: addSampToTree 0 of 59
> liblognorm: parsed literal: '<'
> liblognorm: buildPTree: begin at 0x8b0060, offs 0
> liblognorm: buildPTree: tree 0x8b0060, i 0, char '<'
> liblognorm: case 1.1
> liblognorm: addSampToTree 1 of 59
> liblognorm: parsed field: 'prio'
> liblognorm: got new subtree 0x8b5390
> liblognorm: merging with tree 0x8b0df0
>
> liblognorm: addSampToTree 14 of 59
> liblognorm: parsed literal: '>'
> liblognorm: buildPTree: begin at 0x8b0df0, offs 0
> liblognorm: buildPTree: tree 0x8b0df0, i 0, char '>'
> liblognorm: case 1.1
> liblognorm: addSampToTree 15 of 59
> liblognorm: parsed field: 'time'
> liblognorm: got new subtree 0x8b5c40
> liblognorm: merging with tree 0x8b16a0
>
> liblognorm: addSampToTree 34 of 59
> liblognorm: parsed literal: ' '
> liblognorm: buildPTree: begin at 0x8b16a0, offs 0
> liblognorm: buildPTree: tree 0x8b16a0, i 0, char ' '
> liblognorm: case 1.1
> liblognorm: addSampToTree 35 of 59
> liblognorm: parsed field: 'host'
> liblognorm: got new subtree 0x8b64f0
> liblognorm: merging with tree 0x8b1f50
>
> liblognorm: addSampToTree 46 of 59
> liblognorm: parsed literal: ' sshd'
> liblognorm: buildPTree: begin at 0x8b1f50, offs 0
> liblognorm: buildPTree: tree 0x8b1f50, i 0, char ' '
> liblognorm: buildPTree: tree 0x8b1f50, i 1, char 's'
> liblognorm: buildPTree: tree 0x8b1f50, i 2, char 's'
> liblognorm: buildPTree: tree 0x8b1f50, i 3, char 'h'
> liblognorm: buildPTree: tree 0x8b1f50, i 4, char 'd'
> liblognorm: case 1.2
> liblognorm: splitTree 0x8b1f50 at offs 5
> liblognorm: setPrefix lenBuf 5, offs 0
> liblognorm: splitTree new tree 0x8b6d40 lenPrefix=5, char ' '
> liblognorm: splitTree new case two bb, offs=5, newlen 0
> liblognorm: addSampToTree 51 of 59
> liblognorm: parsed field: '-'
> liblognorm: ERROR: invalid field type 'rest'
> liblognorm: read sample line: 'annotate=passwd:+proc_name="passwd"'
> liblognorm: sample annotation to add: 'passwd:+proc_name="passwd"'
> liblognorm: read sample line: 'annotate=netbackup:+proc_name="netbackup"'
> liblognorm: sample annotation to add: 'netbackup:+proc_name="netbackup"'
> liblognorm: read sample line: 'annotate=httpd:+proc_name="httpd"'
> liblognorm: sample annotation to add: 'httpd:+proc_name="httpd"'
> liblognorm: read sample line: 'annotate=crond:+proc_name="crond"'
> liblognorm: sample annotation to add: 'crond:+proc_name="crond"'
> liblognorm: read sample line: 'annotate=atd:+proc_name="atd"'
> liblognorm: sample annotation to add: 'atd:+proc_name="atd"'
> liblognorm: read sample line: 'annotate=sudo:+proc_name="sudo"'
> liblognorm: sample annotation to add: 'sudo:+proc_name="sudo"'
> liblognorm: read sample line: 'annotate=su:+proc_name="su"'
> liblognorm: sample annotation to add: 'su:+proc_name="su"'
> liblognorm: read sample line: 'annotate=sshd:+proc_name="sshd"'
> liblognorm: sample annotation to add: 'sshd:+proc_name="sshd"'
> liblognorm: read sample line: 'annotate=dhcpd:+proc_name="dhcpd"'
> liblognorm: sample annotation to add: 'dhcpd:+proc_name="dhcpd"'
> liblognorm: read sample line: 'annotate=esx-vpxa:+proc_name="esx-vpxa"'
> liblognorm: sample annotation to add: 'esx-vpxa:+proc_name="esx-vpxa"'
> liblognorm: invalid tag field in annotation, line is 'annotate=esx-vpxa:+proc_name="esx-vpxa"'
> liblognorm: read sample line: 'annotate=vmware:+proc_name="vmware"'
> liblognorm: sample annotation to add: 'vmware:+proc_name="vmware"'
> liblognorm: read sample line: 'annotate=vmware_dcui:+proc_name="vmware-dcui"'
> liblognorm: sample annotation to add: 'vmware_dcui:+proc_name="vmware-dcui"'
> liblognorm: read sample line: 'annotate=vmware_vobd:+proc_name="vmware-vobd"'
> liblognorm: sample annotation to add: 'vmware_vobd:+proc_name="vmware-vobd"'
> liblognorm: read sample line: 'annotate=vmware_hostd:+proc_name="vmware-hostd"'
> liblognorm: sample annotation to add: 'vmware_hostd:+proc_name="vmware-hostd"'
> liblognorm: read sample line: 'annotate=vmware_vmauthd:+proc_name="vmware-vmauthd"'
> liblognorm: sample annotation to add: 'vmware_vmauthd:+proc_name="vmware-vmauthd"'
> liblognorm: read sample line: 'annotate=rsyslogd:+proc_name="rsyslogd"'
> liblognorm: sample annotation to add: 'rsyslogd:+proc_name="rsyslogd"'
> liblognorm: read sample line: 'annotate=autofail:+action="denied"'
> liblognorm: sample annotation to add: 'autofail:+action="denied"'
> liblognorm: read sample line: 'annotate=autosucc:+action="authorized"'
> liblognorm: sample annotation to add: 'autosucc:+action="authorized"'
> liblognorm: read sample line: 'annotate=authfail:+action="auth-fail"'
> liblognorm: sample annotation to add: 'authfail:+action="auth-fail"'
> liblognorm: read sample line: 'annotate=authsucc:+action="auth"'
> liblognorm: sample annotation to add: 'authsucc:+action="auth"'
> liblognorm: read sample line: 'annotate=sesopen:+action="ses-opened"'
> liblognorm: sample annotation to add: 'sesopen:+action="ses-opened"'
> liblognorm: read sample line: 'annotate=sesclose:+action="ses-closed"'
> liblognorm: sample annotation to add: 'sesclose:+action="ses-closed"'
> liblognorm: read sample line: 'annotate=invuser:+details="user invalid"'
> liblognorm: sample annotation to add: 'invuser:+details="user invalid"'
> liblognorm: read sample line: 'annotate=exec:+action="exec"'
> liblognorm: sample annotation to add: 'exec:+action="exec"'
> liblognorm: read sample line: 'annotate=pwch:+action="password change"'
> liblognorm: sample annotation to add: 'pwch:+action="password change"'
> liblognorm: read sample line: 'annotate=open:+action="conn-open"'
> liblognorm: sample annotation to add: 'open:+action="conn-open"'
> liblognorm: read sample line: 'annotate=close:+action="conn-close"'
> liblognorm: sample annotation to add: 'close:+action="conn-close"'
> liblognorm: read sample line: 'annotate=discon:+action="disconn"'
> liblognorm: sample annotation to add: 'discon:+action="disconn"'
> liblognorm: read sample line: 'annotate=discover:+action="discovery"'
> liblognorm: sample annotation to add: 'discover:+action="discovery"'
> liblognorm: read sample line: 'annotate=proterr:+action="prot-error"'
> liblognorm: sample annotation to add: 'proterr:+action="prot-error"'
> liblognorm: read sample line: 'annotate=hb:+action="heartbeat"'
> liblognorm: sample annotation to add: 'hb:+action="heartbeat"'
> number of tree nodes: 13
> To normalize: '<86>Jun 1 12:37:41 felix sshd[23795]: Failed password for root from ::1 port 54849 ssh2'
> liblognorm: 0: prefix compare '<', '<'
> liblognorm: 1: prefix compare succeeded, still valid
> liblognorm: 1:trying parser for field 'prio': 0x7fcfc09bb4d0
> liblognorm: potential hit, trying subtree
> liblognorm: 3: prefix compare '>', '>'
> liblognorm: 4: prefix compare succeeded, still valid
> liblognorm: 4:trying parser for field 'time': 0x7fcfc09baf70
> liblognorm: potential hit, trying subtree
> liblognorm: 19: prefix compare ' ', ' '
> liblognorm: 20: prefix compare succeeded, still valid
> liblognorm: 20:trying parser for field 'host': 0x7fcfc09ba790
> liblognorm: potential hit, trying subtree
> liblognorm: 25: prefix compare ' ', ' '
> liblognorm: 26: prefix compare 's', 's'
> liblognorm: 27: prefix compare 's', 's'
> liblognorm: 28: prefix compare 'h', 'h'
> liblognorm: 29: prefix compare 'd', 'd'
> liblognorm: 30: prefix compare succeeded, still valid
> liblognorm: 30 no field, trying subtree char '[': 0x8b1f50
> liblognorm: 31: prefix compare succeeded, still valid
> liblognorm: 31:trying parser for field 'procid': 0x7fcfc09bb4d0
> liblognorm: potential hit, trying subtree
> liblognorm: 36: prefix compare ']', ']'
> liblognorm: 37: prefix compare ':', ':'
> liblognorm: 38: prefix compare ' ', ' '
> liblognorm: 39: prefix compare 'F', 'F'
> liblognorm: 40: prefix compare 'a', 'a'
> liblognorm: 41: prefix compare 'i', 'i'
> liblognorm: 42: prefix compare 'l', 'l'
> liblognorm: 43: prefix compare 'e', 'e'
> liblognorm: 44: prefix compare 'd', 'd'
> liblognorm: 45: prefix compare ' ', ' '
> liblognorm: 46: prefix compare succeeded, still valid
> liblognorm: 46:trying parser for field 'user_authmethod': 0x7fcfc09ba790
> liblognorm: potential hit, trying subtree
> liblognorm: 54: prefix compare ' ', ' '
> liblognorm: 55: prefix compare 'f', 'f'
> liblognorm: 56: prefix compare 'o', 'o'
> liblognorm: 57: prefix compare 'r', 'r'
> liblognorm: 58: prefix compare ' ', ' '
> liblognorm: 59: prefix compare succeeded, still valid
> liblognorm: 59:trying parser for field 'username': 0x7fcfc09ba790
> liblognorm: potential hit, trying subtree
> liblognorm: 63: prefix compare ' ', ' '
> liblognorm: 64: prefix compare 'f', 'f'
> liblognorm: 65: prefix compare 'r', 'r'
> liblognorm: 66: prefix compare 'o', 'o'
> liblognorm: 67: prefix compare 'm', 'm'
> liblognorm: 68: prefix compare ' ', ' '
> liblognorm: 69: prefix compare succeeded, still valid
> liblognorm: 69:trying parser for field 'src_ip': 0x7fcfc09ba790
> liblognorm: potential hit, trying subtree
> liblognorm: 72: prefix compare ' ', ' '
> liblognorm: 73: prefix compare 'p', 'p'
> liblognorm: 74: prefix compare 'o', 'o'
> liblognorm: 75: prefix compare 'r', 'r'
> liblognorm: 76: prefix compare 't', 't'
> liblognorm: 77: prefix compare ' ', ' '
> liblognorm: 78: prefix compare succeeded, still valid
> liblognorm: 78:trying parser for field 'src_port': 0x7fcfc09ba790
> liblognorm: potential hit, trying subtree
> liblognorm: 83: prefix compare succeeded, still valid
> liblognorm: 83 no field, trying subtree char ' ': (nil)
> liblognorm: 83 returns 5
> liblognorm: 78 nonmatch, backtracking required, left=5
> liblognorm: 78 no field, trying subtree char '5': (nil)
> liblognorm: 78 returns 5
> liblognorm: 69 nonmatch, backtracking required, left=5
> liblognorm: 69 no field, trying subtree char ':': (nil)
> liblognorm: 69 returns 5
> liblognorm: 59 nonmatch, backtracking required, left=5
> liblognorm: 59 no field, trying subtree char 'r': (nil)
> liblognorm: 59 returns 5
> liblognorm: 46 nonmatch, backtracking required, left=5
> liblognorm: 46 no field, trying subtree char 'p': (nil)
> liblognorm: 46 returns 5
> liblognorm: 31 nonmatch, backtracking required, left=5
> liblognorm: 31 no field, trying subtree char '2': (nil)
> liblognorm: 31 returns 5
> liblognorm: 30 returns 5
> liblognorm: 20 nonmatch, backtracking required, left=5
> liblognorm: 20 no field, trying subtree char 'f': (nil)
> liblognorm: 20 returns 5
> liblognorm: 4 nonmatch, backtracking required, left=5
> liblognorm: 4 no field, trying subtree char 'J': (nil)
> liblognorm: 4 returns 5
> liblognorm: 1 nonmatch, backtracking required, left=5
> liblognorm: 1 no field, trying subtree char '8': (nil)
> liblognorm: 1 returns 5
> liblognorm: final result for normalizer: left 5, endNode 0x34352074726f7020
> normalized: '[cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]'
> [cee at 115 originalmsg="<86>Jun 1 12:37:41 server sshd[23795\]: Failed password for root from ::1 port 54849 ssh2" unparsed-data=" ssh2"]
> _______________________________________________
> Lognorm mailing list
> Lognorm at lists.adiscon.com
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
More information about the Lognorm
mailing list