From Jose.Castillo at ssa.gov  Wed Jun 26 22:00:06 2013
From: Jose.Castillo at ssa.gov (Castillo, Jose   Contractor)
Date: Wed, 26 Jun 2013 16:00:06 -0400
Subject: [Lognorm] log classification with liblognorm
Message-ID: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>

Hello all,

I started to play using rsyslog/liblognorm a week ago, I'm testing them on a CentOS 6.4 virtual machine, and next packages have been installed from Adiscon repository:
rsyslog-mysql-7.4.1-1.el6.x86_64
rsyslog-mmjsonparse-7.4.1-1.el6.x86_64
rsyslog-7.4.1-1.el6.x86_64
rsyslog-mmnormalize-7.4.1-1.el6.x86_64
rsyslog-udpspoof-7.4.1-1.el6.x86_64
rsyslog-elasticsearch-7.4.1-1.el6.x86_64

I was reading next link, and now I'm trying to select subsets of messages based on liblognorm tags.
http://blog.gerhards.net/2011/04/log-classification-with-liblognorm.html

So far, I was able to parse a message using mmnormalize module, and now I want to create outputs based on the liblognorm tag(s).
Is this option supported within rsyslog.conf? Any example how to do it?

Below my testing files:

=================================================================
rsyslog.conf file:
module (load="imudp")
module (load="mmnormalize")
module (load="mmjsonparse")

input(type="imudp" address="192.168.1.1" port="514" ruleset="test")

template(name="testFormat" type="string" string="%$!all-json%\n")

ruleset(name="test") {
        action(type="mmnormalize" userawmsg="on" rulebase="/data/syslog/rulebase.rb")
        action(type="omfile" file="/data/syslog/test-syslog.log" template="testFormat")
}

rulebase.rb file:
rule=test:%date:word% %host:ipv4% : %%ASA-%number0:number%-%number1:number%: Denied ICMP type=%number2:number%, code=%number3:number% from %origin:ipv4% on interface outside

test-syslog.log file:
{ "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1": "313001", "number0": "3", "host": "0.0.0.0", "date": "2013-06-26T10:47:42+01:00" }

Message:
"2013-06-26T10:47:42+01:00 0.0.0.0 : %ASA-3-313001: Denied ICMP type=8, code=0 from 77.2.2.2 on interface outside"
======================================================================


Thanks in advance for any help,

Jose Castillo



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20130626/33ef1fda/attachment-0001.htm>

From david at lang.hm  Wed Jun 26 21:10:47 2013
From: david at lang.hm (David Lang)
Date: Wed, 26 Jun 2013 12:10:47 -0700 (PDT)
Subject: [Lognorm] log classification with liblognorm
In-Reply-To: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>
References: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>
Message-ID: <alpine.DEB.2.02.1306261210020.22572@nftneq.ynat.uz>

the tags are accessed in rsyslog by $!<tag name>

so where you would have done %hostname% you could do %$!mytag%

David Lang

On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:

> Date: Wed, 26 Jun 2013 16:00:06 -0400
> From: "Castillo, Jose   Contractor" <Jose.Castillo at ssa.gov>
> Reply-To: lognorm <lognorm at lists.adiscon.com>
> To: "lognorm at lists.adiscon.com" <lognorm at lists.adiscon.com>
> Subject: [Lognorm] log classification with liblognorm
> 
> Hello all,
>
> I started to play using rsyslog/liblognorm a week ago, I'm testing them on a CentOS 6.4 virtual machine, and next packages have been installed from Adiscon repository:
> rsyslog-mysql-7.4.1-1.el6.x86_64
> rsyslog-mmjsonparse-7.4.1-1.el6.x86_64
> rsyslog-7.4.1-1.el6.x86_64
> rsyslog-mmnormalize-7.4.1-1.el6.x86_64
> rsyslog-udpspoof-7.4.1-1.el6.x86_64
> rsyslog-elasticsearch-7.4.1-1.el6.x86_64
>
> I was reading next link, and now I'm trying to select subsets of messages based on liblognorm tags.
> http://blog.gerhards.net/2011/04/log-classification-with-liblognorm.html
>
> So far, I was able to parse a message using mmnormalize module, and now I want to create outputs based on the liblognorm tag(s).
> Is this option supported within rsyslog.conf? Any example how to do it?
>
> Below my testing files:
>
> =================================================================
> rsyslog.conf file:
> module (load="imudp")
> module (load="mmnormalize")
> module (load="mmjsonparse")
>
> input(type="imudp" address="192.168.1.1" port="514" ruleset="test")
>
> template(name="testFormat" type="string" string="%$!all-json%\n")
>
> ruleset(name="test") {
>        action(type="mmnormalize" userawmsg="on" rulebase="/data/syslog/rulebase.rb")
>        action(type="omfile" file="/data/syslog/test-syslog.log" template="testFormat")
> }
>
> rulebase.rb file:
> rule=test:%date:word% %host:ipv4% : %%ASA-%number0:number%-%number1:number%: Denied ICMP type=%number2:number%, code=%number3:number% from %origin:ipv4% on interface outside
>
> test-syslog.log file:
> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1": "313001", "number0": "3", "host": "0.0.0.0", "date": "2013-06-26T10:47:42+01:00" }
>
> Message:
> "2013-06-26T10:47:42+01:00 0.0.0.0 : %ASA-3-313001: Denied ICMP type=8, code=0 from 77.2.2.2 on interface outside"
> ======================================================================
>
>
> Thanks in advance for any help,
>
> Jose Castillo
>
>
>
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From Jose.Castillo at ssa.gov  Wed Jun 26 22:41:53 2013
From: Jose.Castillo at ssa.gov (Castillo, Jose   Contractor)
Date: Wed, 26 Jun 2013 16:41:53 -0400
Subject: [Lognorm] log classification with liblognorm
In-Reply-To: <alpine.DEB.2.02.1306261210020.22572@nftneq.ynat.uz>
References: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>
	<alpine.DEB.2.02.1306261210020.22572@nftneq.ynat.uz>
Message-ID: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD930B@HQ-MB-07.ba.ad.ssa.gov>

David,

I modified the template:
template(name="testFormat" type="string" string="%$!all-json%,tag='%$!mytag%'")

I also modified the rule to include mytag as a tag:
rule=mytag:%date:word% %host:ipv4% : %%ASA-%number0:number%-%number1:number%: Denied ICMP type=%number2:number%, code=%number3:number% from %origin:ipv4% on interface outside

And now I getting :
{ "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1": "313001", "number0": "3", "host": "0.0.0.0", "date": "2013-06-26T10:47:42+01:00" },tag=''

No tags are being written to the output file.

What am I doing wrong?

Thanks,

Jose

-----Original Message-----
From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-bounces at lists.adiscon.com] On Behalf Of David Lang
Sent: Wednesday, June 26, 2013 3:11 PM
To: lognorm
Subject: Re: [Lognorm] log classification with liblognorm

the tags are accessed in rsyslog by $!<tag name>

so where you would have done %hostname% you could do %$!mytag%

David Lang

On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:

> Date: Wed, 26 Jun 2013 16:00:06 -0400
> From: "Castillo, Jose   Contractor" <Jose.Castillo at ssa.gov<mailto:Jose.Castillo at ssa.gov>>
> Reply-To: lognorm <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
> To: "lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>" <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
> Subject: [Lognorm] log classification with liblognorm
>
> Hello all,
>
> I started to play using rsyslog/liblognorm a week ago, I'm testing them on a CentOS 6.4 virtual machine, and next packages have been installed from Adiscon repository:
> rsyslog-mysql-7.4.1-1.el6.x86_64
> rsyslog-mmjsonparse-7.4.1-1.el6.x86_64
> rsyslog-7.4.1-1.el6.x86_64
> rsyslog-mmnormalize-7.4.1-1.el6.x86_64
> rsyslog-udpspoof-7.4.1-1.el6.x86_64
> rsyslog-elasticsearch-7.4.1-1.el6.x86_64
>
> I was reading next link, and now I'm trying to select subsets of messages based on liblognorm tags.
> http://blog.gerhards.net/2011/04/log-classification-with-liblognorm.ht
> ml
>
> So far, I was able to parse a message using mmnormalize module, and now I want to create outputs based on the liblognorm tag(s).
> Is this option supported within rsyslog.conf? Any example how to do it?
>
> Below my testing files:
>
> =================================================================
> rsyslog.conf file:
> module (load="imudp")
> module (load="mmnormalize")
> module (load="mmjsonparse")
>
> input(type="imudp" address="192.168.1.1" port="514" ruleset="test")
>
> template(name="testFormat" type="string" string="%$!all-json%\n")
>
> ruleset(name="test") {
>        action(type="mmnormalize" userawmsg="on" rulebase="/data/syslog/rulebase.rb")
>        action(type="omfile" file="/data/syslog/test-syslog.log"
> template="testFormat") }
>
> rulebase.rb file:
> rule=test:%date:word% %host:ipv4% :
> %%ASA-%number0:number%-%number1:number%: Denied ICMP
> type=%number2:number%, code=%number3:number% from %origin:ipv4% on
> interface outside
>
> test-syslog.log file:
> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1":
> "313001", "number0": "3", "host": "0.0.0.0", "date":
> "2013-06-26T10:47:42+01:00" }
>
> Message:
> "2013-06-26T10:47:42+01:00 0.0.0.0 : %ASA-3-313001: Denied ICMP type=8, code=0 from 77.2.2.2 on interface outside"
> ======================================================================
>
>
> Thanks in advance for any help,
>
> Jose Castillo
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.adiscon.net/pipermail/lognorm/attachments/20130626/f6faff47/attachment.htm>

From david at lang.hm  Wed Jun 26 22:54:43 2013
From: david at lang.hm (David Lang)
Date: Wed, 26 Jun 2013 13:54:43 -0700 (PDT)
Subject: [Lognorm] log classification with liblognorm
In-Reply-To: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD930B@HQ-MB-07.ba.ad.ssa.gov>
References: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>
	<alpine.DEB.2.02.1306261210020.22572@nftneq.ynat.uz>
	<701C784DE99B0C4C9FDA9EDE66E23E661E4CAD930B@HQ-MB-07.ba.ad.ssa.gov>
Message-ID: <alpine.DEB.2.02.1306261351570.22572@nftneq.ynat.uz>

First off, I have not used mmlognorm for anything other than seeing that it 
works, so take that into consideration :-)

However, as I'm looking at your rule, you have "mytag" outside of any matching, 
so it would be treated as a literal, just like "Denied ICMP" wouldn't it?

in your syslog template, try putting %$!number2% and see if it outputs the 
correct value. it looks to me like that variable is being set correctly.

David Lang

On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:

> David,
>
> I modified the template:
> template(name="testFormat" type="string" string="%$!all-json%,tag='%$!mytag%'")
>
> I also modified the rule to include mytag as a tag:
> rule=mytag:%date:word% %host:ipv4% : %%ASA-%number0:number%-%number1:number%: Denied ICMP type=%number2:number%, code=%number3:number% from %origin:ipv4% on interface outside
>
> And now I getting :
> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1": "313001", "number0": "3", "host": "0.0.0.0", "date": "2013-06-26T10:47:42+01:00" },tag=''
>
> No tags are being written to the output file.
>
> What am I doing wrong?
>
> Thanks,
>
> Jose
>
> -----Original Message-----
> From: lognorm-bounces at lists.adiscon.com [mailto:lognorm-bounces at lists.adiscon.com] On Behalf Of David Lang
> Sent: Wednesday, June 26, 2013 3:11 PM
> To: lognorm
> Subject: Re: [Lognorm] log classification with liblognorm
>
> the tags are accessed in rsyslog by $!<tag name>
>
> so where you would have done %hostname% you could do %$!mytag%
>
> David Lang
>
> On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:
>
>> Date: Wed, 26 Jun 2013 16:00:06 -0400
>> From: "Castillo, Jose   Contractor" <Jose.Castillo at ssa.gov<mailto:Jose.Castillo at ssa.gov>>
>> Reply-To: lognorm <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
>> To: "lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>" <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
>> Subject: [Lognorm] log classification with liblognorm
>>
>> Hello all,
>>
>> I started to play using rsyslog/liblognorm a week ago, I'm testing them on a CentOS 6.4 virtual machine, and next packages have been installed from Adiscon repository:
>> rsyslog-mysql-7.4.1-1.el6.x86_64
>> rsyslog-mmjsonparse-7.4.1-1.el6.x86_64
>> rsyslog-7.4.1-1.el6.x86_64
>> rsyslog-mmnormalize-7.4.1-1.el6.x86_64
>> rsyslog-udpspoof-7.4.1-1.el6.x86_64
>> rsyslog-elasticsearch-7.4.1-1.el6.x86_64
>>
>> I was reading next link, and now I'm trying to select subsets of messages based on liblognorm tags.
>> http://blog.gerhards.net/2011/04/log-classification-with-liblognorm.ht
>> ml
>>
>> So far, I was able to parse a message using mmnormalize module, and now I want to create outputs based on the liblognorm tag(s).
>> Is this option supported within rsyslog.conf? Any example how to do it?
>>
>> Below my testing files:
>>
>> =================================================================
>> rsyslog.conf file:
>> module (load="imudp")
>> module (load="mmnormalize")
>> module (load="mmjsonparse")
>>
>> input(type="imudp" address="192.168.1.1" port="514" ruleset="test")
>>
>> template(name="testFormat" type="string" string="%$!all-json%\n")
>>
>> ruleset(name="test") {
>>        action(type="mmnormalize" userawmsg="on" rulebase="/data/syslog/rulebase.rb")
>>        action(type="omfile" file="/data/syslog/test-syslog.log"
>> template="testFormat") }
>>
>> rulebase.rb file:
>> rule=test:%date:word% %host:ipv4% :
>> %%ASA-%number0:number%-%number1:number%: Denied ICMP
>> type=%number2:number%, code=%number3:number% from %origin:ipv4% on
>> interface outside
>>
>> test-syslog.log file:
>> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1":
>> "313001", "number0": "3", "host": "0.0.0.0", "date":
>> "2013-06-26T10:47:42+01:00" }
>>
>> Message:
>> "2013-06-26T10:47:42+01:00 0.0.0.0 : %ASA-3-313001: Denied ICMP type=8, code=0 from 77.2.2.2 on interface outside"
>> ======================================================================
>>
>>
>> Thanks in advance for any help,
>>
>> Jose Castillo
>>
>>
>>
>>
>
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm

From david at lang.hm  Wed Jun 26 22:58:52 2013
From: david at lang.hm (David Lang)
Date: Wed, 26 Jun 2013 13:58:52 -0700 (PDT)
Subject: [Lognorm] log classification with liblognorm
In-Reply-To: <alpine.DEB.2.02.1306261351570.22572@nftneq.ynat.uz>
References: <701C784DE99B0C4C9FDA9EDE66E23E661E4CAD9300@HQ-MB-07.ba.ad.ssa.gov>
	<alpine.DEB.2.02.1306261210020.22572@nftneq.ynat.uz>
	<701C784DE99B0C4C9FDA9EDE66E23E661E4CAD930B@HQ-MB-07.ba.ad.ssa.gov>
	<alpine.DEB.2.02.1306261351570.22572@nftneq.ynat.uz>
Message-ID: <alpine.DEB.2.02.1306261357240.22572@nftneq.ynat.uz>

Ok, now I see what you are talking about. You are talking about the message 
classification tag, not the field tag. My mistake.

I don't know if and how rsyslog supports this.

David Lang

On Wed, 26 Jun 2013, David Lang wrote:

> First off, I have not used mmlognorm for anything other than seeing that it 
> works, so take that into consideration :-)
>
> However, as I'm looking at your rule, you have "mytag" outside of any 
> matching, so it would be treated as a literal, just like "Denied ICMP" 
> wouldn't it?
>
> in your syslog template, try putting %$!number2% and see if it outputs the 
> correct value. it looks to me like that variable is being set correctly.
>
> David Lang
>
> On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:
>
>> David,
>> 
>> I modified the template:
>> template(name="testFormat" type="string" 
>> string="%$!all-json%,tag='%$!mytag%'")
>> 
>> I also modified the rule to include mytag as a tag:
>> rule=mytag:%date:word% %host:ipv4% : 
>> %%ASA-%number0:number%-%number1:number%: Denied ICMP type=%number2:number%, 
>> code=%number3:number% from %origin:ipv4% on interface outside
>> 
>> And now I getting :
>> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1": 
>> "313001", "number0": "3", "host": "0.0.0.0", "date": 
>> "2013-06-26T10:47:42+01:00" },tag=''
>> 
>> No tags are being written to the output file.
>> 
>> What am I doing wrong?
>> 
>> Thanks,
>> 
>> Jose
>> 
>> -----Original Message-----
>> From: lognorm-bounces at lists.adiscon.com 
>> [mailto:lognorm-bounces at lists.adiscon.com] On Behalf Of David Lang
>> Sent: Wednesday, June 26, 2013 3:11 PM
>> To: lognorm
>> Subject: Re: [Lognorm] log classification with liblognorm
>> 
>> the tags are accessed in rsyslog by $!<tag name>
>> 
>> so where you would have done %hostname% you could do %$!mytag%
>> 
>> David Lang
>> 
>> On Wed, 26 Jun 2013, Castillo, Jose   Contractor wrote:
>> 
>>> Date: Wed, 26 Jun 2013 16:00:06 -0400
>>> From: "Castillo, Jose   Contractor" 
>>> <Jose.Castillo at ssa.gov<mailto:Jose.Castillo at ssa.gov>>
>>> Reply-To: lognorm 
>>> <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
>>> To: "lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>" 
>>> <lognorm at lists.adiscon.com<mailto:lognorm at lists.adiscon.com>>
>>> Subject: [Lognorm] log classification with liblognorm
>>> 
>>> Hello all,
>>> 
>>> I started to play using rsyslog/liblognorm a week ago, I'm testing them on 
>>> a CentOS 6.4 virtual machine, and next packages have been installed from 
>>> Adiscon repository:
>>> rsyslog-mysql-7.4.1-1.el6.x86_64
>>> rsyslog-mmjsonparse-7.4.1-1.el6.x86_64
>>> rsyslog-7.4.1-1.el6.x86_64
>>> rsyslog-mmnormalize-7.4.1-1.el6.x86_64
>>> rsyslog-udpspoof-7.4.1-1.el6.x86_64
>>> rsyslog-elasticsearch-7.4.1-1.el6.x86_64
>>> 
>>> I was reading next link, and now I'm trying to select subsets of messages 
>>> based on liblognorm tags.
>>> http://blog.gerhards.net/2011/04/log-classification-with-liblognorm.ht
>>> ml
>>> 
>>> So far, I was able to parse a message using mmnormalize module, and now I 
>>> want to create outputs based on the liblognorm tag(s).
>>> Is this option supported within rsyslog.conf? Any example how to do it?
>>> 
>>> Below my testing files:
>>> 
>>> =================================================================
>>> rsyslog.conf file:
>>> module (load="imudp")
>>> module (load="mmnormalize")
>>> module (load="mmjsonparse")
>>> 
>>> input(type="imudp" address="192.168.1.1" port="514" ruleset="test")
>>> 
>>> template(name="testFormat" type="string" string="%$!all-json%\n")
>>> 
>>> ruleset(name="test") {
>>>        action(type="mmnormalize" userawmsg="on" 
>>> rulebase="/data/syslog/rulebase.rb")
>>>        action(type="omfile" file="/data/syslog/test-syslog.log"
>>> template="testFormat") }
>>> 
>>> rulebase.rb file:
>>> rule=test:%date:word% %host:ipv4% :
>>> %%ASA-%number0:number%-%number1:number%: Denied ICMP
>>> type=%number2:number%, code=%number3:number% from %origin:ipv4% on
>>> interface outside
>>> 
>>> test-syslog.log file:
>>> { "origin": "77.2.2.2", "number3": "0", "number2": "8", "number1":
>>> "313001", "number0": "3", "host": "0.0.0.0", "date":
>>> "2013-06-26T10:47:42+01:00" }
>>> 
>>> Message:
>>> "2013-06-26T10:47:42+01:00 0.0.0.0 : %ASA-3-313001: Denied ICMP type=8, 
>>> code=0 from 77.2.2.2 on interface outside"
>>> ======================================================================
>>> 
>>> 
>>> Thanks in advance for any help,
>>> 
>>> Jose Castillo
>>> 
>>> 
>>> 
>>> 
>> 
>
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm
-------------- next part --------------
_______________________________________________
Lognorm mailing list
Lognorm at lists.adiscon.com
http://lists.adiscon.net/mailman/listinfo/lognorm