From rory at ooma.com  Wed Jun  1 01:13:56 2011
From: rory at ooma.com (Rory Toma)
Date: Tue, 31 May 2011 16:13:56 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>
Message-ID: <4DE57634.5030901@ooma.com>

I have updated to 5.8.1 and it still does not work.

On 5/18/11 10:31 PM, Rainer Gerhards wrote:
> This is a bug in older versions prior to 5.8.0.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>> Sent: Thursday, May 19, 2011 1:44 AM
>> To: rsyslog-users
>> Subject: [rsyslog] Question on host failover
>>
>> I have the following for my clients' rsyslog.conf files:
>>
>> $DefaultNetStreamDriverCAFile /etc/ca.pem
>>
>> $DefaultNetStreamDriver gtls
>> $ActionSendStreamDriverMode 1
>> $ActionSendStreamDriverAuthMode anon
>>
>> $ActionResumeInterval 29
>>
>> $WorkDirectory /var/log
>>
>> $ModLoad imuxsock
>> $SystemLogSocketName /var/log/log
>> $OptimizeForUniprocessor on
>>
>> *.* @@<machine>:110
>> $ActionExecOnlyWhenPreviousIsSuspended on
>> *.* @@<machine>:143
>> $ActionExecOnlyWhenPreviousIsSuspended off
>>
>>
>>
>> If I block port 110 from the client, I would expect that it would
>> failover to port 143. I am not seeing this. Do I have to do something
>> different here? It doesn't work if I remove the ActionExec directive,
>> either. In either case, it seems to keep sending to port 110, no matter
>> what. In this case,<machine>  is the same physical machine, with the
>> same IP address, it's just a different syslog receiver process.
>>
>> This is rsyslog-5.6.2
>>
>> thx
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From taotetek at gmail.com  Wed Jun  1 03:03:32 2011
From: taotetek at gmail.com (Brian Knox)
Date: Tue, 31 May 2011 21:03:32 -0400
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
	<BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>
Message-ID: <BANLkTimQi9yVWduNs+-k8oWy8q9_yJNWvQ@mail.gmail.com>

Christian -

Thank you so much for giving this a spin and for the patches - I'll pass
them on to our developer on the project this week.  I know he was working on
pub / sub config already so I know he'll be excited someone else saw the
utility in it and offered a patch.

Brian

On Tue, May 31, 2011 at 4:43 PM, Christian Brunner <chb at muc.de> wrote:

> This is great! I started an omzeromq module some time ago, but never
> really finished it.
>
> I've now managed to get it running with the current rsyslog master
> branch and have added a configuration option for the messaging pattern
> (please see the following patches).
>
> Christian
>
>
> 2011/5/27 Brian Knox <taotetek at gmail.com>:
> > If anyone has any interest - we released zeromq input and output plugins
> for
> > rsyslog today.  They are relatively new and there's work to do on them,
> but
> > we have them up and running in our lab.  We'd love other people to take a
> > look and provide us with feedback!
> >
> > Thanks
> >
> > https://github.com/aggregateknowledge/rsyslog-zeromq
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed Jun  1 07:23:42 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 07:23:42 +0200
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <BANLkTimQi9yVWduNs+-k8oWy8q9_yJNWvQ@mail.gmail.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com><BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>
	<BANLkTimQi9yVWduNs+-k8oWy8q9_yJNWvQ@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D90@GRFEXC.intern.adiscon.com>

Hi folks,

thanks for the continued effort. I just wanted to let you know that I am
currently redoing large parts of the config system. I expect an initial
release either later today or early next week (there is a public holiday
tomorrow and I'll be probably away for the weekend ;)).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Brian Knox
> Sent: Wednesday, June 01, 2011 3:04 AM
> To: chb at muc.de; rsyslog-users
> Subject: Re: [rsyslog] ZeroMQ input and output plugins
> 
> Christian -
> 
> Thank you so much for giving this a spin and for the patches - I'll
> pass
> them on to our developer on the project this week.  I know he was
> working on
> pub / sub config already so I know he'll be excited someone else saw
> the
> utility in it and offered a patch.
> 
> Brian
> 
> On Tue, May 31, 2011 at 4:43 PM, Christian Brunner <chb at muc.de> wrote:
> 
> > This is great! I started an omzeromq module some time ago, but never
> > really finished it.
> >
> > I've now managed to get it running with the current rsyslog master
> > branch and have added a configuration option for the messaging
> pattern
> > (please see the following patches).
> >
> > Christian
> >
> >
> > 2011/5/27 Brian Knox <taotetek at gmail.com>:
> > > If anyone has any interest - we released zeromq input and output
> plugins
> > for
> > > rsyslog today.  They are relatively new and there's work to do on
> them,
> > but
> > > we have them up and running in our lab.  We'd love other people to
> take a
> > > look and provide us with feedback!
> > >
> > > Thanks
> > >
> > > https://github.com/aggregateknowledge/rsyslog-zeromq
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun  1 08:24:42 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 08:24:42 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE57634.5030901@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>
	<4DE57634.5030901@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com>

Sorry, I was so focused on the bug (which existed anyway) that I did not
notice the config problem. You need to use this directive in a rule chain. So
this should work:

*.* @@<machine>:110
$ActionExecOnlyWhenPreviousIsSuspended on
& @@<machine>:143
$ActionExecOnlyWhenPreviousIsSuspended off

Note that the second filter has been replaced by an "&" which means that the
actions are chained (and using the same filter).

Please let me know if that solves the issue (note that on older v5 builds
this does NOT work due to the bug).

Rainer
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, June 01, 2011 1:14 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> I have updated to 5.8.1 and it still does not work.
> 
> On 5/18/11 10:31 PM, Rainer Gerhards wrote:
> > This is a bug in older versions prior to 5.8.0.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >> Sent: Thursday, May 19, 2011 1:44 AM
> >> To: rsyslog-users
> >> Subject: [rsyslog] Question on host failover
> >>
> >> I have the following for my clients' rsyslog.conf files:
> >>
> >> $DefaultNetStreamDriverCAFile /etc/ca.pem
> >>
> >> $DefaultNetStreamDriver gtls
> >> $ActionSendStreamDriverMode 1
> >> $ActionSendStreamDriverAuthMode anon
> >>
> >> $ActionResumeInterval 29
> >>
> >> $WorkDirectory /var/log
> >>
> >> $ModLoad imuxsock
> >> $SystemLogSocketName /var/log/log
> >> $OptimizeForUniprocessor on
> >>
> >> *.* @@<machine>:110
> >> $ActionExecOnlyWhenPreviousIsSuspended on
> >> *.* @@<machine>:143
> >> $ActionExecOnlyWhenPreviousIsSuspended off
> >>
> >>
> >>
> >> If I block port 110 from the client, I would expect that it would
> >> failover to port 143. I am not seeing this. Do I have to do something
> >> different here? It doesn't work if I remove the ActionExec directive,
> >> either. In either case, it seems to keep sending to port 110, no
> >> matter what. In this case,<machine>  is the same physical machine,
> >> with the same IP address, it's just a different syslog receiver process.
> >>
> >> This is rsyslog-5.6.2
> >>
> >> thx
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Wed Jun  1 08:48:52 2011
From: rory at ooma.com (Rory Toma)
Date: Tue, 31 May 2011 23:48:52 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com>
Message-ID: <4DE5E0D4.5050409@ooma.com>

That did not work, either. Is this behaviour compiled in by default, or 
is there a compile/config time flag that I need to set?

On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
> Sorry, I was so focused on the bug (which existed anyway) that I did not
> notice the config problem. You need to use this directive in a rule chain. So
> this should work:
>
> *.* @@<machine>:110
> $ActionExecOnlyWhenPreviousIsSuspended on
> &  @@<machine>:143
> $ActionExecOnlyWhenPreviousIsSuspended off
>
> Note that the second filter has been replaced by an "&" which means that the
> actions are chained (and using the same filter).
>
> Please let me know if that solves the issue (note that on older v5 builds
> this does NOT work due to the bug).
>
> Rainer
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>> Sent: Wednesday, June 01, 2011 1:14 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> I have updated to 5.8.1 and it still does not work.
>>
>> On 5/18/11 10:31 PM, Rainer Gerhards wrote:
>>> This is a bug in older versions prior to 5.8.0.
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>>>> Sent: Thursday, May 19, 2011 1:44 AM
>>>> To: rsyslog-users
>>>> Subject: [rsyslog] Question on host failover
>>>>
>>>> I have the following for my clients' rsyslog.conf files:
>>>>
>>>> $DefaultNetStreamDriverCAFile /etc/ca.pem
>>>>
>>>> $DefaultNetStreamDriver gtls
>>>> $ActionSendStreamDriverMode 1
>>>> $ActionSendStreamDriverAuthMode anon
>>>>
>>>> $ActionResumeInterval 29
>>>>
>>>> $WorkDirectory /var/log
>>>>
>>>> $ModLoad imuxsock
>>>> $SystemLogSocketName /var/log/log
>>>> $OptimizeForUniprocessor on
>>>>
>>>> *.* @@<machine>:110
>>>> $ActionExecOnlyWhenPreviousIsSuspended on
>>>> *.* @@<machine>:143
>>>> $ActionExecOnlyWhenPreviousIsSuspended off
>>>>
>>>>
>>>>
>>>> If I block port 110 from the client, I would expect that it would
>>>> failover to port 143. I am not seeing this. Do I have to do something
>>>> different here? It doesn't work if I remove the ActionExec directive,
>>>> either. In either case, it seems to keep sending to port 110, no
>>>> matter what. In this case,<machine>   is the same physical machine,
>>>> with the same IP address, it's just a different syslog receiver process.
>>>>
>>>> This is rsyslog-5.6.2
>>>>
>>>> thx
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun  1 08:50:12 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 08:50:12 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE5E0D4.5050409@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com>
	<4DE5E0D4.5050409@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D97@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, June 01, 2011 8:49 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> That did not work, either. Is this behaviour compiled in by default, or is
there
> a compile/config time flag that I need to set?

No, it's available by default. Please provide a debug log, which should
contain both the startup as well as a failover scenario.

Thanks,
Rainer
> 
> On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
> > Sorry, I was so focused on the bug (which existed anyway) that I did
> > not notice the config problem. You need to use this directive in a
> > rule chain. So this should work:
> >
> > *.* @@<machine>:110
> > $ActionExecOnlyWhenPreviousIsSuspended on &  @@<machine>:143
> > $ActionExecOnlyWhenPreviousIsSuspended off
> >
> > Note that the second filter has been replaced by an "&" which means
> > that the actions are chained (and using the same filter).
> >
> > Please let me know if that solves the issue (note that on older v5
> > builds this does NOT work due to the bug).
> >
> > Rainer
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >> Sent: Wednesday, June 01, 2011 1:14 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> I have updated to 5.8.1 and it still does not work.
> >>
> >> On 5/18/11 10:31 PM, Rainer Gerhards wrote:
> >>> This is a bug in older versions prior to 5.8.0.
> >>>
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >>>> Sent: Thursday, May 19, 2011 1:44 AM
> >>>> To: rsyslog-users
> >>>> Subject: [rsyslog] Question on host failover
> >>>>
> >>>> I have the following for my clients' rsyslog.conf files:
> >>>>
> >>>> $DefaultNetStreamDriverCAFile /etc/ca.pem
> >>>>
> >>>> $DefaultNetStreamDriver gtls
> >>>> $ActionSendStreamDriverMode 1
> >>>> $ActionSendStreamDriverAuthMode anon
> >>>>
> >>>> $ActionResumeInterval 29
> >>>>
> >>>> $WorkDirectory /var/log
> >>>>
> >>>> $ModLoad imuxsock
> >>>> $SystemLogSocketName /var/log/log
> >>>> $OptimizeForUniprocessor on
> >>>>
> >>>> *.* @@<machine>:110
> >>>> $ActionExecOnlyWhenPreviousIsSuspended on
> >>>> *.* @@<machine>:143
> >>>> $ActionExecOnlyWhenPreviousIsSuspended off
> >>>>
> >>>>
> >>>>
> >>>> If I block port 110 from the client, I would expect that it would
> >>>> failover to port 143. I am not seeing this. Do I have to do
> >>>> something different here? It doesn't work if I remove the
> >>>> ActionExec directive, either. In either case, it seems to keep sending
to
> port 110, no
> >>>> matter what. In this case,<machine>   is the same physical machine,
> >>>> with the same IP address, it's just a different syslog receiver
process.
> >>>>
> >>>> This is rsyslog-5.6.2
> >>>>
> >>>> thx
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Wed Jun  1 08:50:19 2011
From: rory at ooma.com (Rory Toma)
Date: Tue, 31 May 2011 23:50:19 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE5E0D4.5050409@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com>
	<4DE5E0D4.5050409@ooma.com>
Message-ID: <4DE5E12B.3060600@ooma.com>

I also don't recall if I mentioned this, but this is also on ARM.

On 5/31/2011 11:48 PM, Rory Toma wrote:
> That did not work, either. Is this behaviour compiled in by default, 
> or is there a compile/config time flag that I need to set?
>
> On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
>> Sorry, I was so focused on the bug (which existed anyway) that I did not
>> notice the config problem. You need to use this directive in a rule 
>> chain. So
>> this should work:
>>
>> *.* @@<machine>:110
>> $ActionExecOnlyWhenPreviousIsSuspended on
>> &  @@<machine>:143
>> $ActionExecOnlyWhenPreviousIsSuspended off
>>
>> Note that the second filter has been replaced by an "&" which means 
>> that the
>> actions are chained (and using the same filter).
>>
>> Please let me know if that solves the issue (note that on older v5 
>> builds
>> this does NOT work due to the bug).
>>
>> Rainer


From rgerhards at hq.adiscon.com  Wed Jun  1 08:51:32 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 08:51:32 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE5E12B.3060600@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>
	<4DE5E12B.3060600@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>

> I also don't recall if I mentioned this, but this is also on ARM.

I assume everything else works well? I am asking because I want to make sure
we do not have problems with atomic instructions replacements.

Rainer
> 
> On 5/31/2011 11:48 PM, Rory Toma wrote:
> > That did not work, either. Is this behaviour compiled in by default,
> > or is there a compile/config time flag that I need to set?
> >
> > On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
> >> Sorry, I was so focused on the bug (which existed anyway) that I did
> >> not notice the config problem. You need to use this directive in a
> >> rule chain. So this should work:
> >>
> >> *.* @@<machine>:110
> >> $ActionExecOnlyWhenPreviousIsSuspended on &  @@<machine>:143
> >> $ActionExecOnlyWhenPreviousIsSuspended off
> >>
> >> Note that the second filter has been replaced by an "&" which means
> >> that the actions are chained (and using the same filter).
> >>
> >> Please let me know if that solves the issue (note that on older v5
> >> builds this does NOT work due to the bug).
> >>
> >> Rainer
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Wed Jun  1 08:54:35 2011
From: rory at ooma.com (Rory Toma)
Date: Tue, 31 May 2011 23:54:35 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>
Message-ID: <4DE5E22B.9060001@ooma.com>

I need to check, but I believe that may be an issue. I'll have to check 
my kernel/toolchain, and the configure output. I know there was a time 
frame when stable rsyslog did not compile due to this issue, and then I 
assumed that it had been removed when it started to compile again. I'll 
check tomorrow.

thx

On 5/31/2011 11:51 PM, Rainer Gerhards wrote:
>> I also don't recall if I mentioned this, but this is also on ARM.
> I assume everything else works well? I am asking because I want to make sure
> we do not have problems with atomic instructions replacements.
>
> Rainer
>> On 5/31/2011 11:48 PM, Rory Toma wrote:
>>> That did not work, either. Is this behaviour compiled in by default,
>>> or is there a compile/config time flag that I need to set?
>>>
>>> On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
>>>> Sorry, I was so focused on the bug (which existed anyway) that I did
>>>> not notice the config problem. You need to use this directive in a
>>>> rule chain. So this should work:
>>>>
>>>> *.* @@<machine>:110
>>>> $ActionExecOnlyWhenPreviousIsSuspended on&   @@<machine>:143
>>>> $ActionExecOnlyWhenPreviousIsSuspended off
>>>>
>>>> Note that the second filter has been replaced by an "&" which means
>>>> that the actions are chained (and using the same filter).
>>>>
>>>> Please let me know if that solves the issue (note that on older v5
>>>> builds this does NOT work due to the bug).
>>>>
>>>> Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun  1 08:56:55 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 08:56:55 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE5E22B.9060001@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>
	<4DE5E22B.9060001@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, June 01, 2011 8:55 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> I need to check, but I believe that may be an issue. I'll have to check my
> kernel/toolchain, and the configure output. I know there was a time frame
> when stable rsyslog did not compile due to this issue, and then I assumed
> that it had been removed when it started to compile again. I'll check
> tomorrow.

Well, I and others have done quite some work on the atomics replacements, and
they should work and work reasonably efficient in the current builds.
However, this is not used very often, and this is why I immediately consider
this as a region to look at...

Rainer

> 
> thx
> 
> On 5/31/2011 11:51 PM, Rainer Gerhards wrote:
> >> I also don't recall if I mentioned this, but this is also on ARM.
> > I assume everything else works well? I am asking because I want to make
> sure
> > we do not have problems with atomic instructions replacements.
> >
> > Rainer
> >> On 5/31/2011 11:48 PM, Rory Toma wrote:
> >>> That did not work, either. Is this behaviour compiled in by default,
> >>> or is there a compile/config time flag that I need to set?
> >>>
> >>> On 5/31/2011 11:24 PM, Rainer Gerhards wrote:
> >>>> Sorry, I was so focused on the bug (which existed anyway) that I did
> >>>> not notice the config problem. You need to use this directive in a
> >>>> rule chain. So this should work:
> >>>>
> >>>> *.* @@<machine>:110
> >>>> $ActionExecOnlyWhenPreviousIsSuspended on&   @@<machine>:143
> >>>> $ActionExecOnlyWhenPreviousIsSuspended off
> >>>>
> >>>> Note that the second filter has been replaced by an "&" which means
> >>>> that the actions are chained (and using the same filter).
> >>>>
> >>>> Please let me know if that solves the issue (note that on older v5
> >>>> builds this does NOT work due to the bug).
> >>>>
> >>>> Rainer
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun  1 10:06:16 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 10:06:16 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>

Hi David, Christian,

I finally have found time to look into the provided links. This looks indeed
very simple from an rsyslog PoV. However, I get the feeling that I myself may
not be the best person to do the majority of work, as to develop the actual
OCF scripts access to a test cluster, and experience with it (!), seems to be
very beneficial. So I wonder if anyone of you would be interested in helping
to get this going (with the scripts becoming part of the regular rsyslog
release).

As far as I understand, I would need to implement some facility inside
rsyslog that can be used to check its health by the monitor script. Or would
it even be an alternative for the monitor script to just check if the rsyslog
process to be monitored is in the process list? 

Any comments, advise, collaboration is deeply appreciated.

Rainer
PS: just in case: tomorrow is a public holiday over here, and I may leave for
a long weekend. I still thought I get this effort kicked off... 

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Tuesday, May 24, 2011 8:13 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> take a look at
> 
> http://linux-ha.org/wiki/Resource_Agents
> 
> and
> 
> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
> 
> David Lang
> 
> On Tue, 24 May 2011, Rainer Gerhards wrote:
> 
> > Date: Tue, 24 May 2011 08:09:28 +0200
> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> > To: rsyslog-users <rsyslog at lists.adiscon.com>
> > Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >
> > Thx -- sounds interesting and probably not too much work to do...
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Tuesday, May 24, 2011 8:08 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>
> >> take a look at linux-ha
> >>
> >> It's a framework to manage HA (including active/active load sharing,
> >> quorums, etc)
> >>
> >> it extends the traditional init.d startup scripts to also include a
> >> 'status' call to tell if the service is active or not. the framework
> >> calls this service periodically and if the service fails, it does a
> >> failover.
> >> With the correct configuration (and software), it can do sub-second
> >> failover.
> >>
> >> David Lang
> >>
> >>
> >>   On
> >> Tue, 24 May 2011, Rainer Gerhards wrote:
> >>
> >>> David and all,
> >>>
> >>> are you aware of any high availability APIs that would enable
> >>> rsyslog
> >> to do
> >>> some kind of automatic failover in a cluster environment? I have
> >> never
> >>> specifically programmed for that and wonder if there are any options.
> >>>
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >>>> Sent: Tuesday, May 24, 2011 12:30 AM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>>>
> >>>> depending on how active your logging is, you could watch the logs
> >> and
> >>>> say
> >>>> that if you don't receive any logs for 1 min (or whatever time is
> >>>> approprate), somthing is wrong.
> >>>>
> >>>> you could also generate known UDP logs to yourself and alert if
> >>>> they don't show up.
> >>>>
> >>>> David Lang
> >>>>
> >>>>   On Mon, 23 May 2011, Christian Lete wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I have a small question,  I would need to setup an rsyslog
> >>>>> receiver/forwarder, listening on udp port, since some clients,
> >>>>> only support this option. I would need this service to be highly
> >>>>> available(I don't want to have two machines and having duplicated
> >>>>> information), but since this udp, I can't be for sure if the
> >> service
> >>>>> is running fine. What I thought is to indirectly check it, by
> >> having
> >>>>> another port listening on tcp and checking the tcp service, if the
> >>>>> service is not running on tcp I would assume the whole system is
> >> down
> >>>>> and would failover to the other instance of the cluster, that's
> >>>>> the only way I could think of, do you currently have another way?
> >>>>>
> >>>>>
> >>>>> thank you very much,
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Christian
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>>>
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From friedl at hq.adiscon.com  Wed Jun  1 10:44:41 2011
From: friedl at hq.adiscon.com (Florian Riedl)
Date: Wed, 1 Jun 2011 10:44:41 +0200
Subject: [rsyslog] rsyslog 6.3.0 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DA1@GRFEXC.intern.adiscon.com>

The release of rsyslog 6.3.0 introduces a new major feature. With this
release we introduce a new config system. You can find some information about
the new config system in Rainer's blog:

http://blog.gerhards.net/2011/06/new-rsyslog-config-system-materializes.html

ChangeLog:

http://www.rsyslog.com/changelog-for-6-3-0-devel/

Download:

http://www.rsyslog.com/rsyslog-6-3-0-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From christian.lete at gmail.com  Wed Jun  1 11:56:47 2011
From: christian.lete at gmail.com (Christian Lete)
Date: Wed, 1 Jun 2011 11:56:47 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com>
	<alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTinzamwTOikJU7pRYEtX1e3jm+Kvpg@mail.gmail.com>

Hi David and Rainer,

I have just started playing around with corosync + pacemaker + drbd
and so far have had a good impression of it. So the next week, I would
like to play around with some OCF scripts and see If I can create this
for rsyslog and would be glad to submit it to the project if this
works fine :)

Thanks a lot both for your help and hints on how to supervise this service.

Regards!

Christian

On 1 June 2011 10:06, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> Hi David, Christian,
>
> I finally have found time to look into the provided links. This looks indeed
> very simple from an rsyslog PoV. However, I get the feeling that I myself may
> not be the best person to do the majority of work, as to develop the actual
> OCF scripts access to a test cluster, and experience with it (!), seems to be
> very beneficial. So I wonder if anyone of you would be interested in helping
> to get this going (with the scripts becoming part of the regular rsyslog
> release).
>
> As far as I understand, I would need to implement some facility inside
> rsyslog that can be used to check its health by the monitor script. Or would
> it even be an alternative for the monitor script to just check if the rsyslog
> process to be monitored is in the process list?
>
> Any comments, advise, collaboration is deeply appreciated.
>
> Rainer
> PS: just in case: tomorrow is a public holiday over here, and I may leave for
> a long weekend. I still thought I get this effort kicked off...
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 24, 2011 8:13 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> take a look at
>>
>> http://linux-ha.org/wiki/Resource_Agents
>>
>> and
>>
>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>
>> David Lang
>>
>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>
>> > Date: Tue, 24 May 2011 08:09:28 +0200
>> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>> > To: rsyslog-users <rsyslog at lists.adiscon.com>
>> > Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >
>> > Thx -- sounds interesting and probably not too much work to do...
>> >
>> > Rainer
>> >
>> >> -----Original Message-----
>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> >> Sent: Tuesday, May 24, 2011 8:08 AM
>> >> To: rsyslog-users
>> >> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >>
>> >> take a look at linux-ha
>> >>
>> >> It's a framework to manage HA (including active/active load sharing,
>> >> quorums, etc)
>> >>
>> >> it extends the traditional init.d startup scripts to also include a
>> >> 'status' call to tell if the service is active or not. the framework
>> >> calls this service periodically and if the service fails, it does a
>> >> failover.
>> >> With the correct configuration (and software), it can do sub-second
>> >> failover.
>> >>
>> >> David Lang
>> >>
>> >>
>> >> ? On
>> >> Tue, 24 May 2011, Rainer Gerhards wrote:
>> >>
>> >>> David and all,
>> >>>
>> >>> are you aware of any high availability APIs that would enable
>> >>> rsyslog
>> >> to do
>> >>> some kind of automatic failover in a cluster environment? I have
>> >> never
>> >>> specifically programmed for that and wonder if there are any options.
>> >>>
>> >>> Rainer
>> >>>
>> >>>> -----Original Message-----
>> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> >>>> Sent: Tuesday, May 24, 2011 12:30 AM
>> >>>> To: rsyslog-users
>> >>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>> >>>>
>> >>>> depending on how active your logging is, you could watch the logs
>> >> and
>> >>>> say
>> >>>> that if you don't receive any logs for 1 min (or whatever time is
>> >>>> approprate), somthing is wrong.
>> >>>>
>> >>>> you could also generate known UDP logs to yourself and alert if
>> >>>> they don't show up.
>> >>>>
>> >>>> David Lang
>> >>>>
>> >>>> ? On Mon, 23 May 2011, Christian Lete wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I have a small question, ?I would need to setup an rsyslog
>> >>>>> receiver/forwarder, listening on udp port, since some clients,
>> >>>>> only support this option. I would need this service to be highly
>> >>>>> available(I don't want to have two machines and having duplicated
>> >>>>> information), but since this udp, I can't be for sure if the
>> >> service
>> >>>>> is running fine. What I thought is to indirectly check it, by
>> >> having
>> >>>>> another port listening on tcp and checking the tcp service, if the
>> >>>>> service is not running on tcp I would assume the whole system is
>> >> down
>> >>>>> and would failover to the other instance of the cluster, that's
>> >>>>> the only way I could think of, do you currently have another way?
>> >>>>>
>> >>>>>
>> >>>>> thank you very much,
>> >>>>>
>> >>>>> Regards,
>> >>>>>
>> >>>>> Christian
>> >>>>> _______________________________________________
>> >>>>> rsyslog mailing list
>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>> http://www.rsyslog.com
>> >>>>>
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com
>> >>>
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed Jun  1 11:59:52 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 11:59:52 +0200
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <BANLkTinzamwTOikJU7pRYEtX1e3jm+Kvpg@mail.gmail.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>
	<BANLkTinzamwTOikJU7pRYEtX1e3jm+Kvpg@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DA3@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Christian Lete
> Sent: Wednesday, June 01, 2011 11:57 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> 
> Hi David and Rainer,
> 
> I have just started playing around with corosync + pacemaker + drbd and so
> far have had a good impression of it. So the next week, I would like to
play
> around with some OCF scripts and see If I can create this for rsyslog and
> would be glad to submit it to the project if this works fine :)

Excellent. Just let me know if you need some help in form of a module or
such...

Rainer

> 
> Thanks a lot both for your help and hints on how to supervise this service.
> 
> Regards!
> 
> Christian
> 
> On 1 June 2011 10:06, Rainer Gerhards <rgerhards at hq.adiscon.com> wrote:
> > Hi David, Christian,
> >
> > I finally have found time to look into the provided links. This looks
> > indeed very simple from an rsyslog PoV. However, I get the feeling
> > that I myself may not be the best person to do the majority of work,
> > as to develop the actual OCF scripts access to a test cluster, and
> > experience with it (!), seems to be very beneficial. So I wonder if
> > anyone of you would be interested in helping to get this going (with
> > the scripts becoming part of the regular rsyslog release).
> >
> > As far as I understand, I would need to implement some facility inside
> > rsyslog that can be used to check its health by the monitor script. Or
> > would it even be an alternative for the monitor script to just check
> > if the rsyslog process to be monitored is in the process list?
> >
> > Any comments, advise, collaboration is deeply appreciated.
> >
> > Rainer
> > PS: just in case: tomorrow is a public holiday over here, and I may
> > leave for a long weekend. I still thought I get this effort kicked off...
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> Sent: Tuesday, May 24, 2011 8:13 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >>
> >> take a look at
> >>
> >> http://linux-ha.org/wiki/Resource_Agents
> >>
> >> and
> >>
> >> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
> >>
> >> David Lang
> >>
> >> On Tue, 24 May 2011, Rainer Gerhards wrote:
> >>
> >> > Date: Tue, 24 May 2011 08:09:28 +0200
> >> > From: Rainer Gerhards <rgerhards at hq.adiscon.com>
> >> > Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> > To: rsyslog-users <rsyslog at lists.adiscon.com>
> >> > Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >> >
> >> > Thx -- sounds interesting and probably not too much work to do...
> >> >
> >> > Rainer
> >> >
> >> >> -----Original Message-----
> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> >> Sent: Tuesday, May 24, 2011 8:08 AM
> >> >> To: rsyslog-users
> >> >> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >> >>
> >> >> take a look at linux-ha
> >> >>
> >> >> It's a framework to manage HA (including active/active load
> >> >> sharing, quorums, etc)
> >> >>
> >> >> it extends the traditional init.d startup scripts to also include
> >> >> a 'status' call to tell if the service is active or not. the
> >> >> framework calls this service periodically and if the service
> >> >> fails, it does a failover.
> >> >> With the correct configuration (and software), it can do
> >> >> sub-second failover.
> >> >>
> >> >> David Lang
> >> >>
> >> >>
> >> >> ? On
> >> >> Tue, 24 May 2011, Rainer Gerhards wrote:
> >> >>
> >> >>> David and all,
> >> >>>
> >> >>> are you aware of any high availability APIs that would enable
> >> >>> rsyslog
> >> >> to do
> >> >>> some kind of automatic failover in a cluster environment? I have
> >> >> never
> >> >>> specifically programmed for that and wonder if there are any
options.
> >> >>>
> >> >>> Rainer
> >> >>>
> >> >>>> -----Original Message-----
> >> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> >> >>>> Sent: Tuesday, May 24, 2011 12:30 AM
> >> >>>> To: rsyslog-users
> >> >>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
> >> >>>>
> >> >>>> depending on how active your logging is, you could watch the
> >> >>>> logs
> >> >> and
> >> >>>> say
> >> >>>> that if you don't receive any logs for 1 min (or whatever time
> >> >>>> is approprate), somthing is wrong.
> >> >>>>
> >> >>>> you could also generate known UDP logs to yourself and alert if
> >> >>>> they don't show up.
> >> >>>>
> >> >>>> David Lang
> >> >>>>
> >> >>>> ? On Mon, 23 May 2011, Christian Lete wrote:
> >> >>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> I have a small question, ?I would need to setup an rsyslog
> >> >>>>> receiver/forwarder, listening on udp port, since some clients,
> >> >>>>> only support this option. I would need this service to be
> >> >>>>> highly available(I don't want to have two machines and having
> >> >>>>> duplicated information), but since this udp, I can't be for
> >> >>>>> sure if the
> >> >> service
> >> >>>>> is running fine. What I thought is to indirectly check it, by
> >> >> having
> >> >>>>> another port listening on tcp and checking the tcp service, if
> >> >>>>> the service is not running on tcp I would assume the whole
> >> >>>>> system is
> >> >> down
> >> >>>>> and would failover to the other instance of the cluster, that's
> >> >>>>> the only way I could think of, do you currently have another way?
> >> >>>>>
> >> >>>>>
> >> >>>>> thank you very much,
> >> >>>>>
> >> >>>>> Regards,
> >> >>>>>
> >> >>>>> Christian
> >> >>>>> _______________________________________________
> >> >>>>> rsyslog mailing list
> >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>>>> http://www.rsyslog.com
> >> >>>>>
> >> >>>> _______________________________________________
> >> >>>> rsyslog mailing list
> >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>>> http://www.rsyslog.com
> >> >>> _______________________________________________
> >> >>> rsyslog mailing list
> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >>> http://www.rsyslog.com
> >> >>>
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> http://www.rsyslog.com
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From Ole.Rahn at t-systems.com  Wed Jun  1 14:54:15 2011
From: Ole.Rahn at t-systems.com (Ole.Rahn at t-systems.com)
Date: Wed, 1 Jun 2011 14:54:15 +0200
Subject: [rsyslog] version 4.4.2 and TLS
Message-ID: <50CCFFD1B9C6424389383F7A13A03446013F72421ED8@HE101451.emea1.cds.t-internal.com>

Dear all,

as far as I know, RHEL currently still comes with version 4.4.2 of rsyslog, which is a rather elderly version, even within the 4.x branch. We encounter a few problems related to this version and TLS, currently, and also it seems to me like this version was one of the first 4.x ones to support TLS.

That is why I would like to ask, if it is possible to build a secure (in terms of system stability and in terms of encryption via TLS) syslog architecture based on this version?

Best regards
Ole

From rgerhards at hq.adiscon.com  Wed Jun  1 15:06:33 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 1 Jun 2011 15:06:33 +0200
Subject: [rsyslog] version 4.4.2 and TLS
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DAE@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Ole.Rahn at t-systems.com
> Sent: Wednesday, June 01, 2011 2:54 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] version 4.4.2 and TLS
> 
> Dear all,
> 
> as far as I know, RHEL currently still comes with version 4.4.2 of rsyslog,
which
> is a rather elderly version, even within the 4.x branch. 

To the best of my knowledge, they ship some 3.x version, but I would be
delighted to hear it has been updated.

> We encounter a few
> problems related to this version and TLS, currently, and also it seems to
me
> like this version was one of the first 4.x ones to support TLS.
> 
> That is why I would like to ask, if it is possible to build a secure (in
terms of
> system stability and in terms of encryption via TLS) syslog architecture
based
> on this version?

I strongly recommend against this. The reason is that a very serious bug [1]
is present in this code base. It can lead to clients looking up and doing
nothing but loop (until restart). Looking at the ChangeLog, there are also a
number of other bugs which have been fixed since 4.4.2 was current (roughly
18 month ago, what is a very large time for rsyslog...). 

If you insist on using that version, probably the best thing to do is use
plain tcp syslog together with stunnel. But this has its own operational
drawbacks. The proper thing to do is to use a recent and supported version,
so that we can look and fix at any issues you may experience. 

HTH
Rainer

[1] http://bugzilla.adiscon.com/show_bug.cgi?id=194
> 
> Best regards
> Ole
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From taotetek at gmail.com  Wed Jun  1 19:36:33 2011
From: taotetek at gmail.com (Brian Knox)
Date: Wed, 1 Jun 2011 13:36:33 -0400
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D90@GRFEXC.intern.adiscon.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
	<BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>
	<BANLkTimQi9yVWduNs+-k8oWy8q9_yJNWvQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D90@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTi=asc1skvKRzTN6iaiXSmzyxdhWqQ@mail.gmail.com>

Ken is working on integrating the patches for the zeromq modules - he hopes
to have them integrated by tomorrow, at which point we're going to do some
high load testing of at least the omzeromq module.

Thanks for the feedback everyone!

Brian

On Wed, Jun 1, 2011 at 1:23 AM, Rainer Gerhards <rgerhards at hq.adiscon.com>wrote:

> Hi folks,
>
> thanks for the continued effort. I just wanted to let you know that I am
> currently redoing large parts of the config system. I expect an initial
> release either later today or early next week (there is a public holiday
> tomorrow and I'll be probably away for the weekend ;)).
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Brian Knox
> > Sent: Wednesday, June 01, 2011 3:04 AM
> > To: chb at muc.de; rsyslog-users
> > Subject: Re: [rsyslog] ZeroMQ input and output plugins
> >
> > Christian -
> >
> > Thank you so much for giving this a spin and for the patches - I'll
> > pass
> > them on to our developer on the project this week.  I know he was
> > working on
> > pub / sub config already so I know he'll be excited someone else saw
> > the
> > utility in it and offered a patch.
> >
> > Brian
> >
> > On Tue, May 31, 2011 at 4:43 PM, Christian Brunner <chb at muc.de> wrote:
> >
> > > This is great! I started an omzeromq module some time ago, but never
> > > really finished it.
> > >
> > > I've now managed to get it running with the current rsyslog master
> > > branch and have added a configuration option for the messaging
> > pattern
> > > (please see the following patches).
> > >
> > > Christian
> > >
> > >
> > > 2011/5/27 Brian Knox <taotetek at gmail.com>:
> > > > If anyone has any interest - we released zeromq input and output
> > plugins
> > > for
> > > > rsyslog today.  They are relatively new and there's work to do on
> > them,
> > > but
> > > > we have them up and running in our lab.  We'd love other people to
> > take a
> > > > look and provide us with feedback!
> > > >
> > > > Thanks
> > > >
> > > > https://github.com/aggregateknowledge/rsyslog-zeromq
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From david at lang.hm  Wed Jun  1 21:22:33 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 1 Jun 2011 12:22:33 -0700 (PDT)
Subject: [rsyslog] High availability on rsyslog (cluster)
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>
References: <BANLkTin88E7L-4cohhfP+6ihVSe2DvkbQg@mail.gmail.com><alpine.DEB.2.02.1105231529010.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D21@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1105232304290.8167@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280D22@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1105232312020.8167@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280D9F@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1106011156370.5766@asgard.lang.hm>

On Wed, 1 Jun 2011, Rainer Gerhards wrote:

> Hi David, Christian,
>
> I finally have found time to look into the provided links. This looks indeed
> very simple from an rsyslog PoV. However, I get the feeling that I myself may
> not be the best person to do the majority of work, as to develop the actual
> OCF scripts access to a test cluster, and experience with it (!), seems to be
> very beneficial. So I wonder if anyone of you would be interested in helping
> to get this going (with the scripts becoming part of the regular rsyslog
> release).

I've got the clusters to work with and would be happy to help

but you really don't need a cluster to test it (more below)

> As far as I understand, I would need to implement some facility inside
> rsyslog that can be used to check its health by the monitor script. Or would
> it even be an alternative for the monitor script to just check if the rsyslog
> process to be monitored is in the process list?

it depends how reliable you want the testing to be.

having the monitor check if rsyslog is in the process table is an easy 
first step (and is all that is done for many applications), but it has a 
couple of problems

1. if something goes wrong where rsyslog is up, but not processing 
messages (full buffer, full disk, etc) the cluster software will think 
that everything is fine.

2. you really want to have rsyslog running all the time, even when you 
aren't active so that this system can log.


having some way to ask rsyslog if everything is good would be very handy 
(and possibly provide the start of some interactive debugging tool???). it 
is not a requirement, but would be better.

it's all a matter of how far you want to go.


at a minimum it needs to implement start, stop, monitor, meta-data

A. meta-data is a pretty trivial thing (a series of echo statements to 
output 
some XML)

B. start and stop get a little more complicated since ryslog needs to be 
running even if a box is not active

when rsyslog is active, it means that it's running with the 'active' 
configuration, when it's inactive, it means that it's running with the 
'inactive' configuration

this can be done with no changes to rsyslog, simply by having two 
different .conf files and having the OCF script stop the current instance 
of rsyslog and start the correct one. If rsyslog has the ability to switch 
between two configs internally (which may be possible with the new config 
support), then this could be a signal of some sort to rsyslog.

C. monitor is where things get interesting

monitor needs to return one of three conditions, active (0), standby (7), 
or failed (anything else)

if start/stop are done by starting rsyslog with one of two different 
config files, the script can keep track of which config file it started 
rsyslog with and return the appropriate value. it can also check if 
rsyslog is running and if not return an error.

if start/stop are done by rsyslog internally, monitor needs to ask rsyslog 
which state it is in (or return an error state)


to test this, you don't need a cluster, all you need is to run the script 
to test the various modes. but you need to do it in the following order

start
start
monitor
stop
stop
monitor

this is to test to make sure that things don't get confused if it gets 
'started' or 'stopped' twice


the one other wrinkle in all of this is that when rsyslog starts at system 
boot time, you want it to start in the 'inactive' mode


In my case I have dedicated relay systems, and what I do is have rsyslog 
running on both boxes in the pair all the time. I then have an IP address 
that I move from one box to the other todo the failover. this works 99+% 
of the time, but in the very rare cases where rsyslog dies, this isn't 
detected.

David Lang

> Any comments, advise, collaboration is deeply appreciated.
>
> Rainer
> PS: just in case: tomorrow is a public holiday over here, and I may leave for
> a long weekend. I still thought I get this effort kicked off...
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Tuesday, May 24, 2011 8:13 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>
>> take a look at
>>
>> http://linux-ha.org/wiki/Resource_Agents
>>
>> and
>>
>> http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
>>
>> David Lang
>>
>> On Tue, 24 May 2011, Rainer Gerhards wrote:
>>
>>> Date: Tue, 24 May 2011 08:09:28 +0200
>>> From: Rainer Gerhards <rgerhards at hq.adiscon.com>
>>> Reply-To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> To: rsyslog-users <rsyslog at lists.adiscon.com>
>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>
>>> Thx -- sounds interesting and probably not too much work to do...
>>>
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>> Sent: Tuesday, May 24, 2011 8:08 AM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>
>>>> take a look at linux-ha
>>>>
>>>> It's a framework to manage HA (including active/active load sharing,
>>>> quorums, etc)
>>>>
>>>> it extends the traditional init.d startup scripts to also include a
>>>> 'status' call to tell if the service is active or not. the framework
>>>> calls this service periodically and if the service fails, it does a
>>>> failover.
>>>> With the correct configuration (and software), it can do sub-second
>>>> failover.
>>>>
>>>> David Lang
>>>>
>>>>
>>>>   On
>>>> Tue, 24 May 2011, Rainer Gerhards wrote:
>>>>
>>>>> David and all,
>>>>>
>>>>> are you aware of any high availability APIs that would enable
>>>>> rsyslog
>>>> to do
>>>>> some kind of automatic failover in a cluster environment? I have
>>>> never
>>>>> specifically programmed for that and wonder if there are any options.
>>>>>
>>>>> Rainer
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>>>>> Sent: Tuesday, May 24, 2011 12:30 AM
>>>>>> To: rsyslog-users
>>>>>> Subject: Re: [rsyslog] High availability on rsyslog (cluster)
>>>>>>
>>>>>> depending on how active your logging is, you could watch the logs
>>>> and
>>>>>> say
>>>>>> that if you don't receive any logs for 1 min (or whatever time is
>>>>>> approprate), somthing is wrong.
>>>>>>
>>>>>> you could also generate known UDP logs to yourself and alert if
>>>>>> they don't show up.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>   On Mon, 23 May 2011, Christian Lete wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a small question,  I would need to setup an rsyslog
>>>>>>> receiver/forwarder, listening on udp port, since some clients,
>>>>>>> only support this option. I would need this service to be highly
>>>>>>> available(I don't want to have two machines and having duplicated
>>>>>>> information), but since this udp, I can't be for sure if the
>>>> service
>>>>>>> is running fine. What I thought is to indirectly check it, by
>>>> having
>>>>>>> another port listening on tcp and checking the tcp service, if the
>>>>>>> service is not running on tcp I would assume the whole system is
>>>> down
>>>>>>> and would failover to the other instance of the cluster, that's
>>>>>>> the only way I could think of, do you currently have another way?
>>>>>>>
>>>>>>>
>>>>>>> thank you very much,
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Christian
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rory at ooma.com  Wed Jun  1 23:13:03 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 01 Jun 2011 14:13:03 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>
Message-ID: <4DE6AB5F.9040805@ooma.com>

On 5/31/11 11:56 PM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>> Sent: Wednesday, June 01, 2011 8:55 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> I need to check, but I believe that may be an issue. I'll have to check my
>> kernel/toolchain, and the configure output. I know there was a time frame
>> when stable rsyslog did not compile due to this issue, and then I assumed
>> that it had been removed when it started to compile again. I'll check
>> tomorrow.
> Well, I and others have done quite some work on the atomics replacements, and
> they should work and work reasonably efficient in the current builds.
> However, this is not used very often, and this is why I immediately consider
> this as a region to look at...
>
> Rainer
>
So, I run this:

<stuff>

*.* @@rsyslog:110
$ActionExecOnlyWhenPreviousIsSuspended on
& @@rsyslog:143
$ActionExecOnlyWhenPreviousIsSuspended off


in debug mode. I see a whole bunch of stuff at startup, and then 
nothing. Hopefully it's not trying to syslog the debug output... 8-) 
 From a tcpdump on my firewall, I see it trying to hit the first one, 
and not the second.

It looks like I do have atomic support in the compiler:

checking whether the compiler provides atomic builtins... (cached) yes
checking whether the compiler provides atomic builtins for 64 bit data 
types... no

I explicitly turned off the atomic builtins and no luck there, either.

Let me know what else you'd like me to try.

thx



From marcin at mejor.pl  Thu Jun  2 10:17:32 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Thu, 02 Jun 2011 10:17:32 +0200
Subject: [rsyslog] Rsyslog doesn't terminate after kill -15
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDED2@GRFEXC.intern.adiscon.com>
References: <4D8C91E4.3040304@mejor.pl><4D9321B2.2030304@mejor.pl>	<9B6E2A8877C38245BFB15CC491A11DA71DDECA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDED0@GRFEXC.intern.adiscon.com>	<4D933810.4080502@mejor.pl>
	<9B6E2A8877C38245BFB15CC491A11DA71DDED2@GRFEXC.intern.adiscon.com>
Message-ID: <4DE7471C.4030904@mejor.pl>

W dniu 30.03.2011 16:05, Rainer Gerhards pisze:
> Firewall seems to break tcp and hangs the connection. Fix firewall.

Hello!
As i wrote in other mail, this isn't problem with firewall. Is it
possible to restore old behavior of rsyslog? (without downgrading to
5.6.5 ;) )
Regards,
Marcin.

From rgerhards at hq.adiscon.com  Thu Jun  2 12:32:45 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 2 Jun 2011 12:32:45 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE6AB5F.9040805@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>
	<4DE6AB5F.9040805@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com>

Mmhhh.. can you post a complete debug log (maybe via a website like filebin)?

Thx,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, June 01, 2011 11:13 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> On 5/31/11 11:56 PM, Rainer Gerhards wrote:
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >> Sent: Wednesday, June 01, 2011 8:55 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> I need to check, but I believe that may be an issue. I'll have to
> check my
> >> kernel/toolchain, and the configure output. I know there was a time
> frame
> >> when stable rsyslog did not compile due to this issue, and then I
> assumed
> >> that it had been removed when it started to compile again. I'll
> check
> >> tomorrow.
> > Well, I and others have done quite some work on the atomics
> replacements, and
> > they should work and work reasonably efficient in the current builds.
> > However, this is not used very often, and this is why I immediately
> consider
> > this as a region to look at...
> >
> > Rainer
> >
> So, I run this:
> 
> <stuff>
> 
> *.* @@rsyslog:110
> $ActionExecOnlyWhenPreviousIsSuspended on
> & @@rsyslog:143
> $ActionExecOnlyWhenPreviousIsSuspended off
> 
> 
> in debug mode. I see a whole bunch of stuff at startup, and then
> nothing. Hopefully it's not trying to syslog the debug output... 8-)
>  From a tcpdump on my firewall, I see it trying to hit the first one,
> and not the second.
> 
> It looks like I do have atomic support in the compiler:
> 
> checking whether the compiler provides atomic builtins... (cached) yes
> checking whether the compiler provides atomic builtins for 64 bit data
> types... no
> 
> I explicitly turned off the atomic builtins and no luck there, either.
> 
> Let me know what else you'd like me to try.
> 
> thx
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rory at ooma.com  Thu Jun  2 23:19:56 2011
From: rory at ooma.com (Rory Toma)
Date: Thu, 02 Jun 2011 14:19:56 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com>
Message-ID: <4DE7FE7C.3020604@ooma.com>

On 6/2/11 3:32 AM, Rainer Gerhards wrote:
> Mmhhh.. can you post a complete debug log (maybe via a website like filebin)?
>
> Thx,
> Rainer
>
Let me know when you've downloaded this file and I'll reset the ACL, thx.

http://www.colinburns.com/downloads/darwin0.bz2

From avishai at fewbytes.com  Fri Jun  3 00:51:01 2011
From: avishai at fewbytes.com (Avishai Ish-Shalom)
Date: Fri, 03 Jun 2011 01:51:01 +0300
Subject: [rsyslog] Rsyslog remote/local splitting
Message-ID: <4DE813D5.1090500@fewbytes.com>

Hi all.

As i'm using RELP, i'm unable to utilize RuleSets. I've tried to achieve
remote/local splitting by grouping my rules for remote originating
messages in a separate file and do something like

Include /etc/rsyslog.d/remote.conf

# BSD style block

+@

*.* /varl/log/syslog

....


this failed miserably, as did:

Include /etc/rsyslog.d/remote.conf

if fromhost-ip != "127.0.0.1" then ~

*.* /varl/log/syslog

....

Any ideas? Am i doing something wrong?

-- 
Regards,
Avishai



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6841 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110603/cc459f73/attachment.bin>

From taotetek at gmail.com  Fri Jun  3 12:26:06 2011
From: taotetek at gmail.com (Brian Knox)
Date: Fri, 3 Jun 2011 06:26:06 -0400
Subject: [rsyslog] ZeroMQ input and output plugins
In-Reply-To: <BANLkTi=asc1skvKRzTN6iaiXSmzyxdhWqQ@mail.gmail.com>
References: <BANLkTimgLOSsfXuvNAyS+8SHOH_UcbC6rA@mail.gmail.com>
	<BANLkTi=d5cpZm4Z+wF-Jz0Tpy=vd1o0MOw@mail.gmail.com>
	<BANLkTimQi9yVWduNs+-k8oWy8q9_yJNWvQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280D90@GRFEXC.intern.adiscon.com>
	<BANLkTi=asc1skvKRzTN6iaiXSmzyxdhWqQ@mail.gmail.com>
Message-ID: <BANLkTi=82JWfnrXwHrFCa67c1KBheQNXjg@mail.gmail.com>

Just a heads up that Ken has integrated some of the patches:

https://github.com/aggregateknowledge/rsyslog-zeromq/commits/master

We did some more testing yesterday (of the output plugin) - I sent a several
million events through a chain of rsyslog servers, the final end point being
a zmq push socket feeding a pool of workers that parsed the messages -
everything worked great.  I hope to spend some time this weekend doing a
quick write up.  Pub / sub should be coming soon!

Brian

On Wed, Jun 1, 2011 at 1:36 PM, Brian Knox <taotetek at gmail.com> wrote:

> Ken is working on integrating the patches for the zeromq modules - he hopes
> to have them integrated by tomorrow, at which point we're going to do some
> high load testing of at least the omzeromq module.
>
> Thanks for the feedback everyone!
>
> Brian
>
>
> On Wed, Jun 1, 2011 at 1:23 AM, Rainer Gerhards <rgerhards at hq.adiscon.com>wrote:
>
>> Hi folks,
>>
>> thanks for the continued effort. I just wanted to let you know that I am
>> currently redoing large parts of the config system. I expect an initial
>> release either later today or early next week (there is a public holiday
>> tomorrow and I'll be probably away for the weekend ;)).
>>
>> Rainer
>>
>> > -----Original Message-----
>> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> > bounces at lists.adiscon.com] On Behalf Of Brian Knox
>> > Sent: Wednesday, June 01, 2011 3:04 AM
>> > To: chb at muc.de; rsyslog-users
>> > Subject: Re: [rsyslog] ZeroMQ input and output plugins
>> >
>> > Christian -
>> >
>> > Thank you so much for giving this a spin and for the patches - I'll
>> > pass
>> > them on to our developer on the project this week.  I know he was
>> > working on
>> > pub / sub config already so I know he'll be excited someone else saw
>> > the
>> > utility in it and offered a patch.
>> >
>> > Brian
>> >
>> > On Tue, May 31, 2011 at 4:43 PM, Christian Brunner <chb at muc.de> wrote:
>> >
>> > > This is great! I started an omzeromq module some time ago, but never
>> > > really finished it.
>> > >
>> > > I've now managed to get it running with the current rsyslog master
>> > > branch and have added a configuration option for the messaging
>> > pattern
>> > > (please see the following patches).
>> > >
>> > > Christian
>> > >
>> > >
>> > > 2011/5/27 Brian Knox <taotetek at gmail.com>:
>> > > > If anyone has any interest - we released zeromq input and output
>> > plugins
>> > > for
>> > > > rsyslog today.  They are relatively new and there's work to do on
>> > them,
>> > > but
>> > > > we have them up and running in our lab.  We'd love other people to
>> > take a
>> > > > look and provide us with feedback!
>> > > >
>> > > > Thanks
>> > > >
>> > > > https://github.com/aggregateknowledge/rsyslog-zeromq
>> > > > _______________________________________________
>> > > > rsyslog mailing list
>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > http://www.rsyslog.com
>> > > >
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com
>> > >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>
>

From friedl at hq.adiscon.com  Tue Jun  7 11:33:36 2011
From: friedl at hq.adiscon.com (Florian Riedl)
Date: Tue, 7 Jun 2011 11:33:36 +0200
Subject: [rsyslog] rsyslog 6.3.1 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DE3@GRFEXC.intern.adiscon.com>

Dear rsyslog mailing list members,

We have just released the new rsyslog 6.3.1 (devel). This is a first
implementation of a full-blown DNS cache. While there were some optimizations
for DNS queries in older releases, especially UDP sometimes suffered under
slow DNS resolution performance. 
This is solved with the new dnscache module. Note that the module will
undergo some more enhancements in the next couple of weeks. Feedback on its
effect would be deeply appreciated.
ChangeLog:

http://www.rsyslog.com/changelog-for-6-3-1-devel/

Download:

http://www.rsyslog.com/rsyslog-6-3-1devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From rgerhards at hq.adiscon.com  Tue Jun  7 11:55:59 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 7 Jun 2011 11:55:59 +0200
Subject: [rsyslog] new rsyslog DNS cache
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DE5@GRFEXC.intern.adiscon.com>

Hi all,

a little more background on the new dns cache module (used for reverse
resolution, just to mention ;)):

http://blog.gerhards.net/2011/06/full-blown-dns-cache-for-rsyslog.html

I'd really appreciate some practical feedback, especially from those folks
who use a caching resolver to circumvent the "UDP issue". IMHO the caching
resolver could now be removed. Even with it, I would expect somewhat better
performance. But I have no proof and the current code may even be somewhat
slower. This I would like to know.

Rainer

From kaiwang.chen at gmail.com  Tue Jun  7 16:03:53 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Tue, 7 Jun 2011 22:03:53 +0800
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't conform
 to table type monitorware in loganalyzer-3.2.1 ?
Message-ID: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>

Hello,

In Step 7 of installation process, "Create the first source for syslog
messages", selecting Table type: MonitorWare (the other is SyslogNG)
would load $dbmapping['mnoitorware'] in
include/constants_logstream.php, resulting in SQL like this:

SELECT id, devicereportedtime, facility, priority, fromhost,
syslogtag, processid, infounitid, message FROM SystemEvents ORDER BY
id DESC LIMIT 100

In the case of syslog, the fields are mapped from
./include/functions_config.php:

 501         $CFG['Views']['SYSLOG']= array(
 502
      'ID' =>                 "SYSLOG",
 503
      'DisplayName' =>"Syslog Fields",
 504
      'Columns' =>    array ( SYSLOG_DATE, SYSLOG_FACILITY, S
     YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG, SYSLOG_PROCESSID,
SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
 505
      'userid' =>             null,
 506
      'groupid' =>    null,
 507                                                                    );

Columns array:
            [0] => timereported
            [1] => syslogfacility
            [2] => syslogseverity
            [3] => FROMHOST
            [4] => syslogtag
            [5] => procid
            [6] => IUT
            [7] => msg


Finally, I got a error prompt like this:

No syslog records found - Error Details:

No syslog records found


The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
processid filed is missing)

CREATE TABLE SystemEvents
(
        ID int unsigned not null auto_increment primary key,
        CustomerID bigint,
        ReceivedAt datetime NULL,
        DeviceReportedTime datetime NULL,
        Facility smallint NULL,
        Priority smallint NULL,
        FromHost varchar(60) NULL,
        Message text,
        NTSeverity int NULL,
        Importance int NULL,
        EventSource varchar(60),
        EventUser varchar(60) NULL,
        EventCategory int NULL,
        EventID int NULL,
        EventBinaryData text NULL,
        MaxAvailable int NULL,
        CurrUsage int NULL,
        MinUsage int NULL,
        MaxUsage int NULL,
        InfoUnitID int NULL ,
        SysLogTag varchar(60),
        EventLogType varchar(60),
        GenericFileName VarChar(60),
        SystemID int NULL
);


So, what should I do? I heard of monitorware schema, and assumed it to
be what shipped with rsyslog.


Thanks,
Kaiwang

From rgerhards at hq.adiscon.com  Tue Jun  7 16:50:36 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 7 Jun 2011 16:50:36 +0200
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
	conform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DF5@GRFEXC.intern.adiscon.com>

I just asked Andre, the author or Adiscon LogAnalyzer (Formerly phpLogCon) on
this. I don't have information on that field.

RAiner

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Tuesday, June 07, 2011 4:04 PM
> To: rsyslog-users
> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
> conform to table type monitorware in loganalyzer-3.2.1 ?
> 
> Hello,
> 
> In Step 7 of installation process, "Create the first source for syslog
> messages", selecting Table type: MonitorWare (the other is SyslogNG)
> would load $dbmapping['mnoitorware'] in
> include/constants_logstream.php, resulting in SQL like this:
> 
> SELECT id, devicereportedtime, facility, priority, fromhost,
> syslogtag, processid, infounitid, message FROM SystemEvents ORDER BY
> id DESC LIMIT 100
> 
> In the case of syslog, the fields are mapped from
> ./include/functions_config.php:
> 
>  501         $CFG['Views']['SYSLOG']= array(
>  502
>       'ID' =>                 "SYSLOG",
>  503
>       'DisplayName' =>"Syslog Fields",
>  504
>       'Columns' =>    array ( SYSLOG_DATE, SYSLOG_FACILITY, S
>      YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG, SYSLOG_PROCESSID,
> SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
>  505
>       'userid' =>             null,
>  506
>       'groupid' =>    null,
>  507
> );
> 
> Columns array:
>             [0] => timereported
>             [1] => syslogfacility
>             [2] => syslogseverity
>             [3] => FROMHOST
>             [4] => syslogtag
>             [5] => procid
>             [6] => IUT
>             [7] => msg
> 
> 
> Finally, I got a error prompt like this:
> 
> No syslog records found - Error Details:
> 
> No syslog records found
> 
> 
> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
> processid filed is missing)
> 
> CREATE TABLE SystemEvents
> (
>         ID int unsigned not null auto_increment primary key,
>         CustomerID bigint,
>         ReceivedAt datetime NULL,
>         DeviceReportedTime datetime NULL,
>         Facility smallint NULL,
>         Priority smallint NULL,
>         FromHost varchar(60) NULL,
>         Message text,
>         NTSeverity int NULL,
>         Importance int NULL,
>         EventSource varchar(60),
>         EventUser varchar(60) NULL,
>         EventCategory int NULL,
>         EventID int NULL,
>         EventBinaryData text NULL,
>         MaxAvailable int NULL,
>         CurrUsage int NULL,
>         MinUsage int NULL,
>         MaxUsage int NULL,
>         InfoUnitID int NULL ,
>         SysLogTag varchar(60),
>         EventLogType varchar(60),
>         GenericFileName VarChar(60),
>         SystemID int NULL
> );
> 
> 
> So, what should I do? I heard of monitorware schema, and assumed it to
> be what shipped with rsyslog.
> 
> 
> Thanks,
> Kaiwang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From alorbach at ro1.adiscon.com  Tue Jun  7 17:28:30 2011
From: alorbach at ro1.adiscon.com (Andre Lorbach)
Date: Tue, 7 Jun 2011 17:28:30 +0200
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
	conform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
Message-ID: <E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com>

Hi,

the ProcessID field was added for LogAnalyzer. It wasn't in MonitorWare
either. 
But LogAnalyzer will automatically add missing fields into the logstream
databases, if the database user has sufficient rights to the table. So
granting the database user sufficient rights would solve the problem for now.


Apparently adding this field into the default database schema of MonitorWare
and RSyslog was lost in communication somewhere. 

Best regards,
Andre Lorbach

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Dienstag, 7. Juni 2011 16:04
> To: rsyslog-users
> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't conform
to
> table type monitorware in loganalyzer-3.2.1 ?
> 
> Hello,
> 
> In Step 7 of installation process, "Create the first source for syslog
> messages", selecting Table type: MonitorWare (the other is SyslogNG) would
> load $dbmapping['mnoitorware'] in include/constants_logstream.php,
> resulting in SQL like this:
> 
> SELECT id, devicereportedtime, facility, priority, fromhost, syslogtag,
> processid, infounitid, message FROM SystemEvents ORDER BY id DESC LIMIT
> 100
> 
> In the case of syslog, the fields are mapped from
> ./include/functions_config.php:
> 
>  501         $CFG['Views']['SYSLOG']= array(
>  502
>       'ID' =>                 "SYSLOG",
>  503
>       'DisplayName' =>"Syslog Fields",
>  504
>       'Columns' =>    array ( SYSLOG_DATE, SYSLOG_FACILITY, S
>      YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
> SYSLOG_PROCESSID, SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
>  505
>       'userid' =>             null,
>  506
>       'groupid' =>    null,
>  507                                                                    );
> 
> Columns array:
>             [0] => timereported
>             [1] => syslogfacility
>             [2] => syslogseverity
>             [3] => FROMHOST
>             [4] => syslogtag
>             [5] => procid
>             [6] => IUT
>             [7] => msg
> 
> 
> Finally, I got a error prompt like this:
> 
> No syslog records found - Error Details:
> 
> No syslog records found
> 
> 
> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the processid
> filed is missing)
> 
> CREATE TABLE SystemEvents
> (
>         ID int unsigned not null auto_increment primary key,
>         CustomerID bigint,
>         ReceivedAt datetime NULL,
>         DeviceReportedTime datetime NULL,
>         Facility smallint NULL,
>         Priority smallint NULL,
>         FromHost varchar(60) NULL,
>         Message text,
>         NTSeverity int NULL,
>         Importance int NULL,
>         EventSource varchar(60),
>         EventUser varchar(60) NULL,
>         EventCategory int NULL,
>         EventID int NULL,
>         EventBinaryData text NULL,
>         MaxAvailable int NULL,
>         CurrUsage int NULL,
>         MinUsage int NULL,
>         MaxUsage int NULL,
>         InfoUnitID int NULL ,
>         SysLogTag varchar(60),
>         EventLogType varchar(60),
>         GenericFileName VarChar(60),
>         SystemID int NULL
> );
> 
> 
> So, what should I do? I heard of monitorware schema, and assumed it to be
> what shipped with rsyslog.
> 
> 
> Thanks,
> Kaiwang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From kaiwang.chen at gmail.com  Tue Jun  7 19:34:16 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Wed, 8 Jun 2011 01:34:16 +0800
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
 conform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
	<E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com>
Message-ID: <BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com>

Will the default output template of rsyslog fill the new procid field?
Looks like leaving it NULL should work as well.

Thanks,
Kaiwang

2011/6/7 Andre Lorbach <alorbach at ro1.adiscon.com>:
> Hi,
>
> the ProcessID field was added for LogAnalyzer. It wasn't in MonitorWare
> either.
> But LogAnalyzer will automatically add missing fields into the logstream
> databases, if the database user has sufficient rights to the table. So
> granting the database user sufficient rights would solve the problem for now.
>
>
> Apparently adding this field into the default database schema of MonitorWare
> and RSyslog was lost in communication somewhere.
>
> Best regards,
> Andre Lorbach
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Dienstag, 7. Juni 2011 16:04
>> To: rsyslog-users
>> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't conform
> to
>> table type monitorware in loganalyzer-3.2.1 ?
>>
>> Hello,
>>
>> In Step 7 of installation process, "Create the first source for syslog
>> messages", selecting Table type: MonitorWare (the other is SyslogNG) would
>> load $dbmapping['mnoitorware'] in include/constants_logstream.php,
>> resulting in SQL like this:
>>
>> SELECT id, devicereportedtime, facility, priority, fromhost, syslogtag,
>> processid, infounitid, message FROM SystemEvents ORDER BY id DESC LIMIT
>> 100
>>
>> In the case of syslog, the fields are mapped from
>> ./include/functions_config.php:
>>
>> ?501 ? ? ? ? $CFG['Views']['SYSLOG']= array(
>> ?502
>> ? ? ? 'ID' => ? ? ? ? ? ? ? ? "SYSLOG",
>> ?503
>> ? ? ? 'DisplayName' =>"Syslog Fields",
>> ?504
>> ? ? ? 'Columns' => ? ?array ( SYSLOG_DATE, SYSLOG_FACILITY, S
>> ? ? ?YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
>> SYSLOG_PROCESSID, SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
>> ?505
>> ? ? ? 'userid' => ? ? ? ? ? ? null,
>> ?506
>> ? ? ? 'groupid' => ? ?null,
>> ?507 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?);
>>
>> Columns array:
>> ? ? ? ? ? ? [0] => timereported
>> ? ? ? ? ? ? [1] => syslogfacility
>> ? ? ? ? ? ? [2] => syslogseverity
>> ? ? ? ? ? ? [3] => FROMHOST
>> ? ? ? ? ? ? [4] => syslogtag
>> ? ? ? ? ? ? [5] => procid
>> ? ? ? ? ? ? [6] => IUT
>> ? ? ? ? ? ? [7] => msg
>>
>>
>> Finally, I got a error prompt like this:
>>
>> No syslog records found - Error Details:
>>
>> No syslog records found
>>
>>
>> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the processid
>> filed is missing)
>>
>> CREATE TABLE SystemEvents
>> (
>> ? ? ? ? ID int unsigned not null auto_increment primary key,
>> ? ? ? ? CustomerID bigint,
>> ? ? ? ? ReceivedAt datetime NULL,
>> ? ? ? ? DeviceReportedTime datetime NULL,
>> ? ? ? ? Facility smallint NULL,
>> ? ? ? ? Priority smallint NULL,
>> ? ? ? ? FromHost varchar(60) NULL,
>> ? ? ? ? Message text,
>> ? ? ? ? NTSeverity int NULL,
>> ? ? ? ? Importance int NULL,
>> ? ? ? ? EventSource varchar(60),
>> ? ? ? ? EventUser varchar(60) NULL,
>> ? ? ? ? EventCategory int NULL,
>> ? ? ? ? EventID int NULL,
>> ? ? ? ? EventBinaryData text NULL,
>> ? ? ? ? MaxAvailable int NULL,
>> ? ? ? ? CurrUsage int NULL,
>> ? ? ? ? MinUsage int NULL,
>> ? ? ? ? MaxUsage int NULL,
>> ? ? ? ? InfoUnitID int NULL ,
>> ? ? ? ? SysLogTag varchar(60),
>> ? ? ? ? EventLogType varchar(60),
>> ? ? ? ? GenericFileName VarChar(60),
>> ? ? ? ? SystemID int NULL
>> );
>>
>>
>> So, what should I do? I heard of monitorware schema, and assumed it to be
>> what shipped with rsyslog.
>>
>>
>> Thanks,
>> Kaiwang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Tue Jun  7 19:42:20 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Wed, 8 Jun 2011 01:42:20 +0800
Subject: [rsyslog] patch against loganalyzer-3.2.1 supporting customized
 port for UserDB and SourceDB
Message-ID: <BANLkTimn3WPZ5Loz8YAKL8_BhPVpNoYtEQ@mail.gmail.com>

Hello,

In the case of MySQL instances listening on non-standard port, the
following patch should be helpful. I didn't touch PDO SourceType for
config.php, since lack of knowledge.


diff -Nur loganalyzer-3.2.1.orig/src/classes/logstreamdb.class.php
loganalyzer-3.2.1/src/classes/logstreamdb.class.php
--- loganalyzer-3.2.1.orig/src/classes/logstreamdb.class.php	2011-04-12
22:02:08.000000000 +0800
+++ loganalyzer-3.2.1/src/classes/logstreamdb.class.php	2011-06-08
01:15:37.482080785 +0800
@@ -135,7 +135,7 @@
 		if ( $this->_dbhandle == null )
 		{
 			// Forces to open a new link in all cases!
-			$this->_dbhandle =
@mysql_connect($this->_logStreamConfigObj->DBServer,$this->_logStreamConfigObj->DBUser,$this->_logStreamConfigObj->DBPassword,
true);
+			$this->_dbhandle =
@mysql_connect($this->_logStreamConfigObj->DBServer . ":" .
$this->_logStreamConfigObj->DBPort,$this->_logStreamConfigObj->DBUser,$this->_logStreamConfigObj->DBPassword,
true);
 			if (!$this->_dbhandle)
 			{
 				if ( isset($php_errormsg) )
diff -Nur loganalyzer-3.2.1.orig/src/include/db_template.txt
loganalyzer-3.2.1/src/include/db_template.txt
--- loganalyzer-3.2.1.orig/src/include/db_template.txt	2011-04-12
22:02:08.000000000 +0800
+++ loganalyzer-3.2.1/src/include/db_template.txt	2011-06-08
01:15:37.482080785 +0800
@@ -75,6 +75,7 @@
   `DBTableType` varchar(64) default NULL,
   `DBType` tinyint(4) default NULL,
   `DBServer` varchar(255) default NULL,
+  `DBPort` smallint unsigned NOT NULL default 3306,
   `DBName` varchar(64) default NULL,
   `DBUser` varchar(64) default NULL,
   `DBPassword` varchar(255) default NULL,
diff -Nur loganalyzer-3.2.1.orig/src/include/functions_installhelpers.php
loganalyzer-3.2.1/src/include/functions_installhelpers.php
--- loganalyzer-3.2.1.orig/src/include/functions_installhelpers.php	2011-04-12
22:02:08.000000000 +0800
+++ loganalyzer-3.2.1/src/include/functions_installhelpers.php	2011-06-08
01:15:37.488746592 +0800
@@ -214,7 +214,7 @@
 				$mySource['DBType'] = DB_MYSQL;

 			// Perform the insert
-			$result = DB_Query("INSERT INTO  " . DB_SOURCES . " (Name,
Description, SourceType, MsgParserList, MsgNormalize, ViewID,
DBTableType, DBType, DBServer, DBName, DBUser, DBPassword,
DBTableName, DBEnableRowCounting) VALUES ( " .
+			$result = DB_Query("INSERT INTO  " . DB_SOURCES . " (Name,
Description, SourceType, MsgParserList, MsgNormalize, ViewID,
DBTableType, DBType, DBServer, DBPort, DBName, DBUser, DBPassword,
DBTableName, DBEnableRowCounting) VALUES ( " .
 				"'" . PrepareValueForDB($mySource['Name']) . "', " .
 				"'" . PrepareValueForDB($mySource['Description']) . "', " .
 				" " . PrepareValueForDB($mySource['SourceType']) . " , " .
@@ -224,6 +224,7 @@
 				"'" . PrepareValueForDB($mySource['DBTableType']) . "', " .
 				" " . PrepareValueForDB($mySource['DBType']) . " , " .
 				"'" . PrepareValueForDB($mySource['DBServer']) . "', " .
+				" " . PrepareValueForDB($mySource['DBPort']) . " , " .
 				"'" . PrepareValueForDB($mySource['DBName']) . "', " .
 				"'" . PrepareValueForDB($mySource['DBUser']) . "', " .
 				"'" . PrepareValueForDB($mySource['DBPassword']) . "', " .
diff -Nur loganalyzer-3.2.1.orig/src/install.php
loganalyzer-3.2.1/src/install.php
--- loganalyzer-3.2.1.orig/src/install.php	2011-04-12 22:02:08.000000000 +0800
+++ loganalyzer-3.2.1/src/install.php	2011-06-08 01:15:37.492079498 +0800
@@ -309,7 +309,7 @@
 			

 			// Now Check database connect
-			$link_id = mysql_connect( $_SESSION['UserDBServer'],
$_SESSION['UserDBUser'], $_SESSION['UserDBPass']);
+			$link_id = mysql_connect( $_SESSION['UserDBServer'] . ":" .
$_SESSION['UserDBPort'], $_SESSION['UserDBUser'],
$_SESSION['UserDBPass']);
 			if (!$link_id)
 				RevertOneStep( $content['INSTALL_STEP']-1, GetAndReplaceLangStr(
$content['LN_INSTALL_ERRORCONNECTFAILED'], $_SESSION['UserDBServer'])
. "<br>" . DB_ReturnSimpleErrorMsg() );
 			
@@ -537,6 +537,7 @@

 	if ( isset($_SESSION['SourceDBName']) ) { $content['SourceDBName'] =
$_SESSION['SourceDBName']; } else { $content['SourceDBName'] =
"loganalyzer"; }
 	if ( isset($_SESSION['SourceDBServer']) ) {
$content['SourceDBServer'] = $_SESSION['SourceDBServer']; } else {
$content['SourceDBServer'] = "localhost"; }
+	if ( isset($_SESSION['SourceDBPort']) ) { $content['SourceDBPort'] =
$_SESSION['SourceDBPort']; } else { $content['SourceDBPort'] = 3306; }
 	if ( isset($_SESSION['SourceDBTableName']) ) {
$content['SourceDBTableName'] = $_SESSION['SourceDBTableName']; } else
{ $content['SourceDBTableName'] = "systemevents"; }
 	if ( isset($_SESSION['SourceDBUser']) ) { $content['SourceDBUser'] =
$_SESSION['SourceDBUser']; } else { $content['SourceDBUser'] = "user";
}
 	if ( isset($_SESSION['SourceDBPassword']) ) {
$content['SourceDBPassword'] = $_SESSION['SourceDBPassword']; } else {
$content['SourceDBPassword'] = ""; }
@@ -618,6 +619,11 @@
 		else
 			RevertOneStep( $content['INSTALL_STEP']-1,
$content['LN_CFG_PARAMMISSING'] . $content['LN_CFG_DBSERVER'] );

+		if ( isset($_POST['SourceDBPort']) )
+			$_SESSION['SourceDBPort'] =
intval(DB_RemoveBadChars($_POST['SourceDBPort']));
+		else
+			RevertOneStep( $content['INSTALL_STEP']-1,
$content['LN_CFG_PARAMMISSING'] . $content['LN_CFG_DBPORT'] );
+
 		if ( isset($_POST['SourceDBTableName']) )
 			$_SESSION['SourceDBTableName'] =
DB_RemoveBadChars($_POST['SourceDBTableName']);
 		else
@@ -714,6 +720,7 @@
 						"\$CFG['Sources']['Source1']['DBTableType'] = '" .
$_SESSION['SourceDBTableType'] . "';\n" .
 						"\$CFG['Sources']['Source1']['DBType'] = " .
$content['DBTYPES'][$_SESSION['SourceDBType']]['typeastext'] . ";\n" .
 						"\$CFG['Sources']['Source1']['DBServer'] = '" .
$_SESSION['SourceDBServer'] . "';\n" .
+						"\$CFG['Sources']['Source1']['DBPort'] = '" .
$_SESSION['SourceDBPort'] . "';\n" .
 						"\$CFG['Sources']['Source1']['DBName'] = '" .
$_SESSION['SourceDBName'] . "';\n" .
 						"\$CFG['Sources']['Source1']['DBUser'] = '" .
$_SESSION['SourceDBUser'] . "';\n" .
 						"\$CFG['Sources']['Source1']['DBPassword'] = '" .
$_SESSION['SourceDBPassword'] . "';\n" .
diff -Nur loganalyzer-3.2.1.orig/src/templates/install.html
loganalyzer-3.2.1/src/templates/install.html
--- loganalyzer-3.2.1.orig/src/templates/install.html	2011-04-12
22:02:08.000000000 +0800
+++ loganalyzer-3.2.1/src/templates/install.html	2011-06-08
01:15:37.495412402 +0800
@@ -414,6 +414,10 @@
 					<td align="right" class="line0" width="100%"><input type="text"
name="SourceDBServer" size="40" maxlength="255"
value="{SourceDBServer}"></td>
 				</tr>
 				<tr>
+					<td align="left" class="cellmenu2" nowrap><b>{LN_CFG_DBPORT}</b></td>
+					<td align="right" class="line0" width="100%"><input type="text"
name="SourceDBPort" size="40" maxlength="5"
value="{SourceDBPort}"></td>
+				</tr>
+				<tr>
 					<td align="left" class="cellmenu2" nowrap><b>{LN_CFG_DBNAME}</b></td>
 					<td align="right" class="line1" width="100%"><input type="text"
name="SourceDBName" size="40" maxlength="255"
value="{SourceDBName}"></td>
 				</tr>


Thanks,
Kaiwang

From sean at conman.org  Tue Jun  7 21:21:07 2011
From: sean at conman.org (Sean Conner)
Date: Tue, 7 Jun 2011 15:21:07 -0400
Subject: [rsyslog] rsyslog 6.3.1 (devel) released
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280DE3@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280DE3@GRFEXC.intern.adiscon.com>
Message-ID: <20110607192106.GA1476@brevard.conman.org>

It was thus said that the Great Florian Riedl once stated:
> Dear rsyslog mailing list members,
> 
> We have just released the new rsyslog 6.3.1 (devel). This is a first
> implementation of a full-blown DNS cache. While there were some optimizations
> for DNS queries in older releases, especially UDP sometimes suffered under
> slow DNS resolution performance. 
> This is solved with the new dnscache module. Note that the module will
> undergo some more enhancements in the next couple of weeks. Feedback on its
> effect would be deeply appreciated.
> ChangeLog:
> 
> http://www.rsyslog.com/changelog-for-6-3-1-devel/
> 
> Download:
> 
> http://www.rsyslog.com/rsyslog-6-3-1devel/
> 
> As always, feedback is appreciated.

  I've scanned the code, and the two things that leaped out at me were: 

	1. getnameinfo() could potentially block doing the PTR lookup.
	2. It doesn't return the TTL for the PTR record.

  The second issue concerns me (so does the first, but there are ways around
that).  There's currently no provision in the code (it' s just stubbed out)
to flush DNS cache entries, and if one of my servers changes IP address,
outdated information could be logged.

  These two issues seem to scream out to use a full blown DNS resolving
library, in order to run DNS queries in the background and obtain the TTL to
do proper DNS caching.  I might suggest SPCDNS:

	http://www.conman.org/software/spcdns/

as a decent starting point [1].

  -spc

[1]	At work, a project started out using C-Ares for DNS resolution, but
	it proved too problematic to integrate fully into the project (the
	networking model between our project and C-Ares didn't fully mesh)
	so we switched to SPCDNS.  Memory consumption dropped from 15M
	(using C-Ares) to 400k (using SPCDNS) with no loss of speed
	(considering we didn't optimized the compilation of SPCDNS and left
	the asserts enabled).

	Yes, the networking side of SPCDNS is lacking, but that was a plus
	as we could integrated it easier into our networking code.


From chb at muc.de  Tue Jun  7 21:43:05 2011
From: chb at muc.de (Christian Brunner)
Date: Tue, 7 Jun 2011 21:43:05 +0200
Subject: [rsyslog] [PATCH] a json strgen module
Message-ID: <20110607194305.GA30320@sir.fritz.box>

I was playing with liblognorm and a document-oriented database that
is taking input in json format. Therefore I wanted to generate the
json messages directly in rsyslog.

With this module you can generate json messages and send it through any
of the rsyslog output modules.

The module isn't completely finished yet, but it is working and I would
like to get some feedback.

Thanks
Christian
---
 Makefile.am                |    5 +
 configure.ac               |   21 ++++
 doc/smjson.html            |   64 +++++++++++
 plugins/smjson/Makefile.am |    6 +
 plugins/smjson/smjson.c    |  262 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 358 insertions(+), 0 deletions(-)
 create mode 100644 doc/smjson.html
 create mode 100644 plugins/smjson/Makefile.am
 create mode 100644 plugins/smjson/smjson.c

diff --git a/Makefile.am b/Makefile.am
index d689b9e..3a3db59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -208,6 +208,10 @@ if ENABLE_ORACLE
 SUBDIRS += plugins/omoracle
 endif
 
+if ENABLE_SMJSON
+SUBDIRS += plugins/smjson
+endif
+
 if ENABLE_GUI
 SUBDIRS += java
 endif
@@ -253,5 +257,6 @@ DISTCHECK_CONFIGURE_FLAGS=	--enable-gssapi_krb5 \
 				--enable-imtemplate \
 				--enable-omtemplate \
 				--enable-mmsnmptrapd \
+				--enable-smjson \
 				--with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir)
 ACLOCAL_AMFLAGS = -I m4
diff --git a/configure.ac b/configure.ac
index f6a09fa..d3307d0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1220,6 +1220,25 @@ AM_CONDITIONAL(ENABLE_OMMONGODB, test x$enable_ommongodb = xyes)
 # end of copy template - be sure to search for omtemplate to find everything!
 
 
+# SMJSON SUPPORT
+
+AC_ARG_ENABLE(smjson,
+        [AS_HELP_STRING([--enable-smjson],[Compiles smjson strgen module @<:@default=no@:>@])],
+        [case "${enableval}" in
+         yes) enable_smjson="yes" ;;
+          no) enable_smjson="no" ;;
+           *) AC_MSG_ERROR(bad value ${enableval} for --enable-smjson) ;;
+         esac],
+        [enable_smjson=no]
+)
+#
+# you may want to do some library checks here - see snmp, mysql, pgsql modules
+# for samples
+#
+AM_CONDITIONAL(ENABLE_SMJSON, test x$enable_smjson = xyes)
+# end of copy template - be sure to search for omtemplate to find everything!
+
+
 AC_CONFIG_FILES([Makefile \
 		runtime/Makefile \
 		tools/Makefile \
@@ -1262,6 +1281,7 @@ AC_CONFIG_FILES([Makefile \
 		plugins/omoracle/Makefile \
 		plugins/omudpspoof/Makefile \
 		plugins/mmnormalize/Makefile \
+		plugins/smjson/Makefile \
 		plugins/sm_cust_bindcdr/Makefile \
 		plugins/mmsnmptrapd/Makefile \
 		plugins/cust1/Makefile \
@@ -1317,6 +1337,7 @@ echo "    mmsnmptrapd module will be compiled:      $enable_mmsnmptrapd"
 echo
 echo "---{ strgen modules }---"
 echo "    sm_cust_bindcdr module will be compiled:  $enable_sm_cust_bindcdr"
+echo "    smjsonmodule will be compiled:            $enable_smjson"
 echo
 echo "---{ database support }---"
 echo "    MySql support enabled:                    $enable_mysql"
diff --git a/doc/smjson.html b/doc/smjson.html
new file mode 100644
index 0000000..870163b
--- /dev/null
+++ b/doc/smjson.html
@@ -0,0 +1,64 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html><head>
+<meta http-equiv="Content-Language" content="en">
+<title>JSON Strgen Module</title></head>
+
+<body>
+<a href="rsyslog_conf_modules.html">back</a>
+
+<h1>JSON Strgen Module</h1>
+<p><b>Module Name:&nbsp;&nbsp;&nbsp; smjson</b></p>
+<p><b>Author: Christian Brunner &lt;chb at muc.de&gt;</b></p>
+<p><b>Description</b>:</p>
+<p>Provides the ability to format syslog messages in JSON syntax.
+This module uses the <a target="_blank" href="http://libestr.adiscon.com/">
+libestr</a> and the <a target="_blank" href="http://www.libee.org/">libee</a> library. In order to compile this module, you will need to have the
+corresponding developer (headers) package installed. </p>
+<p>&nbsp;</p>
+<p><b>Configuration Directives</b>:</p>
+<ul>
+	<li><strong>$smjsonadd key,value</strong><br>
+	Add a key/value pair to the generated message.<br>
+	transport types which are supported by NET-SNMP. <br>
+	<strong>key</strong> is an arbitrary text string.<br>
+	<strong>value</strong> can be a rsyslog template string (has to start
+        and end with %) or a text string.<br>
+	<br>
+	Example: <strong>$smjsonadd date,%TIMESTAMP:::date-rfc3339%<br>
+	</strong></li>
+</ul>
+<p>&nbsp;</p>
+<p><b>Caveats/Known Bugs:</b></p><ul><li>Json output will be in reverse order. </li></ul>
+<p><b>Sample:</b></p>
+<p>To generate a logfile with json messages like this:</p>
+
+<textarea rows="10" cols="60"> {"type": "SYSLOG", "cee":{"srcport": "54161", "srcip": "192.168.2.195", "user": "chb"}, "msg": "Accepted publickey for chb from 192.168.2.195 port 54161 ssh2", "hostname": "sir", "facility": "authpriv", "level": "info", "date": "2011-06-07T21:20:48+02:00"}
+</textarea>
+
+<p>You will need the following commands:</p>
+<textarea rows="10" cols="60">$ModLoad omsnmp
+
+$ModLoad smjson.so
+$template json,=SMJSON
+$smjsonadd date,%TIMESTAMP:::date-rfc3339%
+$smjsonadd level,%syslogpriority-text%
+$smjsonadd facility,%syslogfacility-text%
+$smjsonadd hostname,%hostname%
+$smjsonadd msg,%msg%
+$smjsonadd cee,%$!all-json%
+$smjsonadd type,SYSLOG
+*.* /var/tmp/json.log;json
+
+</textarea>
+
+<p>The example above is using mmnormalize to normalize the log message ($!all-json).</p>
+
+<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual 
+index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
+<p><font size="2">This documentation is part of the
+<a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
+Copyright ? 2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
+<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL 
+version 3 or higher.</font></p>
+
+</body></html>
diff --git a/plugins/smjson/Makefile.am b/plugins/smjson/Makefile.am
new file mode 100644
index 0000000..c0e9327
--- /dev/null
+++ b/plugins/smjson/Makefile.am
@@ -0,0 +1,6 @@
+pkglib_LTLIBRARIES = smjson.la
+
+smjson_la_SOURCES = smjson.c
+smjson_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS) $(RSRT_CFLAGS) $(LIBEE_CFLAGS)
+smjson_la_LDFLAGS = -module -avoid-version $(LIBEE_LIBS)
+smjson_la_LIBADD =
diff --git a/plugins/smjson/smjson.c b/plugins/smjson/smjson.c
new file mode 100644
index 0000000..9d0b7d3
--- /dev/null
+++ b/plugins/smjson/smjson.c
@@ -0,0 +1,262 @@
+/* smjson.c
+ *
+ * A strgen module to transform log messages into the json format.
+ *
+ * File begun on 2011-06-02 by Christian Brunner <chb at muc.de>
+ *
+ * Rsyslog is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ * 
+ * Rsyslog is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ * 
+ * You should have received a copy of the GNU General Public License
+ * along with Rsyslog.  If not, see <http://www.gnu.org/licenses/>.
+ * 
+ * A copy of the GPL can be found in the file "COPYING" in this distribution.
+ */
+
+#include "config.h"
+#include "rsyslog.h"
+#include <stdlib.h>
+#include <ctype.h>
+#include <string.h>
+#include <assert.h>
+#include <errno.h>
+#include "conf.h"
+#include "syslogd-types.h"
+#include "cfsysline.h"
+#include "template.h"
+#include "msg.h"
+#include "module-template.h"
+#include "unicode-helper.h"
+#include "errmsg.h"
+
+#include <libee/libee.h>
+#include <libee/field.h>
+
+MODULE_TYPE_STRGEN
+MODULE_TYPE_NOKEEP
+STRGEN_NAME("SMJSON")
+
+/* internal structures
+ */
+DEF_SMOD_STATIC_DATA
+DEFobjCurrIf(errmsg)
+
+typedef struct jsonentry_s {
+	char *key;
+	struct ee_field *constfield;
+        struct templateEntry *pTpe;
+        struct jsonentry_s *next;
+} jsonentry_t;
+
+static jsonentry_t *root;
+static ee_ctx ctx;
+
+static rsRetVal addJsonEntry(void __attribute__((unused)) *pVal, uchar *pNewVal)
+{
+	jsonentry_t *pNew;
+	DEFiRet;
+
+	struct template *pTpl;
+        struct templateEntry *pTpe;
+
+	char *key, *value;
+	es_str_t *estr;
+
+	struct ee_value *eevalue;
+
+	key = (char *) pNewVal;
+	if (!(value = strchr((char *) pNewVal, ','))) {
+		errmsg.LogError(0, NO_ERRCODE, "error: key/value separator "
+			"not found in '%s'", pNewVal);
+                ABORT_FINALIZE(RS_RET_ERR);
+	}
+	*value = '\0';
+	value++;
+
+	CHKmalloc(pNew = malloc(sizeof(jsonentry_t)));
+	pNew->key = key;
+
+	if (value[0] == '%') {
+		unsigned char *templateString;
+		CHKmalloc(templateString = malloc(strlen(value) + 3));
+		templateString[0] = '"';
+		memcpy(templateString+1, value, strlen(value));
+		memcpy(templateString+1+strlen(value), "\"", 2);
+
+		pTpl = tplAddLine("smjson-intern", &templateString);
+		pTpe = pTpl->pEntryRoot;
+
+		pNew->constfield = NULL;
+		pNew->pTpe = pTpe;
+
+		free(templateString);
+	} else {
+		pNew->constfield = ee_newField(ctx);
+		pNew->constfield->name = es_newStrFromBuf(key, strlen(key));
+		estr = es_newStrFromBuf(value, strlen(value));
+		eevalue = ee_newValue(ctx);
+		ee_setStrValue(eevalue, estr);
+		ee_addValueToField(pNew->constfield, eevalue);
+
+		pNew->pTpe = NULL;
+	}
+
+	pNew->next = root;
+	root = pNew;
+
+	DBGPRINTF("smjson: key/value '%s':'%s' added.\n", key, value);
+
+finalize_it:
+	if(iRet != RS_RET_OK) {
+		free(pNewVal);
+	}
+
+	RETiRet;
+}
+	
+
+/* This strgen uses libee to generate the output string.
+ */
+
+#define JSON_END "\n"
+BEGINstrgen
+	es_str_t *json;
+	es_str_t *estr;
+	struct ee_field *field;
+	struct ee_value *eevalue;
+	propid_t propID;
+	size_t propLen, jsonLen;
+	uchar *pszProp = NULL;
+	unsigned short bMustBeFreed = 0;
+	char *jsonStr;
+
+	char *key;
+
+	jsonentry_t *iter;
+        struct templateEntry *pTpe;
+
+	if((json = es_newStr(256)) == NULL) goto finalize_it;
+
+        es_addChar(&json, '{');
+
+	for(iter = root; iter != NULL; iter = iter->next)
+	{
+		key = iter->key;
+
+		if (iter->constfield) {
+			ee_addField_JSON(iter->constfield, &json, 0);
+			
+			goto nextiter;
+		}
+
+		pTpe = iter->pTpe;
+		propID = pTpe->data.field.propid;
+
+		if(propID == PROP_CEE_ALL_JSON) { 
+			char *ceestr;
+
+			es_addChar(&json, '"');
+			es_addBuf(&json, key, strlen(key));
+			es_addBuf(&json, "\":", 2);
+			ee_fmtEventToJSON(pMsg->event, &estr);
+			ceestr = es_str2cstr(estr, "#000");
+			es_deleteStr(estr);
+			es_addBuf(&json, ceestr, strlen(ceestr));
+			free(ceestr);
+
+			goto nextiter;
+		} 
+
+		field = ee_newField(ctx);
+		field->name = es_newStrFromBuf(key, strlen(key));
+
+		pszProp = MsgGetProp(pMsg, pTpe, propID, 
+			pTpe->data.field.propName, &propLen, &bMustBeFreed);
+
+		if(propID == PROP_MSG && pszProp[0] == ' ') { 
+			estr = es_newStrFromBuf((char *) pszProp+1,
+				strlen((char *) pszProp)-1);
+		} else {
+			estr = es_newStrFromBuf((char *) pszProp,
+				strlen((char *) pszProp));
+		}
+
+		eevalue = ee_newValue(ctx);
+		ee_setStrValue(eevalue, estr);
+		ee_addValueToField(field, eevalue);
+
+		ee_addField_JSON(field, &json, 0);
+		ee_deleteField(field);
+
+		if(bMustBeFreed) {
+			free(pszProp);
+			bMustBeFreed = 0;
+		}
+
+nextiter:
+		if (iter->next)
+			es_addBuf(&json, ", ", ee_ctxIsEncUltraCompact(ctx) ? 1 : 2);
+	}
+
+        es_addChar(&json, '}');
+
+	jsonStr = (char*) es_str2cstr(json, "#000");
+	jsonLen = strlen(jsonStr);
+
+	es_deleteStr(json);
+
+        if((jsonLen + sizeof(JSON_END)) >= *pLenBuf)
+                CHKiRet(ExtendBuf(ppBuf, pLenBuf, jsonLen + sizeof(JSON_END)));
+
+	memcpy(*ppBuf, jsonStr, jsonLen);
+	memcpy(*ppBuf + jsonLen, JSON_END, sizeof(JSON_END));
+
+	free(jsonStr);
+	ee_exitCtx(ctx);
+
+finalize_it:
+        if(bMustBeFreed)
+                free(pszProp);
+ENDstrgen
+
+BEGINmodExit
+	jsonentry_t *iter, *jdel;
+CODESTARTmodExit
+	for(iter = root; iter != NULL; ) {
+		jdel = iter;
+		iter = iter->next;
+		if (jdel->constfield) {
+			ee_deleteField(jdel->constfield);
+		}
+		free(jdel->key);
+	}
+	objRelease(errmsg, CORE_COMPONENT);
+ENDmodExit
+
+
+BEGINqueryEtryPt
+CODESTARTqueryEtryPt
+CODEqueryEtryPt_STD_SMOD_QUERIES
+ENDqueryEtryPt
+
+BEGINmodInit()
+CODESTARTmodInit
+        *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
+CODEmodInit_QueryRegCFSLineHdlr
+        CHKiRet(objUse(errmsg, CORE_COMPONENT));
+
+	root = NULL;
+	ctx = ee_initCtx();
+
+        CHKiRet(omsdRegCFSLineHdlr((uchar *)"smjsonadd", 0, eCmdHdlrGetWord,
+                addJsonEntry, NULL, STD_LOADABLE_MODULE_ID, eConfObjGlobal));
+
+        dbgprintf("rsyslog json strgen init called, compiled with version %s\n", VERSION);
+ENDmodInit
-- 
1.7.1


From alorbach at ro1.adiscon.com  Wed Jun  8 11:42:36 2011
From: alorbach at ro1.adiscon.com (Andre Lorbach)
Date: Wed, 8 Jun 2011 11:42:36 +0200
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
	conform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com><E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com>
	<BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com>
Message-ID: <E152A03EC5F8DF4D8567006AF38A282D1ED5E5@vmexch1.intern.adiscon.com>

The ProcessID field is more or less an optional, so having a NULL value in it
is fine. 
Populating it with the ProcessID field will be useful for filtering within
LogAnalyzer.

However as far as I know, the default template does not include the ProcessID
field, but it can be easily extended. 

Best regards,
Andre Lorbach

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Dienstag, 7. Juni 2011 19:34
> To: rsyslog-users
> Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
> conform to table type monitorware in loganalyzer-3.2.1 ?
> 
> Will the default output template of rsyslog fill the new procid field?
> Looks like leaving it NULL should work as well.
> 
> Thanks,
> Kaiwang
> 
> 2011/6/7 Andre Lorbach <alorbach at ro1.adiscon.com>:
> > Hi,
> >
> > the ProcessID field was added for LogAnalyzer. It wasn't in
> > MonitorWare either.
> > But LogAnalyzer will automatically add missing fields into the
> > logstream databases, if the database user has sufficient rights to the
> > table. So granting the database user sufficient rights would solve the
> problem for now.
> >
> >
> > Apparently adding this field into the default database schema of
> > MonitorWare and RSyslog was lost in communication somewhere.
> >
> > Best regards,
> > Andre Lorbach
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> Sent: Dienstag, 7. Juni 2011 16:04
> >> To: rsyslog-users
> >> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
> >> conform
> > to
> >> table type monitorware in loganalyzer-3.2.1 ?
> >>
> >> Hello,
> >>
> >> In Step 7 of installation process, "Create the first source for
> >> syslog messages", selecting Table type: MonitorWare (the other is
> >> SyslogNG) would load $dbmapping['mnoitorware'] in
> >> include/constants_logstream.php, resulting in SQL like this:
> >>
> >> SELECT id, devicereportedtime, facility, priority, fromhost,
> >> syslogtag, processid, infounitid, message FROM SystemEvents ORDER BY
> >> id DESC LIMIT
> >> 100
> >>
> >> In the case of syslog, the fields are mapped from
> >> ./include/functions_config.php:
> >>
> >> ?501 ? ? ? ? $CFG['Views']['SYSLOG']= array(
> >> ?502
> >> ? ? ? 'ID' => ? ? ? ? ? ? ? ? "SYSLOG",
> >> ?503
> >> ? ? ? 'DisplayName' =>"Syslog Fields",
> >> ?504
> >> ? ? ? 'Columns' => ? ?array ( SYSLOG_DATE, SYSLOG_FACILITY, S
> >> ? ? ?YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
> SYSLOG_PROCESSID,
> >> SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
> >> ?505
> >> ? ? ? 'userid' => ? ? ? ? ? ? null,
> >> ?506
> >> ? ? ? 'groupid' => ? ?null,
> >> ?507
> >> );
> >>
> >> Columns array:
> >> ? ? ? ? ? ? [0] => timereported
> >> ? ? ? ? ? ? [1] => syslogfacility
> >> ? ? ? ? ? ? [2] => syslogseverity
> >> ? ? ? ? ? ? [3] => FROMHOST
> >> ? ? ? ? ? ? [4] => syslogtag
> >> ? ? ? ? ? ? [5] => procid
> >> ? ? ? ? ? ? [6] => IUT
> >> ? ? ? ? ? ? [7] => msg
> >>
> >>
> >> Finally, I got a error prompt like this:
> >>
> >> No syslog records found - Error Details:
> >>
> >> No syslog records found
> >>
> >>
> >> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
> >> processid filed is missing)
> >>
> >> CREATE TABLE SystemEvents
> >> (
> >> ? ? ? ? ID int unsigned not null auto_increment primary key,
> >> ? ? ? ? CustomerID bigint,
> >> ? ? ? ? ReceivedAt datetime NULL,
> >> ? ? ? ? DeviceReportedTime datetime NULL,
> >> ? ? ? ? Facility smallint NULL,
> >> ? ? ? ? Priority smallint NULL,
> >> ? ? ? ? FromHost varchar(60) NULL,
> >> ? ? ? ? Message text,
> >> ? ? ? ? NTSeverity int NULL,
> >> ? ? ? ? Importance int NULL,
> >> ? ? ? ? EventSource varchar(60),
> >> ? ? ? ? EventUser varchar(60) NULL,
> >> ? ? ? ? EventCategory int NULL,
> >> ? ? ? ? EventID int NULL,
> >> ? ? ? ? EventBinaryData text NULL,
> >> ? ? ? ? MaxAvailable int NULL,
> >> ? ? ? ? CurrUsage int NULL,
> >> ? ? ? ? MinUsage int NULL,
> >> ? ? ? ? MaxUsage int NULL,
> >> ? ? ? ? InfoUnitID int NULL ,
> >> ? ? ? ? SysLogTag varchar(60),
> >> ? ? ? ? EventLogType varchar(60),
> >> ? ? ? ? GenericFileName VarChar(60),
> >> ? ? ? ? SystemID int NULL
> >> );
> >>
> >>
> >> So, what should I do? I heard of monitorware schema, and assumed it
> >> to be what shipped with rsyslog.
> >>
> >>
> >> Thanks,
> >> Kaiwang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun  8 14:26:40 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 8 Jun 2011 14:26:40 +0200
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
	doesn'tconform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <E152A03EC5F8DF4D8567006AF38A282D1ED5E5@vmexch1.intern.adiscon.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com><E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com><BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com>
	<E152A03EC5F8DF4D8567006AF38A282D1ED5E5@vmexch1.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280DFE@GRFEXC.intern.adiscon.com>

This field is not populated and I am bit hesitant to change the default
template. That will probably break a number of running configurations. Also,
I can not reliably populate that field due to the variety of different ways a
process ID is expressed...

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Andre Lorbach
> Sent: Wednesday, June 08, 2011 11:43 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
doesn'tconform
> to table type monitorware in loganalyzer-3.2.1 ?
> 
> The ProcessID field is more or less an optional, so having a NULL value in
it is
> fine.
> Populating it with the ProcessID field will be useful for filtering within
> LogAnalyzer.
> 
> However as far as I know, the default template does not include the
> ProcessID field, but it can be easily extended.
> 
> Best regards,
> Andre Lorbach
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> > Sent: Dienstag, 7. Juni 2011 19:34
> > To: rsyslog-users
> > Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
> > conform to table type monitorware in loganalyzer-3.2.1 ?
> >
> > Will the default output template of rsyslog fill the new procid field?
> > Looks like leaving it NULL should work as well.
> >
> > Thanks,
> > Kaiwang
> >
> > 2011/6/7 Andre Lorbach <alorbach at ro1.adiscon.com>:
> > > Hi,
> > >
> > > the ProcessID field was added for LogAnalyzer. It wasn't in
> > > MonitorWare either.
> > > But LogAnalyzer will automatically add missing fields into the
> > > logstream databases, if the database user has sufficient rights to
> > > the table. So granting the database user sufficient rights would
> > > solve the
> > problem for now.
> > >
> > >
> > > Apparently adding this field into the default database schema of
> > > MonitorWare and RSyslog was lost in communication somewhere.
> > >
> > > Best regards,
> > > Andre Lorbach
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> > >> Sent: Dienstag, 7. Juni 2011 16:04
> > >> To: rsyslog-users
> > >> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
> > >> conform
> > > to
> > >> table type monitorware in loganalyzer-3.2.1 ?
> > >>
> > >> Hello,
> > >>
> > >> In Step 7 of installation process, "Create the first source for
> > >> syslog messages", selecting Table type: MonitorWare (the other is
> > >> SyslogNG) would load $dbmapping['mnoitorware'] in
> > >> include/constants_logstream.php, resulting in SQL like this:
> > >>
> > >> SELECT id, devicereportedtime, facility, priority, fromhost,
> > >> syslogtag, processid, infounitid, message FROM SystemEvents ORDER
> > >> BY id DESC LIMIT
> > >> 100
> > >>
> > >> In the case of syslog, the fields are mapped from
> > >> ./include/functions_config.php:
> > >>
> > >> ?501 ? ? ? ? $CFG['Views']['SYSLOG']= array(
> > >> ?502
> > >> ? ? ? 'ID' => ? ? ? ? ? ? ? ? "SYSLOG",
> > >> ?503
> > >> ? ? ? 'DisplayName' =>"Syslog Fields",
> > >> ?504
> > >> ? ? ? 'Columns' => ? ?array ( SYSLOG_DATE, SYSLOG_FACILITY, S
> > >> ? ? ?YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
> > SYSLOG_PROCESSID,
> > >> SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
> > >> ?505
> > >> ? ? ? 'userid' => ? ? ? ? ? ? null,
> > >> ?506
> > >> ? ? ? 'groupid' => ? ?null,
> > >> ?507
> > >> );
> > >>
> > >> Columns array:
> > >> ? ? ? ? ? ? [0] => timereported
> > >> ? ? ? ? ? ? [1] => syslogfacility
> > >> ? ? ? ? ? ? [2] => syslogseverity
> > >> ? ? ? ? ? ? [3] => FROMHOST
> > >> ? ? ? ? ? ? [4] => syslogtag
> > >> ? ? ? ? ? ? [5] => procid
> > >> ? ? ? ? ? ? [6] => IUT
> > >> ? ? ? ? ? ? [7] => msg
> > >>
> > >>
> > >> Finally, I got a error prompt like this:
> > >>
> > >> No syslog records found - Error Details:
> > >>
> > >> No syslog records found
> > >>
> > >>
> > >> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
> > >> processid filed is missing)
> > >>
> > >> CREATE TABLE SystemEvents
> > >> (
> > >> ? ? ? ? ID int unsigned not null auto_increment primary key,
> > >> ? ? ? ? CustomerID bigint,
> > >> ? ? ? ? ReceivedAt datetime NULL,
> > >> ? ? ? ? DeviceReportedTime datetime NULL,
> > >> ? ? ? ? Facility smallint NULL,
> > >> ? ? ? ? Priority smallint NULL,
> > >> ? ? ? ? FromHost varchar(60) NULL,
> > >> ? ? ? ? Message text,
> > >> ? ? ? ? NTSeverity int NULL,
> > >> ? ? ? ? Importance int NULL,
> > >> ? ? ? ? EventSource varchar(60),
> > >> ? ? ? ? EventUser varchar(60) NULL,
> > >> ? ? ? ? EventCategory int NULL,
> > >> ? ? ? ? EventID int NULL,
> > >> ? ? ? ? EventBinaryData text NULL,
> > >> ? ? ? ? MaxAvailable int NULL,
> > >> ? ? ? ? CurrUsage int NULL,
> > >> ? ? ? ? MinUsage int NULL,
> > >> ? ? ? ? MaxUsage int NULL,
> > >> ? ? ? ? InfoUnitID int NULL ,
> > >> ? ? ? ? SysLogTag varchar(60),
> > >> ? ? ? ? EventLogType varchar(60),
> > >> ? ? ? ? GenericFileName VarChar(60),
> > >> ? ? ? ? SystemID int NULL
> > >> );
> > >>
> > >>
> > >> So, what should I do? I heard of monitorware schema, and assumed it
> > >> to be what shipped with rsyslog.
> > >>
> > >>
> > >> Thanks,
> > >> Kaiwang
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From kaiwang.chen at gmail.com  Wed Jun  8 15:16:10 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Wed, 8 Jun 2011 21:16:10 +0800
Subject: [rsyslog] Problem with forwarded multiline message
Message-ID: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>

Hello,

I set up two hosts to test rsyslogd, dns1 as client, z6 as server, and
found that the server interpreted a copy of forwarded multiline
message(3rd entry in the following raw messages) into multiple
entries(3rd and 4th entry in actual output), while locally generated
multiline message was fine. What's the problem?

The client setting:
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
*.*                @@10.3.254.106:514

The server setting:
$InputPTCPServerRun 514
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
$template rawfmt,"%rawmsg%\n"
*.*    /var/log/rawmessages;rawfmt
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

Other settings were action queue tuning, I guess they were irrelevant.

The raw messages:
<6>Jun  8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg started.
<46>Jun  8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
<43>Jun  8 20:52:48 dns1 rsyslogd-2066: could not load module
'/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
cannot open shared object file: No such file or directory
 [try http://www.rsyslog.com/e/2066 ]
<43>Jun  8 20:52:48 dns1 rsyslogd: the last error occured in
/etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
<43>Jun  8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
interpret master config file '/etc/rsyslog.conf'. [try
http://www.rsyslog.com/e/2124 ]
imklog 5.8.1, log source = /proc/kmsg started.
 [origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
x-info="http://www.rsyslog.com"] start
could not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
/usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
file or directory
 [try http://www.rsyslog.com/e/2066 ]
the last error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
CONFIG ERROR: could not interpret master config file
'/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]

Actual ouput:
<6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - -  imklog 5.8.1, log
source = /proc/kmsg started.
<46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  [origin
software="rsyslogd" swVersion="5.8.1" x-pid="4152"
x-info="http://www.rsyslog.com"] start
<43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - -  could not
load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
/usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
file or directory
<13>1 2011-06-08T20:52:48.251337+08:00 bogon  - - -  [try
http://www.rsyslog.com/e/2066 ]
<43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  the last error
occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
<43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - -  CONFIG
ERROR: could not interpret master config file '/etc/rsyslog.conf'.
[try http://www.rsyslog.com/e/2124 ]
<6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1,
log source = /proc/kmsg started.
<46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - -  [origin
software="rsyslogd" swVersion="5.8.1" x-pid="11033"
x-info="http://www.rsyslog.com"] start
<43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could
not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
/usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
file or directory
 [try http://www.rsyslog.com/e/2066 ]
<43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
<43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
ERROR: could not interpret master config file '/etc/rsyslog.conf'.
[try http://www.rsyslog.com/e/2124 ]


Thanks,
Kaiwang

From rgerhards at hq.adiscon.com  Wed Jun  8 15:42:23 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 8 Jun 2011 15:42:23 +0200
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>

Multi-line messages are not supported by legacy plain syslogd. But you can
turn on the (o) option, which enables octect-counted framing, with which it
works. However, non-rsyslog receivers probably do not understand that
framing.
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Wednesday, June 08, 2011 3:16 PM
> To: rsyslog-users
> Subject: [rsyslog] Problem with forwarded multiline message
> 
> Hello,
> 
> I set up two hosts to test rsyslogd, dns1 as client, z6 as server, and
found that
> the server interpreted a copy of forwarded multiline message(3rd entry in
> the following raw messages) into multiple entries(3rd and 4th entry in
actual
> output), while locally generated multiline message was fine. What's the
> problem?
> 
> The client setting:
> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> *.*                @@10.3.254.106:514
> 
> The server setting:
> $InputPTCPServerRun 514
> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template
> rawfmt,"%rawmsg%\n"
> *.*    /var/log/rawmessages;rawfmt
> *.info;mail.none;authpriv.none;cron.none                /var/log/messages
> 
> Other settings were action queue tuning, I guess they were irrelevant.
> 
> The raw messages:
> <6>Jun  8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
started.
> <46>Jun  8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
> <43>Jun  8 20:52:48 dns1 rsyslogd-2066: could not load module
> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
> cannot open shared object file: No such file or directory  [try
> http://www.rsyslog.com/e/2066 ] <43>Jun  8 20:52:48 dns1 rsyslogd: the last
> error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> <43>Jun  8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not interpret
> master config file '/etc/rsyslog.conf'. [try
> http://www.rsyslog.com/e/2124 ]
> imklog 5.8.1, log source = /proc/kmsg started.
>  [origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> x-info="http://www.rsyslog.com"] start
> could not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
or
> directory  [try http://www.rsyslog.com/e/2066 ] the last error occured in
> /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'.
[try
> http://www.rsyslog.com/e/2124 ]
> 
> Actual ouput:
> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - -  imklog 5.8.1, log source
=
> /proc/kmsg started.
> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  [origin
> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
> x-info="http://www.rsyslog.com"] start
> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - -  could not load
> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
or
> directory
> <13>1 2011-06-08T20:52:48.251337+08:00 bogon  - - -  [try
> http://www.rsyslog.com/e/2066 ]
> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  the last error occured
in
> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - -  CONFIG
> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try http://www.rsyslog.com/e/2124 ]
> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1, log
source
> = /proc/kmsg started.
> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - -  [origin
> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> x-info="http://www.rsyslog.com"] start
> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could not
load
> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
or
> directory  [try http://www.rsyslog.com/e/2066 ]
> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last error
> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try http://www.rsyslog.com/e/2124 ]
> 
> 
> Thanks,
> Kaiwang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From kaiwang.chen at gmail.com  Wed Jun  8 16:03:53 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Wed, 8 Jun 2011 22:03:53 +0800
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>

So the (o) option just affects the receiver, and would be of no harm
being turned on at the terminal end of syslog message flow.  Actually,
I am going to use rsyslog on both ends, except bridge and router
sources.

I came across such a framing option days ago, and just can't locate
it. How to turn the (o) option, is it a compilation flag or a
configuration directive?

Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I
understand stock sysklogd can't deal with multiline messages.

Thanks,
Kaiwang

2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Multi-line messages are not supported by legacy plain syslogd. But you can
> turn on the (o) option, which enables octect-counted framing, with which it
> works. However, non-rsyslog receivers probably do not understand that
> framing.
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Wednesday, June 08, 2011 3:16 PM
>> To: rsyslog-users
>> Subject: [rsyslog] Problem with forwarded multiline message
>>
>> Hello,
>>
>> I set up two hosts to test rsyslogd, dns1 as client, z6 as server, and
> found that
>> the server interpreted a copy of forwarded multiline message(3rd entry in
>> the following raw messages) into multiple entries(3rd and 4th entry in
> actual
>> output), while locally generated multiline message was fine. What's the
>> problem?
>>
>> The client setting:
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
>>
>> The server setting:
>> $InputPTCPServerRun 514
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template
>> rawfmt,"%rawmsg%\n"
>> *.* ? ?/var/log/rawmessages;rawfmt
>> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?/var/log/messages
>>
>> Other settings were action queue tuning, I guess they were irrelevant.
>>
>> The raw messages:
>> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
> started.
>> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
>> <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load module
>> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
>> cannot open shared object file: No such file or directory ?[try
>> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd: the last
>> error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not interpret
>> master config file '/etc/rsyslog.conf'. [try
>> http://www.rsyslog.com/e/2124 ]
>> imklog 5.8.1, log source = /proc/kmsg started.
>> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> x-info="http://www.rsyslog.com"] start
>> could not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory ?[try http://www.rsyslog.com/e/2066 ] the last error occured in
>> /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try
>> http://www.rsyslog.com/e/2124 ]
>>
>> Actual ouput:
>> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1, log source
> =
>> /proc/kmsg started.
>> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
>> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>> x-info="http://www.rsyslog.com"] start
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could not load
>> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory
>> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
>> http://www.rsyslog.com/e/2066 ]
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last error occured
> in
>> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1, log
> source
>> = /proc/kmsg started.
>> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
>> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> x-info="http://www.rsyslog.com"] start
>> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could not
> load
>> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory ?[try http://www.rsyslog.com/e/2066 ]
>> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last error
>> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>>
>>
>> Thanks,
>> Kaiwang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed Jun  8 16:39:08 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 8 Jun 2011 16:39:08 +0200
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
	<BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Wednesday, June 08, 2011 4:04 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problem with forwarded multiline message
> 
> So the (o) option just affects the receiver, and would be of no harm being
> turned on at the terminal end of syslog message flow.  Actually, I am going
to
> use rsyslog on both ends, except bridge and router sources.

Actually the sender. The receiver automatically handles both. The option is
at the action, I think it is along the lines of

@@(o)host

> 
> I came across such a framing option days ago, and just can't locate it. How
to
> turn the (o) option, is it a compilation flag or a configuration directive?
> 
> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I understand
stock
> sysklogd can't deal with multiline messages.

Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
(protocol)". This is what most applications understand under "TCP syslog". It
uses \n to end a message and start a new one. Usually this is not a problem,
as control character escaping removes \n in any case.

Rainer
> 
> Thanks,
> Kaiwang
> 
> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > Multi-line messages are not supported by legacy plain syslogd. But you
> > can turn on the (o) option, which enables octect-counted framing, with
> > which it works. However, non-rsyslog receivers probably do not
> > understand that framing.
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> Sent: Wednesday, June 08, 2011 3:16 PM
> >> To: rsyslog-users
> >> Subject: [rsyslog] Problem with forwarded multiline message
> >>
> >> Hello,
> >>
> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
> >> and
> > found that
> >> the server interpreted a copy of forwarded multiline message(3rd
> >> entry in the following raw messages) into multiple entries(3rd and
> >> 4th entry in
> > actual
> >> output), while locally generated multiline message was fine. What's
> >> the problem?
> >>
> >> The client setting:
> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
> >>
> >> The server setting:
> >> $InputPTCPServerRun 514
> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template
> >> rawfmt,"%rawmsg%\n"
> >> *.* ? ?/var/log/rawmessages;rawfmt
> >> *.info;mail.none;authpriv.none;cron.none
> >> /var/log/messages
> >>
> >> Other settings were action queue tuning, I guess they were irrelevant.
> >>
> >> The raw messages:
> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
> > started.
> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load module
> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
> >> cannot open shared object file: No such file or directory ?[try
> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
> >> interpret master config file '/etc/rsyslog.conf'. [try
> >> http://www.rsyslog.com/e/2124 ]
> >> imklog 5.8.1, log source = /proc/kmsg started.
> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> >> x-info="http://www.rsyslog.com"] start could not load module
> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
> >> file
> > or
> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> >> CONFIG ERROR: could not interpret master config file
'/etc/rsyslog.conf'.
> > [try
> >> http://www.rsyslog.com/e/2124 ]
> >>
> >> Actual ouput:
> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1, log
> >> source
> > =
> >> /proc/kmsg started.
> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
> >> x-info="http://www.rsyslog.com"] start
> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could not
> >> load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
> >> file
> > or
> >> directory
> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
> >> http://www.rsyslog.com/e/2066 ]
> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last error
> >> occured
> > in
> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> >> [try http://www.rsyslog.com/e/2124 ]
> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1,
> >> log
> > source
> >> = /proc/kmsg started.
> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> >> x-info="http://www.rsyslog.com"] start
> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could
> >> not
> > load
> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
> >> file
> > or
> >> directory ?[try http://www.rsyslog.com/e/2066 ]
> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> >> [try http://www.rsyslog.com/e/2124 ]
> >>
> >>
> >> Thanks,
> >> Kaiwang
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun  8 17:42:06 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 8 Jun 2011 17:42:06 +0200
Subject: [rsyslog] [PATCH] a json strgen module
In-Reply-To: <20110607194305.GA30320@sir.fritz.box>
References: <20110607194305.GA30320@sir.fritz.box>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E0E@GRFEXC.intern.adiscon.com>

Sorry, I am a bit swamped, thus the sluggish reply. Will be on the road most
of tomorrow and Friday as well (with Monday being a public holiday over
here). This sounds interesting. I just want to make sure you have seen the
all-json property which generates a json format. I am not sure if that is
useful for your case... Will have a better review asap.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Christian Brunner
> Sent: Tuesday, June 07, 2011 9:43 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] [PATCH] a json strgen module
> 
> I was playing with liblognorm and a document-oriented database that is
> taking input in json format. Therefore I wanted to generate the json
> messages directly in rsyslog.
> 
> With this module you can generate json messages and send it through any of
> the rsyslog output modules.
> 
> The module isn't completely finished yet, but it is working and I would
like to
> get some feedback.
> 
> Thanks
> Christian
> ---
>  Makefile.am                |    5 +
>  configure.ac               |   21 ++++
>  doc/smjson.html            |   64 +++++++++++
>  plugins/smjson/Makefile.am |    6 +
>  plugins/smjson/smjson.c    |  262
> ++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 358 insertions(+), 0 deletions(-)  create mode 100644
> doc/smjson.html  create mode 100644 plugins/smjson/Makefile.am  create
> mode 100644 plugins/smjson/smjson.c
> 
> diff --git a/Makefile.am b/Makefile.am
> index d689b9e..3a3db59 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -208,6 +208,10 @@ if ENABLE_ORACLE
>  SUBDIRS += plugins/omoracle
>  endif
> 
> +if ENABLE_SMJSON
> +SUBDIRS += plugins/smjson
> +endif
> +
>  if ENABLE_GUI
>  SUBDIRS += java
>  endif
> @@ -253,5 +257,6 @@ DISTCHECK_CONFIGURE_FLAGS=	--enable-
> gssapi_krb5 \
>  				--enable-imtemplate \
>  				--enable-omtemplate \
>  				--enable-mmsnmptrapd \
> +				--enable-smjson \
>  				--with-
> systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir)
>  ACLOCAL_AMFLAGS = -I m4
> diff --git a/configure.ac b/configure.ac index f6a09fa..d3307d0 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -1220,6 +1220,25 @@ AM_CONDITIONAL(ENABLE_OMMONGODB, test
> x$enable_ommongodb = xyes)  # end of copy template - be sure to search
> for omtemplate to find everything!
> 
> 
> +# SMJSON SUPPORT
> +
> +AC_ARG_ENABLE(smjson,
> +        [AS_HELP_STRING([--enable-smjson],[Compiles smjson strgen module
> @<:@default=no@:>@])],
> +        [case "${enableval}" in
> +         yes) enable_smjson="yes" ;;
> +          no) enable_smjson="no" ;;
> +           *) AC_MSG_ERROR(bad value ${enableval} for --enable-smjson) ;;
> +         esac],
> +        [enable_smjson=no]
> +)
> +#
> +# you may want to do some library checks here - see snmp, mysql, pgsql
> +modules # for samples # AM_CONDITIONAL(ENABLE_SMJSON, test
> +x$enable_smjson = xyes) # end of copy template - be sure to search for
> +omtemplate to find everything!
> +
> +
>  AC_CONFIG_FILES([Makefile \
>  		runtime/Makefile \
>  		tools/Makefile \
> @@ -1262,6 +1281,7 @@ AC_CONFIG_FILES([Makefile \
>  		plugins/omoracle/Makefile \
>  		plugins/omudpspoof/Makefile \
>  		plugins/mmnormalize/Makefile \
> +		plugins/smjson/Makefile \
>  		plugins/sm_cust_bindcdr/Makefile \
>  		plugins/mmsnmptrapd/Makefile \
>  		plugins/cust1/Makefile \
> @@ -1317,6 +1337,7 @@ echo "    mmsnmptrapd module will be compiled:
> $enable_mmsnmptrapd"
>  echo
>  echo "---{ strgen modules }---"
>  echo "    sm_cust_bindcdr module will be compiled:
> $enable_sm_cust_bindcdr"
> +echo "    smjsonmodule will be compiled:            $enable_smjson"
>  echo
>  echo "---{ database support }---"
>  echo "    MySql support enabled:                    $enable_mysql"
> diff --git a/doc/smjson.html b/doc/smjson.html new file mode 100644 index
> 0000000..870163b
> --- /dev/null
> +++ b/doc/smjson.html
> @@ -0,0 +1,64 @@
> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
> +<html><head> <meta http-equiv="Content-Language" content="en">
> +<title>JSON Strgen Module</title></head>
> +
> +<body>
> +<a href="rsyslog_conf_modules.html">back</a>
> +
> +<h1>JSON Strgen Module</h1>
> +<p><b>Module Name:&nbsp;&nbsp;&nbsp; smjson</b></p>
> +<p><b>Author: Christian Brunner &lt;chb at muc.de&gt;</b></p>
> +<p><b>Description</b>:</p> <p>Provides the ability to format syslog
> +messages in JSON syntax.
> +This module uses the <a target="_blank"
> +href="http://libestr.adiscon.com/">
> +libestr</a> and the <a target="_blank"
> +href="http://www.libee.org/">libee</a> library. In order to compile
> +this module, you will need to have the corresponding developer
> +(headers) package installed. </p> <p>&nbsp;</p> <p><b>Configuration
> +Directives</b>:</p> <ul>
> +	<li><strong>$smjsonadd key,value</strong><br>
> +	Add a key/value pair to the generated message.<br>
> +	transport types which are supported by NET-SNMP. <br>
> +	<strong>key</strong> is an arbitrary text string.<br>
> +	<strong>value</strong> can be a rsyslog template string (has to start
> +        and end with %) or a text string.<br>
> +	<br>
> +	Example: <strong>$smjsonadd date,%TIMESTAMP:::date-
> rfc3339%<br>
> +	</strong></li>
> +</ul>
> +<p>&nbsp;</p>
> +<p><b>Caveats/Known Bugs:</b></p><ul><li>Json output will be in
> reverse
> +order. </li></ul> <p><b>Sample:</b></p> <p>To generate a logfile with
> +json messages like this:</p>
> +
> +<textarea rows="10" cols="60"> {"type": "SYSLOG", "cee":{"srcport":
> +"54161", "srcip": "192.168.2.195", "user": "chb"}, "msg": "Accepted
> +publickey for chb from 192.168.2.195 port 54161 ssh2", "hostname":
> +"sir", "facility": "authpriv", "level": "info", "date":
> +"2011-06-07T21:20:48+02:00"} </textarea>
> +
> +<p>You will need the following commands:</p> <textarea rows="10"
> +cols="60">$ModLoad omsnmp
> +
> +$ModLoad smjson.so
> +$template json,=SMJSON
> +$smjsonadd date,%TIMESTAMP:::date-rfc3339% $smjsonadd
> +level,%syslogpriority-text% $smjsonadd facility,%syslogfacility-text%
> +$smjsonadd hostname,%hostname% $smjsonadd msg,%msg% $smjsonadd
> +cee,%$!all-json% $smjsonadd type,SYSLOG
> +*.* /var/tmp/json.log;json
> +
> +</textarea>
> +
> +<p>The example above is using mmnormalize to normalize the log message
> +($!all-json).</p>
> +
> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a
> +href="manual.html">manual index</a>] [<a
> +href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font
> +size="2">This documentation is part of the <a
> +href="http://www.rsyslog.com/">rsyslog</a> project.<br> Copyright (c)
> +2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a>
> +and <a href="http://www.adiscon.com/">Adiscon</a>. Released under
> the
> +GNU GPL version 3 or higher.</font></p>
> +
> +</body></html>
> diff --git a/plugins/smjson/Makefile.am b/plugins/smjson/Makefile.am new
> file mode 100644 index 0000000..c0e9327
> --- /dev/null
> +++ b/plugins/smjson/Makefile.am
> @@ -0,0 +1,6 @@
> +pkglib_LTLIBRARIES = smjson.la
> +
> +smjson_la_SOURCES = smjson.c
> +smjson_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS)
> $(RSRT_CFLAGS)
> +$(LIBEE_CFLAGS) smjson_la_LDFLAGS = -module -avoid-version
> +$(LIBEE_LIBS) smjson_la_LIBADD =
> diff --git a/plugins/smjson/smjson.c b/plugins/smjson/smjson.c new file
> mode 100644 index 0000000..9d0b7d3
> --- /dev/null
> +++ b/plugins/smjson/smjson.c
> @@ -0,0 +1,262 @@
> +/* smjson.c
> + *
> + * A strgen module to transform log messages into the json format.
> + *
> + * File begun on 2011-06-02 by Christian Brunner <chb at muc.de>
> + *
> + * Rsyslog is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * Rsyslog is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with Rsyslog.  If not, see <http://www.gnu.org/licenses/>.
> + *
> + * A copy of the GPL can be found in the file "COPYING" in this
distribution.
> + */
> +
> +#include "config.h"
> +#include "rsyslog.h"
> +#include <stdlib.h>
> +#include <ctype.h>
> +#include <string.h>
> +#include <assert.h>
> +#include <errno.h>
> +#include "conf.h"
> +#include "syslogd-types.h"
> +#include "cfsysline.h"
> +#include "template.h"
> +#include "msg.h"
> +#include "module-template.h"
> +#include "unicode-helper.h"
> +#include "errmsg.h"
> +
> +#include <libee/libee.h>
> +#include <libee/field.h>
> +
> +MODULE_TYPE_STRGEN
> +MODULE_TYPE_NOKEEP
> +STRGEN_NAME("SMJSON")
> +
> +/* internal structures
> + */
> +DEF_SMOD_STATIC_DATA
> +DEFobjCurrIf(errmsg)
> +
> +typedef struct jsonentry_s {
> +	char *key;
> +	struct ee_field *constfield;
> +        struct templateEntry *pTpe;
> +        struct jsonentry_s *next;
> +} jsonentry_t;
> +
> +static jsonentry_t *root;
> +static ee_ctx ctx;
> +
> +static rsRetVal addJsonEntry(void __attribute__((unused)) *pVal, uchar
> +*pNewVal) {
> +	jsonentry_t *pNew;
> +	DEFiRet;
> +
> +	struct template *pTpl;
> +        struct templateEntry *pTpe;
> +
> +	char *key, *value;
> +	es_str_t *estr;
> +
> +	struct ee_value *eevalue;
> +
> +	key = (char *) pNewVal;
> +	if (!(value = strchr((char *) pNewVal, ','))) {
> +		errmsg.LogError(0, NO_ERRCODE, "error: key/value separator
> "
> +			"not found in '%s'", pNewVal);
> +                ABORT_FINALIZE(RS_RET_ERR);
> +	}
> +	*value = '\0';
> +	value++;
> +
> +	CHKmalloc(pNew = malloc(sizeof(jsonentry_t)));
> +	pNew->key = key;
> +
> +	if (value[0] == '%') {
> +		unsigned char *templateString;
> +		CHKmalloc(templateString = malloc(strlen(value) + 3));
> +		templateString[0] = '"';
> +		memcpy(templateString+1, value, strlen(value));
> +		memcpy(templateString+1+strlen(value), "\"", 2);
> +
> +		pTpl = tplAddLine("smjson-intern", &templateString);
> +		pTpe = pTpl->pEntryRoot;
> +
> +		pNew->constfield = NULL;
> +		pNew->pTpe = pTpe;
> +
> +		free(templateString);
> +	} else {
> +		pNew->constfield = ee_newField(ctx);
> +		pNew->constfield->name = es_newStrFromBuf(key,
> strlen(key));
> +		estr = es_newStrFromBuf(value, strlen(value));
> +		eevalue = ee_newValue(ctx);
> +		ee_setStrValue(eevalue, estr);
> +		ee_addValueToField(pNew->constfield, eevalue);
> +
> +		pNew->pTpe = NULL;
> +	}
> +
> +	pNew->next = root;
> +	root = pNew;
> +
> +	DBGPRINTF("smjson: key/value '%s':'%s' added.\n", key, value);
> +
> +finalize_it:
> +	if(iRet != RS_RET_OK) {
> +		free(pNewVal);
> +	}
> +
> +	RETiRet;
> +}
> +
> +
> +/* This strgen uses libee to generate the output string.
> + */
> +
> +#define JSON_END "\n"
> +BEGINstrgen
> +	es_str_t *json;
> +	es_str_t *estr;
> +	struct ee_field *field;
> +	struct ee_value *eevalue;
> +	propid_t propID;
> +	size_t propLen, jsonLen;
> +	uchar *pszProp = NULL;
> +	unsigned short bMustBeFreed = 0;
> +	char *jsonStr;
> +
> +	char *key;
> +
> +	jsonentry_t *iter;
> +        struct templateEntry *pTpe;
> +
> +	if((json = es_newStr(256)) == NULL) goto finalize_it;
> +
> +        es_addChar(&json, '{');
> +
> +	for(iter = root; iter != NULL; iter = iter->next)
> +	{
> +		key = iter->key;
> +
> +		if (iter->constfield) {
> +			ee_addField_JSON(iter->constfield, &json, 0);
> +
> +			goto nextiter;
> +		}
> +
> +		pTpe = iter->pTpe;
> +		propID = pTpe->data.field.propid;
> +
> +		if(propID == PROP_CEE_ALL_JSON) {
> +			char *ceestr;
> +
> +			es_addChar(&json, '"');
> +			es_addBuf(&json, key, strlen(key));
> +			es_addBuf(&json, "\":", 2);
> +			ee_fmtEventToJSON(pMsg->event, &estr);
> +			ceestr = es_str2cstr(estr, "#000");
> +			es_deleteStr(estr);
> +			es_addBuf(&json, ceestr, strlen(ceestr));
> +			free(ceestr);
> +
> +			goto nextiter;
> +		}
> +
> +		field = ee_newField(ctx);
> +		field->name = es_newStrFromBuf(key, strlen(key));
> +
> +		pszProp = MsgGetProp(pMsg, pTpe, propID,
> +			pTpe->data.field.propName, &propLen,
> &bMustBeFreed);
> +
> +		if(propID == PROP_MSG && pszProp[0] == ' ') {
> +			estr = es_newStrFromBuf((char *) pszProp+1,
> +				strlen((char *) pszProp)-1);
> +		} else {
> +			estr = es_newStrFromBuf((char *) pszProp,
> +				strlen((char *) pszProp));
> +		}
> +
> +		eevalue = ee_newValue(ctx);
> +		ee_setStrValue(eevalue, estr);
> +		ee_addValueToField(field, eevalue);
> +
> +		ee_addField_JSON(field, &json, 0);
> +		ee_deleteField(field);
> +
> +		if(bMustBeFreed) {
> +			free(pszProp);
> +			bMustBeFreed = 0;
> +		}
> +
> +nextiter:
> +		if (iter->next)
> +			es_addBuf(&json, ", ", ee_ctxIsEncUltraCompact(ctx)
> ? 1 : 2);
> +	}
> +
> +        es_addChar(&json, '}');
> +
> +	jsonStr = (char*) es_str2cstr(json, "#000");
> +	jsonLen = strlen(jsonStr);
> +
> +	es_deleteStr(json);
> +
> +        if((jsonLen + sizeof(JSON_END)) >= *pLenBuf)
> +                CHKiRet(ExtendBuf(ppBuf, pLenBuf, jsonLen +
> + sizeof(JSON_END)));
> +
> +	memcpy(*ppBuf, jsonStr, jsonLen);
> +	memcpy(*ppBuf + jsonLen, JSON_END, sizeof(JSON_END));
> +
> +	free(jsonStr);
> +	ee_exitCtx(ctx);
> +
> +finalize_it:
> +        if(bMustBeFreed)
> +                free(pszProp);
> +ENDstrgen
> +
> +BEGINmodExit
> +	jsonentry_t *iter, *jdel;
> +CODESTARTmodExit
> +	for(iter = root; iter != NULL; ) {
> +		jdel = iter;
> +		iter = iter->next;
> +		if (jdel->constfield) {
> +			ee_deleteField(jdel->constfield);
> +		}
> +		free(jdel->key);
> +	}
> +	objRelease(errmsg, CORE_COMPONENT);
> +ENDmodExit
> +
> +
> +BEGINqueryEtryPt
> +CODESTARTqueryEtryPt
> +CODEqueryEtryPt_STD_SMOD_QUERIES
> +ENDqueryEtryPt
> +
> +BEGINmodInit()
> +CODESTARTmodInit
> +        *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the
> +current interface specification */ CODEmodInit_QueryRegCFSLineHdlr
> +        CHKiRet(objUse(errmsg, CORE_COMPONENT));
> +
> +	root = NULL;
> +	ctx = ee_initCtx();
> +
> +        CHKiRet(omsdRegCFSLineHdlr((uchar *)"smjsonadd", 0,
> eCmdHdlrGetWord,
> +                addJsonEntry, NULL, STD_LOADABLE_MODULE_ID,
> + eConfObjGlobal));
> +
> +        dbgprintf("rsyslog json strgen init called, compiled with
> +version %s\n", VERSION); ENDmodInit
> --
> 1.7.1
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From tbergfeld at hq.adiscon.com  Wed Jun  8 17:45:58 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Wed, 8 Jun 2011 17:45:58 +0200
Subject: [rsyslog] rsyslog 5.9.0 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E0F@GRFEXC.intern.adiscon.com>

This version starts a new development branch for version v5 which brings some
new, not-so-intrusive features. This release includes enhancements for
imfile, support for TCP KEEP ALIVE and a way to name actions (primarily for
impstats).

Note that the main development activity is still focused at version 6.
So this release is only for those interested in the minor enhancements to v5.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-0-v5-devel/

Download:

http://www.rsyslog.com/rsyslog-5-9-0-devel/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From kaiwang.chen at gmail.com  Wed Jun  8 19:10:07 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Thu, 9 Jun 2011 01:10:07 +0800
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
	<BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>

2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Wednesday, June 08, 2011 4:04 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>
>> So the (o) option just affects the receiver, and would be of no harm being
>> turned on at the terminal end of syslog message flow. ?Actually, I am going
> to
>> use rsyslog on both ends, except bridge and router sources.
>
> Actually the sender. The receiver automatically handles both. The option is
> at the action, I think it is along the lines of
>
> @@(o)host


Yes, it's documented in "Remote Machine" section from
http://www.rsyslog.com/doc/rsyslog_conf_actions.html
However, when rsyslogd as sender was configured
*.*                @@(o)10.3.254.106:514

the receiver (same version 5.8.1) recorded nothing. I confirmed with
"tcpdump -i eth1 tcp port 514 -s0" that two data packets reached the
receiver with payload:

0000   37 38 20 3c 36 3e 4a 75 6e 20 20 39 20 30 30 3a  78 <6>Jun  9 00:
0010   34 34 3a 31 39 20 64 6e 73 31 20 6b 65 72 6e 65  44:19 dns1 kerne
0020   6c 3a 20 69 6d 6b 6c 6f 67 20 35 2e 38 2e 31 2c  l: imklog 5.8.1,
0030   20 6c 6f 67 20 73 6f 75 72 63 65 20 3d 20 2f 70   log source = /p
0040   72 6f 63 2f 6b 6d 73 67 20 73 74 61 72 74 65 64  roc/kmsg started
0050   2e                                               .

0000   31 33 32 20 3c 34 36 3e 4a 75 6e 20 20 39 20 30  132 <46>Jun  9 0
0010   30 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79  0:44:19 dns1 rsy
0020   73 6c 6f 67 64 3a 20 5b 6f 72 69 67 69 6e 20 73  slogd: [origin s
0030   6f 66 74 77 61 72 65 3d 22 72 73 79 73 6c 6f 67  oftware="rsyslog
0040   64 22 20 73 77 56 65 72 73 69 6f 6e 3d 22 35 2e  d" swVersion="5.
0050   38 2e 31 22 20 78 2d 70 69 64 3d 22 36 32 32 30  8.1" x-pid="6220
0060   22 20 78 2d 69 6e 66 6f 3d 22 68 74 74 70 3a 2f  " x-info="http:/
0070   2f 77 77 77 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d  /www.rsyslog.com
0080   22 5d 20 73 74 61 72 74 32 32 37 20 3c 34 33 3e  "] start227 <43>
0090   4a 75 6e 20 20 39 20 30 30 3a 34 34 3a 31 39 20  Jun  9 00:44:19
00a0   64 6e 73 31 20 72 73 79 73 6c 6f 67 64 2d 32 30  dns1 rsyslogd-20
00b0   36 36 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 6c 6f  66: could not lo
00c0   61 64 20 6d 6f 64 75 6c 65 20 27 2f 75 73 72 2f  ad module '/usr/
00d0   6c 69 62 36 34 2f 72 73 79 73 6c 6f 67 2f 6f 6d  lib64/rsyslog/om
00e0   68 64 66 73 2e 73 6f 27 2c 20 64 6c 6f 70 65 6e  hdfs.so', dlopen
00f0   3a 20 2f 75 73 72 2f 6c 69 62 36 34 2f 72 73 79  : /usr/lib64/rsy
0100   73 6c 6f 67 2f 6f 6d 68 64 66 73 2e 73 6f 3a 20  slog/omhdfs.so:
0110   63 61 6e 6e 6f 74 20 6f 70 65 6e 20 73 68 61 72  cannot open shar
0120   65 64 20 6f 62 6a 65 63 74 20 66 69 6c 65 3a 20  ed object file:
0130   4e 6f 20 73 75 63 68 20 66 69 6c 65 20 6f 72 20  No such file or
0140   64 69 72 65 63 74 6f 72 79 0a 20 5b 74 72 79 20  directory. [try
0150   68 74 74 70 3a 2f 2f 77 77 77 2e 72 73 79 73 6c  http://www.rsysl
0160   6f 67 2e 63 6f 6d 2f 65 2f 32 30 36 36 20 5d 31  og.com/e/2066 ]1
0170   30 34 20 3c 34 33 3e 4a 75 6e 20 20 39 20 30 30  04 <43>Jun  9 00
0180   3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 73  :44:19 dns1 rsys
0190   6c 6f 67 64 3a 20 74 68 65 20 6c 61 73 74 20 65  logd: the last e
01a0   72 72 6f 72 20 6f 63 63 75 72 65 64 20 69 6e 20  rror occured in
01b0   2f 65 74 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e  /etc/rsyslog.con
01c0   66 2c 20 6c 69 6e 65 20 33 3a 22 24 4d 6f 64 4c  f, line 3:"$ModL
01d0   6f 61 64 20 6f 6d 68 64 66 73 22 31 35 30 20 3c  oad omhdfs"150 <
01e0   34 33 3e 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a  43>Jun  9 00:44:
01f0   31 39 20 64 6e 73 31 20 72 73 79 73 6c 6f 67 64  19 dns1 rsyslogd
0200   2d 32 31 32 34 3a 20 43 4f 4e 46 49 47 20 45 52  -2124: CONFIG ER
0210   52 4f 52 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 69  ROR: could not i
0220   6e 74 65 72 70 72 65 74 20 6d 61 73 74 65 72 20  nterpret master
0230   63 6f 6e 66 69 67 20 66 69 6c 65 20 27 2f 65 74  config file '/et
0240   63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e 66 27 2e  c/rsyslog.conf'.
0250   20 5b 74 72 79 20 68 74 74 70 3a 2f 2f 77 77 77   [try http://www
0260   2e 72 73 79 73 6c 6f 67 2e 63 6f 6d 2f 65 2f 32  .rsyslog.com/e/2
0270   31 32 34 20 5d                                   124 ]


>
>>
>> I came across such a framing option days ago, and just can't locate it. How
> to
>> turn the (o) option, is it a compilation flag or a configuration directive?
>>
>> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I understand
> stock
>> sysklogd can't deal with multiline messages.
>
> Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
> (protocol)". This is what most applications understand under "TCP syslog". It
> uses \n to end a message and start a new one. Usually this is not a problem,
> as control character escaping removes \n in any case.

Will omrelp supress this problem, or is there any other way to get rid
of it, if plain tcp with (o) option does not work well?


Thanks,
Kaiwang

>
> Rainer
>>
>> Thanks,
>> Kaiwang
>>
>> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> > Multi-line messages are not supported by legacy plain syslogd. But you
>> > can turn on the (o) option, which enables octect-counted framing, with
>> > which it works. However, non-rsyslog receivers probably do not
>> > understand that framing.
>> > Rainer
>> >
>> >> -----Original Message-----
>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> >> Sent: Wednesday, June 08, 2011 3:16 PM
>> >> To: rsyslog-users
>> >> Subject: [rsyslog] Problem with forwarded multiline message
>> >>
>> >> Hello,
>> >>
>> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
>> >> and
>> > found that
>> >> the server interpreted a copy of forwarded multiline message(3rd
>> >> entry in the following raw messages) into multiple entries(3rd and
>> >> 4th entry in
>> > actual
>> >> output), while locally generated multiline message was fine. What's
>> >> the problem?
>> >>
>> >> The client setting:
>> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
>> >>
>> >> The server setting:
>> >> $InputPTCPServerRun 514
>> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template
>> >> rawfmt,"%rawmsg%\n"
>> >> *.* ? ?/var/log/rawmessages;rawfmt
>> >> *.info;mail.none;authpriv.none;cron.none
>> >> /var/log/messages
>> >>
>> >> Other settings were action queue tuning, I guess they were irrelevant.
>> >>
>> >> The raw messages:
>> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
>> > started.
>> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
>> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load module
>> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
>> >> cannot open shared object file: No such file or directory ?[try
>> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
>> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
>> >> interpret master config file '/etc/rsyslog.conf'. [try
>> >> http://www.rsyslog.com/e/2124 ]
>> >> imklog 5.8.1, log source = /proc/kmsg started.
>> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> >> x-info="http://www.rsyslog.com"] start could not load module
>> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
>> >> file
>> > or
>> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
>> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> >> CONFIG ERROR: could not interpret master config file
> '/etc/rsyslog.conf'.
>> > [try
>> >> http://www.rsyslog.com/e/2124 ]
>> >>
>> >> Actual ouput:
>> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1, log
>> >> source
>> > =
>> >> /proc/kmsg started.
>> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
>> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>> >> x-info="http://www.rsyslog.com"] start
>> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could not
>> >> load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
>> >> file
>> > or
>> >> directory
>> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
>> >> http://www.rsyslog.com/e/2066 ]
>> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last error
>> >> occured
>> > in
>> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
>> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> >> [try http://www.rsyslog.com/e/2124 ]
>> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1,
>> >> log
>> > source
>> >> = /proc/kmsg started.
>> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
>> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> >> x-info="http://www.rsyslog.com"] start
>> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could
>> >> not
>> > load
>> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
>> >> file
>> > or
>> >> directory ?[try http://www.rsyslog.com/e/2066 ]
>> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
>> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
>> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> >> [try http://www.rsyslog.com/e/2124 ]
>> >>
>> >>
>> >> Thanks,
>> >> Kaiwang
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rgerhards at hq.adiscon.com  Wed Jun  8 19:11:35 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 8 Jun 2011 19:11:35 +0200
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com><BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
	<BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E13@GRFEXC.intern.adiscon.com>

Lets keep focssed. Please provide me a debug log of the receiver.

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Wednesday, June 08, 2011 7:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Problem with forwarded multiline message
> 
> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> Sent: Wednesday, June 08, 2011 4:04 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Problem with forwarded multiline message
> >>
> >> So the (o) option just affects the receiver, and would be of no harm
> >> being turned on at the terminal end of syslog message flow.
> >> Actually, I am going
> > to
> >> use rsyslog on both ends, except bridge and router sources.
> >
> > Actually the sender. The receiver automatically handles both. The
> > option is at the action, I think it is along the lines of
> >
> > @@(o)host
> 
> 
> Yes, it's documented in "Remote Machine" section from
> http://www.rsyslog.com/doc/rsyslog_conf_actions.html
> However, when rsyslogd as sender was configured
> *.*                @@(o)10.3.254.106:514
> 
> the receiver (same version 5.8.1) recorded nothing. I confirmed with
> "tcpdump -i eth1 tcp port 514 -s0" that two data packets reached the
receiver
> with payload:
> 
> 0000   37 38 20 3c 36 3e 4a 75 6e 20 20 39 20 30 30 3a  78 <6>Jun  9 00:
> 0010   34 34 3a 31 39 20 64 6e 73 31 20 6b 65 72 6e 65  44:19 dns1 kerne
> 0020   6c 3a 20 69 6d 6b 6c 6f 67 20 35 2e 38 2e 31 2c  l: imklog 5.8.1,
> 0030   20 6c 6f 67 20 73 6f 75 72 63 65 20 3d 20 2f 70   log source = /p
> 0040   72 6f 63 2f 6b 6d 73 67 20 73 74 61 72 74 65 64  roc/kmsg started
> 0050   2e                                               .
> 
> 0000   31 33 32 20 3c 34 36 3e 4a 75 6e 20 20 39 20 30  132 <46>Jun  9 0
> 0010   30 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79  0:44:19 dns1 rsy
> 0020   73 6c 6f 67 64 3a 20 5b 6f 72 69 67 69 6e 20 73  slogd: [origin s
> 0030   6f 66 74 77 61 72 65 3d 22 72 73 79 73 6c 6f 67  oftware="rsyslog
> 0040   64 22 20 73 77 56 65 72 73 69 6f 6e 3d 22 35 2e  d" swVersion="5.
> 0050   38 2e 31 22 20 78 2d 70 69 64 3d 22 36 32 32 30  8.1" x-pid="6220
> 0060   22 20 78 2d 69 6e 66 6f 3d 22 68 74 74 70 3a 2f  " x-info="http:/
> 0070   2f 77 77 77 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d  /www.rsyslog.com
> 0080   22 5d 20 73 74 61 72 74 32 32 37 20 3c 34 33 3e  "] start227 <43>
> 0090   4a 75 6e 20 20 39 20 30 30 3a 34 34 3a 31 39 20  Jun  9 00:44:19
> 00a0   64 6e 73 31 20 72 73 79 73 6c 6f 67 64 2d 32 30  dns1 rsyslogd-20
> 00b0   36 36 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 6c 6f  66: could not lo
> 00c0   61 64 20 6d 6f 64 75 6c 65 20 27 2f 75 73 72 2f  ad module '/usr/
> 00d0   6c 69 62 36 34 2f 72 73 79 73 6c 6f 67 2f 6f 6d  lib64/rsyslog/om
> 00e0   68 64 66 73 2e 73 6f 27 2c 20 64 6c 6f 70 65 6e  hdfs.so', dlopen
> 00f0   3a 20 2f 75 73 72 2f 6c 69 62 36 34 2f 72 73 79  : /usr/lib64/rsy
> 0100   73 6c 6f 67 2f 6f 6d 68 64 66 73 2e 73 6f 3a 20  slog/omhdfs.so:
> 0110   63 61 6e 6e 6f 74 20 6f 70 65 6e 20 73 68 61 72  cannot open shar
> 0120   65 64 20 6f 62 6a 65 63 74 20 66 69 6c 65 3a 20  ed object file:
> 0130   4e 6f 20 73 75 63 68 20 66 69 6c 65 20 6f 72 20  No such file or
> 0140   64 69 72 65 63 74 6f 72 79 0a 20 5b 74 72 79 20  directory. [try
> 0150   68 74 74 70 3a 2f 2f 77 77 77 2e 72 73 79 73 6c  http://www.rsysl
> 0160   6f 67 2e 63 6f 6d 2f 65 2f 32 30 36 36 20 5d 31  og.com/e/2066 ]1
> 0170   30 34 20 3c 34 33 3e 4a 75 6e 20 20 39 20 30 30  04 <43>Jun  9 00
> 0180   3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 73  :44:19 dns1 rsys
> 0190   6c 6f 67 64 3a 20 74 68 65 20 6c 61 73 74 20 65  logd: the last e
> 01a0   72 72 6f 72 20 6f 63 63 75 72 65 64 20 69 6e 20  rror occured in
> 01b0   2f 65 74 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e  /etc/rsyslog.con
> 01c0   66 2c 20 6c 69 6e 65 20 33 3a 22 24 4d 6f 64 4c  f, line 3:"$ModL
> 01d0   6f 61 64 20 6f 6d 68 64 66 73 22 31 35 30 20 3c  oad omhdfs"150 <
> 01e0   34 33 3e 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a  43>Jun  9 00:44:
> 01f0   31 39 20 64 6e 73 31 20 72 73 79 73 6c 6f 67 64  19 dns1 rsyslogd
> 0200   2d 32 31 32 34 3a 20 43 4f 4e 46 49 47 20 45 52  -2124: CONFIG ER
> 0210   52 4f 52 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 69  ROR: could not i
> 0220   6e 74 65 72 70 72 65 74 20 6d 61 73 74 65 72 20  nterpret master
> 0230   63 6f 6e 66 69 67 20 66 69 6c 65 20 27 2f 65 74  config file '/et
> 0240   63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e 66 27 2e  c/rsyslog.conf'.
> 0250   20 5b 74 72 79 20 68 74 74 70 3a 2f 2f 77 77 77   [try http://www
> 0260   2e 72 73 79 73 6c 6f 67 2e 63 6f 6d 2f 65 2f 32  .rsyslog.com/e/2
> 0270   31 32 34 20 5d                                   124 ]
> 
> 
> >
> >>
> >> I came across such a framing option days ago, and just can't locate
> >> it. How
> > to
> >> turn the (o) option, is it a compilation flag or a configuration
directive?
> >>
> >> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I
> >> understand
> > stock
> >> sysklogd can't deal with multiline messages.
> >
> > Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
> > (protocol)". This is what most applications understand under "TCP
> > syslog". It uses \n to end a message and start a new one. Usually this
> > is not a problem, as control character escaping removes \n in any case.
> 
> Will omrelp supress this problem, or is there any other way to get rid of
it, if
> plain tcp with (o) option does not work well?
> 
> 
> Thanks,
> Kaiwang
> 
> >
> > Rainer
> >>
> >> Thanks,
> >> Kaiwang
> >>
> >> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> >> > Multi-line messages are not supported by legacy plain syslogd. But
> >> > you can turn on the (o) option, which enables octect-counted
> >> > framing, with which it works. However, non-rsyslog receivers
> >> > probably do not understand that framing.
> >> > Rainer
> >> >
> >> >> -----Original Message-----
> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> >> Sent: Wednesday, June 08, 2011 3:16 PM
> >> >> To: rsyslog-users
> >> >> Subject: [rsyslog] Problem with forwarded multiline message
> >> >>
> >> >> Hello,
> >> >>
> >> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
> >> >> and
> >> > found that
> >> >> the server interpreted a copy of forwarded multiline message(3rd
> >> >> entry in the following raw messages) into multiple entries(3rd and
> >> >> 4th entry in
> >> > actual
> >> >> output), while locally generated multiline message was fine.
> >> >> What's the problem?
> >> >>
> >> >> The client setting:
> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> >> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
> >> >>
> >> >> The server setting:
> >> >> $InputPTCPServerRun 514
> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
> >> >> $template rawfmt,"%rawmsg%\n"
> >> >> *.* ? ?/var/log/rawmessages;rawfmt
> >> >> *.info;mail.none;authpriv.none;cron.none
> >> >> /var/log/messages
> >> >>
> >> >> Other settings were action queue tuning, I guess they were
irrelevant.
> >> >>
> >> >> The raw messages:
> >> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source =
> >> >> /proc/kmsg
> >> > started.
> >> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
> >> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"]
> >> >> start <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load
> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so:
> >> >> cannot open shared object file: No such file or directory ?[try
> >> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
> >> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> >> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
> >> >> interpret master config file '/etc/rsyslog.conf'. [try
> >> >> http://www.rsyslog.com/e/2124 ]
> >> >> imklog 5.8.1, log source = /proc/kmsg started.
> >> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> >> >> x-info="http://www.rsyslog.com"] start could not load module
> >> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
> >> >> such file
> >> > or
> >> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
> >> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> >> >> CONFIG ERROR: could not interpret master config file
> > '/etc/rsyslog.conf'.
> >> > [try
> >> >> http://www.rsyslog.com/e/2124 ]
> >> >>
> >> >> Actual ouput:
> >> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1,
> >> >> log source
> >> > =
> >> >> /proc/kmsg started.
> >> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
> >> >> x-info="http://www.rsyslog.com"] start
> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could
> >> >> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
> >> >> such file
> >> > or
> >> >> directory
> >> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
> >> >> http://www.rsyslog.com/e/2066 ]
> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last
> >> >> error occured
> >> > in
> >> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> >> >> [try http://www.rsyslog.com/e/2124 ]
> >> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog
> >> >> 5.8.1, log
> >> > source
> >> >> = /proc/kmsg started.
> >> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
> >> >> x-info="http://www.rsyslog.com"] start
> >> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - -
> >> >> could not
> >> > load
> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
> >> >> such file
> >> > or
> >> >> directory ?[try http://www.rsyslog.com/e/2066 ]
> >> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
> >> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> >> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - -
> >> >> CONFIG
> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> >> >> [try http://www.rsyslog.com/e/2124 ]
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Kaiwang
> >> >> _______________________________________________
> >> >> rsyslog mailing list
> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> http://www.rsyslog.com
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> >
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From kaiwang.chen at gmail.com  Wed Jun  8 19:42:44 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Thu, 9 Jun 2011 01:42:44 +0800
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E13@GRFEXC.intern.adiscon.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
	<BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
	<BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E13@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTi=9C5KKsObJT9NeBdhCsfC0ovdE2w@mail.gmail.com>

Not sure what "debug log" is. I try to represent the problem with
packets and raw messages.

I repeated the steps with: 1) stopping sender, 2) starting tcpdump on
sender, 3) starting tcpdump on receiver,  4) starting sender with bad
configuration so that it compained with multiline message, 5) stopping
tcpdump on sender, 6) stopping tcpdump on receiver.

Two data packets reached receiver during step 2) and 5), with
printable text payload
(packet 1)
78 <6>Jun  9 01:23:53 dns1 kernel: imklog 5.8.1, log source =
/proc/kmsg started.
(packet 2)
132 <46>Jun  9 01:23:53 dns1 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.1" x-pid="6632" x-info="http://www.rsyslog.com"]
start68 <6>Jun  9 01:23:53 dns1 kernel: device eth1 entered
promiscuous mode227 <43>Jun  9 01:23:53 dns1 rsyslogd-2066: could not
load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
/usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
file or directory
 [try http://www.rsyslog.com/e/2066 ]104 <43>Jun  9 01:23:53 dns1
rsyslogd: the last error occured in /etc/rsyslog.conf, line
3:"$ModLoad omhdfs"150 <43>Jun  9 01:23:53 dns1 rsyslogd-2124: CONFIG
ERROR: could not interpret master config file '/etc/rsyslog.conf'.
[try http://www.rsyslog.com/e/2124 ]

In the meanwhile, the sender recorded formated messages:
<6>1 2011-06-09T01:23:53.994948+08:00 dns1 kernel - - - imklog 5.8.1,
log source = /proc/kmsg started.
<46>1 2011-06-09T01:23:53.995068+08:00 dns1 rsyslogd - - -  [origin
software="rsyslogd" swVersion="5.8.1" x-pid="6632"
x-info="http://www.rsyslog.com"] start
<6>1 2011-06-09T01:23:53.995255+08:00 dns1 kernel - - - device eth1
entered promiscuous mode
<43>1 2011-06-09T01:23:53.993615+08:00 dns1 rsyslogd-2066 - - - could
not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
/usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
file or directory
 [try http://www.rsyslog.com/e/2066 ]
<43>1 2011-06-09T01:23:53.993692+08:00 dns1 rsyslogd - - - the last
error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
<43>1 2011-06-09T01:23:53.994860+08:00 dns1 rsyslogd-2124 - - - CONFIG
ERROR: could not interpret master config file '/etc/rsyslog.conf'.
[try http://www.rsyslog.com/e/2124 ]
<6>1 2011-06-09T01:24:41.727433+08:00 dns1 kernel - - - device eth1
left promiscuous mode

The raw messages on receiver:
device eth1 entered promiscuous mode
<6>Jun  9 01:23:53 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg started.
<6>Jun  9 01:24:41 dns1 kernel: device eth1 left promiscuous mode
device eth1 left promiscuous mode

The missing packet should be after 2nd raw message and before 3rd one.
Obviously the first packet(2nd raw message) reached the receiver and
was recorded. The second packet reached receiver and was not recorded.


Thanks,
Kaiwang

2011/6/9 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Lets keep focssed. Please provide me a debug log of the receiver.
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Wednesday, June 08, 2011 7:10 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>
>> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> >
>> >> -----Original Message-----
>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> >> Sent: Wednesday, June 08, 2011 4:04 PM
>> >> To: rsyslog-users
>> >> Subject: Re: [rsyslog] Problem with forwarded multiline message
>> >>
>> >> So the (o) option just affects the receiver, and would be of no harm
>> >> being turned on at the terminal end of syslog message flow.
>> >> Actually, I am going
>> > to
>> >> use rsyslog on both ends, except bridge and router sources.
>> >
>> > Actually the sender. The receiver automatically handles both. The
>> > option is at the action, I think it is along the lines of
>> >
>> > @@(o)host
>>
>>
>> Yes, it's documented in "Remote Machine" section from
>> http://www.rsyslog.com/doc/rsyslog_conf_actions.html
>> However, when rsyslogd as sender was configured
>> *.* ? ? ? ? ? ? ? ?@@(o)10.3.254.106:514
>>
>> the receiver (same version 5.8.1) recorded nothing. I confirmed with
>> "tcpdump -i eth1 tcp port 514 -s0" that two data packets reached the
> receiver
>> with payload:
>>
>> 0000 ? 37 38 20 3c 36 3e 4a 75 6e 20 20 39 20 30 30 3a ?78 <6>Jun ?9 00:
>> 0010 ? 34 34 3a 31 39 20 64 6e 73 31 20 6b 65 72 6e 65 ?44:19 dns1 kerne
>> 0020 ? 6c 3a 20 69 6d 6b 6c 6f 67 20 35 2e 38 2e 31 2c ?l: imklog 5.8.1,
>> 0030 ? 20 6c 6f 67 20 73 6f 75 72 63 65 20 3d 20 2f 70 ? log source = /p
>> 0040 ? 72 6f 63 2f 6b 6d 73 67 20 73 74 61 72 74 65 64 ?roc/kmsg started
>> 0050 ? 2e ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? .
>>
>> 0000 ? 31 33 32 20 3c 34 36 3e 4a 75 6e 20 20 39 20 30 ?132 <46>Jun ?9 0
>> 0010 ? 30 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 ?0:44:19 dns1 rsy
>> 0020 ? 73 6c 6f 67 64 3a 20 5b 6f 72 69 67 69 6e 20 73 ?slogd: [origin s
>> 0030 ? 6f 66 74 77 61 72 65 3d 22 72 73 79 73 6c 6f 67 ?oftware="rsyslog
>> 0040 ? 64 22 20 73 77 56 65 72 73 69 6f 6e 3d 22 35 2e ?d" swVersion="5.
>> 0050 ? 38 2e 31 22 20 78 2d 70 69 64 3d 22 36 32 32 30 ?8.1" x-pid="6220
>> 0060 ? 22 20 78 2d 69 6e 66 6f 3d 22 68 74 74 70 3a 2f ?" x-info="http:/
>> 0070 ? 2f 77 77 77 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d ?/www.rsyslog.com
>> 0080 ? 22 5d 20 73 74 61 72 74 32 32 37 20 3c 34 33 3e ?"] start227 <43>
>> 0090 ? 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a 31 39 20 ?Jun ?9 00:44:19
>> 00a0 ? 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 2d 32 30 ?dns1 rsyslogd-20
>> 00b0 ? 36 36 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 6c 6f ?66: could not lo
>> 00c0 ? 61 64 20 6d 6f 64 75 6c 65 20 27 2f 75 73 72 2f ?ad module '/usr/
>> 00d0 ? 6c 69 62 36 34 2f 72 73 79 73 6c 6f 67 2f 6f 6d ?lib64/rsyslog/om
>> 00e0 ? 68 64 66 73 2e 73 6f 27 2c 20 64 6c 6f 70 65 6e ?hdfs.so', dlopen
>> 00f0 ? 3a 20 2f 75 73 72 2f 6c 69 62 36 34 2f 72 73 79 ?: /usr/lib64/rsy
>> 0100 ? 73 6c 6f 67 2f 6f 6d 68 64 66 73 2e 73 6f 3a 20 ?slog/omhdfs.so:
>> 0110 ? 63 61 6e 6e 6f 74 20 6f 70 65 6e 20 73 68 61 72 ?cannot open shar
>> 0120 ? 65 64 20 6f 62 6a 65 63 74 20 66 69 6c 65 3a 20 ?ed object file:
>> 0130 ? 4e 6f 20 73 75 63 68 20 66 69 6c 65 20 6f 72 20 ?No such file or
>> 0140 ? 64 69 72 65 63 74 6f 72 79 0a 20 5b 74 72 79 20 ?directory. [try
>> 0150 ? 68 74 74 70 3a 2f 2f 77 77 77 2e 72 73 79 73 6c ?http://www.rsysl
>> 0160 ? 6f 67 2e 63 6f 6d 2f 65 2f 32 30 36 36 20 5d 31 ?og.com/e/2066 ]1
>> 0170 ? 30 34 20 3c 34 33 3e 4a 75 6e 20 20 39 20 30 30 ?04 <43>Jun ?9 00
>> 0180 ? 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 73 ?:44:19 dns1 rsys
>> 0190 ? 6c 6f 67 64 3a 20 74 68 65 20 6c 61 73 74 20 65 ?logd: the last e
>> 01a0 ? 72 72 6f 72 20 6f 63 63 75 72 65 64 20 69 6e 20 ?rror occured in
>> 01b0 ? 2f 65 74 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e ?/etc/rsyslog.con
>> 01c0 ? 66 2c 20 6c 69 6e 65 20 33 3a 22 24 4d 6f 64 4c ?f, line 3:"$ModL
>> 01d0 ? 6f 61 64 20 6f 6d 68 64 66 73 22 31 35 30 20 3c ?oad omhdfs"150 <
>> 01e0 ? 34 33 3e 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a ?43>Jun ?9 00:44:
>> 01f0 ? 31 39 20 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 ?19 dns1 rsyslogd
>> 0200 ? 2d 32 31 32 34 3a 20 43 4f 4e 46 49 47 20 45 52 ?-2124: CONFIG ER
>> 0210 ? 52 4f 52 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 69 ?ROR: could not i
>> 0220 ? 6e 74 65 72 70 72 65 74 20 6d 61 73 74 65 72 20 ?nterpret master
>> 0230 ? 63 6f 6e 66 69 67 20 66 69 6c 65 20 27 2f 65 74 ?config file '/et
>> 0240 ? 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e 66 27 2e ?c/rsyslog.conf'.
>> 0250 ? 20 5b 74 72 79 20 68 74 74 70 3a 2f 2f 77 77 77 ? [try http://www
>> 0260 ? 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d 2f 65 2f 32 ?.rsyslog.com/e/2
>> 0270 ? 31 32 34 20 5d ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 124 ]
>>
>>
>> >
>> >>
>> >> I came across such a framing option days ago, and just can't locate
>> >> it. How
>> > to
>> >> turn the (o) option, is it a compilation flag or a configuration
> directive?
>> >>
>> >> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I
>> >> understand
>> > stock
>> >> sysklogd can't deal with multiline messages.
>> >
>> > Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
>> > (protocol)". This is what most applications understand under "TCP
>> > syslog". It uses \n to end a message and start a new one. Usually this
>> > is not a problem, as control character escaping removes \n in any case.
>>
>> Will omrelp supress this problem, or is there any other way to get rid of
> it, if
>> plain tcp with (o) option does not work well?
>>
>>
>> Thanks,
>> Kaiwang
>>
>> >
>> > Rainer
>> >>
>> >> Thanks,
>> >> Kaiwang
>> >>
>> >> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> >> > Multi-line messages are not supported by legacy plain syslogd. But
>> >> > you can turn on the (o) option, which enables octect-counted
>> >> > framing, with which it works. However, non-rsyslog receivers
>> >> > probably do not understand that framing.
>> >> > Rainer
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> >> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> >> >> Sent: Wednesday, June 08, 2011 3:16 PM
>> >> >> To: rsyslog-users
>> >> >> Subject: [rsyslog] Problem with forwarded multiline message
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
>> >> >> and
>> >> > found that
>> >> >> the server interpreted a copy of forwarded multiline message(3rd
>> >> >> entry in the following raw messages) into multiple entries(3rd and
>> >> >> 4th entry in
>> >> > actual
>> >> >> output), while locally generated multiline message was fine.
>> >> >> What's the problem?
>> >> >>
>> >> >> The client setting:
>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> >> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
>> >> >>
>> >> >> The server setting:
>> >> >> $InputPTCPServerRun 514
>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> >> >> $template rawfmt,"%rawmsg%\n"
>> >> >> *.* ? ?/var/log/rawmessages;rawfmt
>> >> >> *.info;mail.none;authpriv.none;cron.none
>> >> >> /var/log/messages
>> >> >>
>> >> >> Other settings were action queue tuning, I guess they were
> irrelevant.
>> >> >>
>> >> >> The raw messages:
>> >> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source =
>> >> >> /proc/kmsg
>> >> > started.
>> >> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>> >> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"]
>> >> >> start <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load
>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so:
>> >> >> cannot open shared object file: No such file or directory ?[try
>> >> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
>> >> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> >> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
>> >> >> interpret master config file '/etc/rsyslog.conf'. [try
>> >> >> http://www.rsyslog.com/e/2124 ]
>> >> >> imklog 5.8.1, log source = /proc/kmsg started.
>> >> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> >> >> x-info="http://www.rsyslog.com"] start could not load module
>> >> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>> >> >> such file
>> >> > or
>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
>> >> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> >> >> CONFIG ERROR: could not interpret master config file
>> > '/etc/rsyslog.conf'.
>> >> > [try
>> >> >> http://www.rsyslog.com/e/2124 ]
>> >> >>
>> >> >> Actual ouput:
>> >> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1,
>> >> >> log source
>> >> > =
>> >> >> /proc/kmsg started.
>> >> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>> >> >> x-info="http://www.rsyslog.com"] start
>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could
>> >> >> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>> >> >> such file
>> >> > or
>> >> >> directory
>> >> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
>> >> >> http://www.rsyslog.com/e/2066 ]
>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last
>> >> >> error occured
>> >> > in
>> >> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> >> >> [try http://www.rsyslog.com/e/2124 ]
>> >> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog
>> >> >> 5.8.1, log
>> >> > source
>> >> >> = /proc/kmsg started.
>> >> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> >> >> x-info="http://www.rsyslog.com"] start
>> >> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - -
>> >> >> could not
>> >> > load
>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>> >> >> such file
>> >> > or
>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ]
>> >> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
>> >> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> >> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - -
>> >> >> CONFIG
>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> >> >> [try http://www.rsyslog.com/e/2124 ]
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Kaiwang
>> >> >> _______________________________________________
>> >> >> rsyslog mailing list
>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> http://www.rsyslog.com
>> >> > _______________________________________________
>> >> > rsyslog mailing list
>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> > http://www.rsyslog.com
>> >> >
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Wed Jun  8 20:18:57 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Thu, 9 Jun 2011 02:18:57 +0800
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <BANLkTi=9C5KKsObJT9NeBdhCsfC0ovdE2w@mail.gmail.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
	<BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
	<BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E13@GRFEXC.intern.adiscon.com>
	<BANLkTi=9C5KKsObJT9NeBdhCsfC0ovdE2w@mail.gmail.com>
Message-ID: <BANLkTimECXNT=DYnOK6RZS5To4uTE1D-DQ@mail.gmail.com>

Sorry, it's false alarm. I noticed that the problem occured regardless
of (o) option. After a few check, I found myself was testing new
directives:

$ModLoad ommail

$ActionMailSMTPServer ....
$ActionMailFrom ....
$ActionMailTo ....
$template mailSubject,"test problem on %HOSTNAME%"
$template mailBody,"RSYSLOG Alert\r\nmsg='%rawmsg%'"
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 1
if $msg contains 'started' then :ommail:;mailBody

With directives related to ommail commented out, the (o) option worked
well and recorded:
<43>Jun  9 02:06:48 dns1 rsyslogd-2066: could not load module
'/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
cannot open shared object file: No such file or directory#012 [try
http://www.rsyslog.com/e/2066 ]

However, I am not sure why ommail suppressed the recording of the
second packet. The full configuration of receiver is

$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format

$ModLoad imklog
$ModLoad imuxsock
$ModLoad imudp
$ModLoad imptcp
$ModLoad ommysql
$ModLoad impstats

#$ModLoad ommail

#$ActionMailSMTPServer ....
#$ActionMailFrom ....
#$ActionMailTo ....
#$template mailSubject,"test problem on %HOSTNAME%"
#$template mailBody,"RSYSLOG Alert\r\nmsg='%rawmsg%'"
#$ActionMailSubject mailSubject
#$ActionExecOnlyOnceEveryInterval 1
#if $msg contains 'started' then :ommail:;mailBody

$PStatInterval 600
$PStatSeverity 7

$InputPTCPServerNotifyOnConnectionClose on

$UDPServerRun 514
$InputPTCPServerRun 514

$MainMsgQueueSaveOnShutdown on
$MainMsgQueueFileName mq
$MainMsgQueueMaxFileSize 5m

$template DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
*.*                                                  ?DynFile

$WorkDirectory /var/spool/rsyslog

$ActionQueueType LinkedList
$ActionQueueSize 200000
$ActionQueueDiscardMark 190000
$ActionQueueHighWaterMark 160000
$ActionQueueLowWaterMark 40000
$ActionQueueFileName dbq
$ActionQueueSaveOnShutdown on
$ActionQueueMaxFileSize 128m
$ActionResumeRetryCount -1
$actionommysqlserverport 6666
*.*       :ommysql:10.3.254.109,syslog,syslogwriter,tops3cr3t

syslog.debug  /var/log/rsyslog-stats


$template rawfmt,"%rawmsg%\n"
*.*    /var/log/rawmessages;rawfmt

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


Thanks,
Kaiwang


2011/6/9 Kaiwang Chen <kaiwang.chen at gmail.com>:
> Not sure what "debug log" is. I try to represent the problem with
> packets and raw messages.
>
> I repeated the steps with: 1) stopping sender, 2) starting tcpdump on
> sender, 3) starting tcpdump on receiver, ?4) starting sender with bad
> configuration so that it compained with multiline message, 5) stopping
> tcpdump on sender, 6) stopping tcpdump on receiver.
>
> Two data packets reached receiver during step 2) and 5), with
> printable text payload
> (packet 1)
> 78 <6>Jun ?9 01:23:53 dns1 kernel: imklog 5.8.1, log source =
> /proc/kmsg started.
> (packet 2)
> 132 <46>Jun ?9 01:23:53 dns1 rsyslogd: [origin software="rsyslogd"
> swVersion="5.8.1" x-pid="6632" x-info="http://www.rsyslog.com"]
> start68 <6>Jun ?9 01:23:53 dns1 kernel: device eth1 entered
> promiscuous mode227 <43>Jun ?9 01:23:53 dns1 rsyslogd-2066: could not
> load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
> file or directory
> ?[try http://www.rsyslog.com/e/2066 ]104 <43>Jun ?9 01:23:53 dns1
> rsyslogd: the last error occured in /etc/rsyslog.conf, line
> 3:"$ModLoad omhdfs"150 <43>Jun ?9 01:23:53 dns1 rsyslogd-2124: CONFIG
> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try http://www.rsyslog.com/e/2124 ]
>
> In the meanwhile, the sender recorded formated messages:
> <6>1 2011-06-09T01:23:53.994948+08:00 dns1 kernel - - - imklog 5.8.1,
> log source = /proc/kmsg started.
> <46>1 2011-06-09T01:23:53.995068+08:00 dns1 rsyslogd - - - ?[origin
> software="rsyslogd" swVersion="5.8.1" x-pid="6632"
> x-info="http://www.rsyslog.com"] start
> <6>1 2011-06-09T01:23:53.995255+08:00 dns1 kernel - - - device eth1
> entered promiscuous mode
> <43>1 2011-06-09T01:23:53.993615+08:00 dns1 rsyslogd-2066 - - - could
> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
> file or directory
> ?[try http://www.rsyslog.com/e/2066 ]
> <43>1 2011-06-09T01:23:53.993692+08:00 dns1 rsyslogd - - - the last
> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> <43>1 2011-06-09T01:23:53.994860+08:00 dns1 rsyslogd-2124 - - - CONFIG
> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try http://www.rsyslog.com/e/2124 ]
> <6>1 2011-06-09T01:24:41.727433+08:00 dns1 kernel - - - device eth1
> left promiscuous mode
>
> The raw messages on receiver:
> device eth1 entered promiscuous mode
> <6>Jun ?9 01:23:53 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg started.
> <6>Jun ?9 01:24:41 dns1 kernel: device eth1 left promiscuous mode
> device eth1 left promiscuous mode
>
> The missing packet should be after 2nd raw message and before 3rd one.
> Obviously the first packet(2nd raw message) reached the receiver and
> was recorded. The second packet reached receiver and was not recorded.
>
>
> Thanks,
> Kaiwang
>
> 2011/6/9 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>> Lets keep focssed. Please provide me a debug log of the receiver.
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>> Sent: Wednesday, June 08, 2011 7:10 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>>
>>> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>> >
>>> >> -----Original Message-----
>>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>> >> Sent: Wednesday, June 08, 2011 4:04 PM
>>> >> To: rsyslog-users
>>> >> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>> >>
>>> >> So the (o) option just affects the receiver, and would be of no harm
>>> >> being turned on at the terminal end of syslog message flow.
>>> >> Actually, I am going
>>> > to
>>> >> use rsyslog on both ends, except bridge and router sources.
>>> >
>>> > Actually the sender. The receiver automatically handles both. The
>>> > option is at the action, I think it is along the lines of
>>> >
>>> > @@(o)host
>>>
>>>
>>> Yes, it's documented in "Remote Machine" section from
>>> http://www.rsyslog.com/doc/rsyslog_conf_actions.html
>>> However, when rsyslogd as sender was configured
>>> *.* ? ? ? ? ? ? ? ?@@(o)10.3.254.106:514
>>>
>>> the receiver (same version 5.8.1) recorded nothing. I confirmed with
>>> "tcpdump -i eth1 tcp port 514 -s0" that two data packets reached the
>> receiver
>>> with payload:
>>>
>>> 0000 ? 37 38 20 3c 36 3e 4a 75 6e 20 20 39 20 30 30 3a ?78 <6>Jun ?9 00:
>>> 0010 ? 34 34 3a 31 39 20 64 6e 73 31 20 6b 65 72 6e 65 ?44:19 dns1 kerne
>>> 0020 ? 6c 3a 20 69 6d 6b 6c 6f 67 20 35 2e 38 2e 31 2c ?l: imklog 5.8.1,
>>> 0030 ? 20 6c 6f 67 20 73 6f 75 72 63 65 20 3d 20 2f 70 ? log source = /p
>>> 0040 ? 72 6f 63 2f 6b 6d 73 67 20 73 74 61 72 74 65 64 ?roc/kmsg started
>>> 0050 ? 2e ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? .
>>>
>>> 0000 ? 31 33 32 20 3c 34 36 3e 4a 75 6e 20 20 39 20 30 ?132 <46>Jun ?9 0
>>> 0010 ? 30 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 ?0:44:19 dns1 rsy
>>> 0020 ? 73 6c 6f 67 64 3a 20 5b 6f 72 69 67 69 6e 20 73 ?slogd: [origin s
>>> 0030 ? 6f 66 74 77 61 72 65 3d 22 72 73 79 73 6c 6f 67 ?oftware="rsyslog
>>> 0040 ? 64 22 20 73 77 56 65 72 73 69 6f 6e 3d 22 35 2e ?d" swVersion="5.
>>> 0050 ? 38 2e 31 22 20 78 2d 70 69 64 3d 22 36 32 32 30 ?8.1" x-pid="6220
>>> 0060 ? 22 20 78 2d 69 6e 66 6f 3d 22 68 74 74 70 3a 2f ?" x-info="http:/
>>> 0070 ? 2f 77 77 77 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d ?/www.rsyslog.com
>>> 0080 ? 22 5d 20 73 74 61 72 74 32 32 37 20 3c 34 33 3e ?"] start227 <43>
>>> 0090 ? 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a 31 39 20 ?Jun ?9 00:44:19
>>> 00a0 ? 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 2d 32 30 ?dns1 rsyslogd-20
>>> 00b0 ? 36 36 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 6c 6f ?66: could not lo
>>> 00c0 ? 61 64 20 6d 6f 64 75 6c 65 20 27 2f 75 73 72 2f ?ad module '/usr/
>>> 00d0 ? 6c 69 62 36 34 2f 72 73 79 73 6c 6f 67 2f 6f 6d ?lib64/rsyslog/om
>>> 00e0 ? 68 64 66 73 2e 73 6f 27 2c 20 64 6c 6f 70 65 6e ?hdfs.so', dlopen
>>> 00f0 ? 3a 20 2f 75 73 72 2f 6c 69 62 36 34 2f 72 73 79 ?: /usr/lib64/rsy
>>> 0100 ? 73 6c 6f 67 2f 6f 6d 68 64 66 73 2e 73 6f 3a 20 ?slog/omhdfs.so:
>>> 0110 ? 63 61 6e 6e 6f 74 20 6f 70 65 6e 20 73 68 61 72 ?cannot open shar
>>> 0120 ? 65 64 20 6f 62 6a 65 63 74 20 66 69 6c 65 3a 20 ?ed object file:
>>> 0130 ? 4e 6f 20 73 75 63 68 20 66 69 6c 65 20 6f 72 20 ?No such file or
>>> 0140 ? 64 69 72 65 63 74 6f 72 79 0a 20 5b 74 72 79 20 ?directory. [try
>>> 0150 ? 68 74 74 70 3a 2f 2f 77 77 77 2e 72 73 79 73 6c ?http://www.rsysl
>>> 0160 ? 6f 67 2e 63 6f 6d 2f 65 2f 32 30 36 36 20 5d 31 ?og.com/e/2066 ]1
>>> 0170 ? 30 34 20 3c 34 33 3e 4a 75 6e 20 20 39 20 30 30 ?04 <43>Jun ?9 00
>>> 0180 ? 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 73 ?:44:19 dns1 rsys
>>> 0190 ? 6c 6f 67 64 3a 20 74 68 65 20 6c 61 73 74 20 65 ?logd: the last e
>>> 01a0 ? 72 72 6f 72 20 6f 63 63 75 72 65 64 20 69 6e 20 ?rror occured in
>>> 01b0 ? 2f 65 74 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e ?/etc/rsyslog.con
>>> 01c0 ? 66 2c 20 6c 69 6e 65 20 33 3a 22 24 4d 6f 64 4c ?f, line 3:"$ModL
>>> 01d0 ? 6f 61 64 20 6f 6d 68 64 66 73 22 31 35 30 20 3c ?oad omhdfs"150 <
>>> 01e0 ? 34 33 3e 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a ?43>Jun ?9 00:44:
>>> 01f0 ? 31 39 20 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 ?19 dns1 rsyslogd
>>> 0200 ? 2d 32 31 32 34 3a 20 43 4f 4e 46 49 47 20 45 52 ?-2124: CONFIG ER
>>> 0210 ? 52 4f 52 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 69 ?ROR: could not i
>>> 0220 ? 6e 74 65 72 70 72 65 74 20 6d 61 73 74 65 72 20 ?nterpret master
>>> 0230 ? 63 6f 6e 66 69 67 20 66 69 6c 65 20 27 2f 65 74 ?config file '/et
>>> 0240 ? 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e 66 27 2e ?c/rsyslog.conf'.
>>> 0250 ? 20 5b 74 72 79 20 68 74 74 70 3a 2f 2f 77 77 77 ? [try http://www
>>> 0260 ? 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d 2f 65 2f 32 ?.rsyslog.com/e/2
>>> 0270 ? 31 32 34 20 5d ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 124 ]
>>>
>>>
>>> >
>>> >>
>>> >> I came across such a framing option days ago, and just can't locate
>>> >> it. How
>>> > to
>>> >> turn the (o) option, is it a compilation flag or a configuration
>> directive?
>>> >>
>>> >> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I
>>> >> understand
>>> > stock
>>> >> sysklogd can't deal with multiline messages.
>>> >
>>> > Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
>>> > (protocol)". This is what most applications understand under "TCP
>>> > syslog". It uses \n to end a message and start a new one. Usually this
>>> > is not a problem, as control character escaping removes \n in any case.
>>>
>>> Will omrelp supress this problem, or is there any other way to get rid of
>> it, if
>>> plain tcp with (o) option does not work well?
>>>
>>>
>>> Thanks,
>>> Kaiwang
>>>
>>> >
>>> > Rainer
>>> >>
>>> >> Thanks,
>>> >> Kaiwang
>>> >>
>>> >> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>> >> > Multi-line messages are not supported by legacy plain syslogd. But
>>> >> > you can turn on the (o) option, which enables octect-counted
>>> >> > framing, with which it works. However, non-rsyslog receivers
>>> >> > probably do not understand that framing.
>>> >> > Rainer
>>> >> >
>>> >> >> -----Original Message-----
>>> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> >> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>> >> >> Sent: Wednesday, June 08, 2011 3:16 PM
>>> >> >> To: rsyslog-users
>>> >> >> Subject: [rsyslog] Problem with forwarded multiline message
>>> >> >>
>>> >> >> Hello,
>>> >> >>
>>> >> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
>>> >> >> and
>>> >> > found that
>>> >> >> the server interpreted a copy of forwarded multiline message(3rd
>>> >> >> entry in the following raw messages) into multiple entries(3rd and
>>> >> >> 4th entry in
>>> >> > actual
>>> >> >> output), while locally generated multiline message was fine.
>>> >> >> What's the problem?
>>> >> >>
>>> >> >> The client setting:
>>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>> >> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
>>> >> >>
>>> >> >> The server setting:
>>> >> >> $InputPTCPServerRun 514
>>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>> >> >> $template rawfmt,"%rawmsg%\n"
>>> >> >> *.* ? ?/var/log/rawmessages;rawfmt
>>> >> >> *.info;mail.none;authpriv.none;cron.none
>>> >> >> /var/log/messages
>>> >> >>
>>> >> >> Other settings were action queue tuning, I guess they were
>> irrelevant.
>>> >> >>
>>> >> >> The raw messages:
>>> >> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source =
>>> >> >> /proc/kmsg
>>> >> > started.
>>> >> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>>> >> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"]
>>> >> >> start <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load
>>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>> /usr/lib64/rsyslog/omhdfs.so:
>>> >> >> cannot open shared object file: No such file or directory ?[try
>>> >> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
>>> >> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>>> >> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
>>> >> >> interpret master config file '/etc/rsyslog.conf'. [try
>>> >> >> http://www.rsyslog.com/e/2124 ]
>>> >> >> imklog 5.8.1, log source = /proc/kmsg started.
>>> >> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>>> >> >> x-info="http://www.rsyslog.com"] start could not load module
>>> >> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>> >> >> such file
>>> >> > or
>>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
>>> >> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>>> >> >> CONFIG ERROR: could not interpret master config file
>>> > '/etc/rsyslog.conf'.
>>> >> > [try
>>> >> >> http://www.rsyslog.com/e/2124 ]
>>> >> >>
>>> >> >> Actual ouput:
>>> >> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1,
>>> >> >> log source
>>> >> > =
>>> >> >> /proc/kmsg started.
>>> >> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
>>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>>> >> >> x-info="http://www.rsyslog.com"] start
>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could
>>> >> >> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>> >> >> such file
>>> >> > or
>>> >> >> directory
>>> >> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
>>> >> >> http://www.rsyslog.com/e/2066 ]
>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last
>>> >> >> error occured
>>> >> > in
>>> >> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
>>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>>> >> >> [try http://www.rsyslog.com/e/2124 ]
>>> >> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog
>>> >> >> 5.8.1, log
>>> >> > source
>>> >> >> = /proc/kmsg started.
>>> >> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
>>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>>> >> >> x-info="http://www.rsyslog.com"] start
>>> >> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - -
>>> >> >> could not
>>> >> > load
>>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>> >> >> such file
>>> >> > or
>>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ]
>>> >> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
>>> >> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>>> >> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - -
>>> >> >> CONFIG
>>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>>> >> >> [try http://www.rsyslog.com/e/2124 ]
>>> >> >>
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Kaiwang
>>> >> >> _______________________________________________
>>> >> >> rsyslog mailing list
>>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> >> >> http://www.rsyslog.com
>>> >> > _______________________________________________
>>> >> > rsyslog mailing list
>>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> >> > http://www.rsyslog.com
>>> >> >
>>> >> _______________________________________________
>>> >> rsyslog mailing list
>>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> >> http://www.rsyslog.com
>>> > _______________________________________________
>>> > rsyslog mailing list
>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> > http://www.rsyslog.com
>>> >
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>
>

From chb at muc.de  Wed Jun  8 20:47:39 2011
From: chb at muc.de (Christian Brunner)
Date: Wed, 8 Jun 2011 20:47:39 +0200
Subject: [rsyslog] [PATCH] a json strgen module
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E0E@GRFEXC.intern.adiscon.com>
References: <20110607194305.GA30320@sir.fritz.box>
	<9B6E2A8877C38245BFB15CC491A11DA7280E0E@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTikdKc2dJriCsT-3ZecqmAuQwTf4Sg@mail.gmail.com>

No hurry here. - I will be offline for a few days, too.

Yes, I've seen all-json. The module even has some extra code to format
the all-json property.

As far as I can tell all-json only prints the elements detected with
the normalizer. I wanted to have the ability to print data that is not
directly available in the message (e.g. the hostname).

Christian

2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> Sorry, I am a bit swamped, thus the sluggish reply. Will be on the road most
> of tomorrow and Friday as well (with Monday being a public holiday over
> here). This sounds interesting. I just want to make sure you have seen the
> all-json property which generates a json format. I am not sure if that is
> useful for your case... Will have a better review asap.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Christian Brunner
>> Sent: Tuesday, June 07, 2011 9:43 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] [PATCH] a json strgen module
>>
>> I was playing with liblognorm and a document-oriented database that is
>> taking input in json format. Therefore I wanted to generate the json
>> messages directly in rsyslog.
>>
>> With this module you can generate json messages and send it through any of
>> the rsyslog output modules.
>>
>> The module isn't completely finished yet, but it is working and I would
> like to
>> get some feedback.
>>
>> Thanks
>> Christian
>> ---
>> ?Makefile.am ? ? ? ? ? ? ? ?| ? ?5 +
>> ?configure.ac ? ? ? ? ? ? ? | ? 21 ++++
>> ?doc/smjson.html ? ? ? ? ? ?| ? 64 +++++++++++
>> ?plugins/smjson/Makefile.am | ? ?6 +
>> ?plugins/smjson/smjson.c ? ?| ?262
>> ++++++++++++++++++++++++++++++++++++++++++++
>> ?5 files changed, 358 insertions(+), 0 deletions(-) ?create mode 100644
>> doc/smjson.html ?create mode 100644 plugins/smjson/Makefile.am ?create
>> mode 100644 plugins/smjson/smjson.c
>>
>> diff --git a/Makefile.am b/Makefile.am
>> index d689b9e..3a3db59 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -208,6 +208,10 @@ if ENABLE_ORACLE
>> ?SUBDIRS += plugins/omoracle
>> ?endif
>>
>> +if ENABLE_SMJSON
>> +SUBDIRS += plugins/smjson
>> +endif
>> +
>> ?if ENABLE_GUI
>> ?SUBDIRS += java
>> ?endif
>> @@ -253,5 +257,6 @@ DISTCHECK_CONFIGURE_FLAGS= ? ? ? ?--enable-
>> gssapi_krb5 \
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? --enable-imtemplate \
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? --enable-omtemplate \
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? --enable-mmsnmptrapd \
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? --enable-smjson \
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? --with-
>> systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir)
>> ?ACLOCAL_AMFLAGS = -I m4
>> diff --git a/configure.ac b/configure.ac index f6a09fa..d3307d0 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -1220,6 +1220,25 @@ AM_CONDITIONAL(ENABLE_OMMONGODB, test
>> x$enable_ommongodb = xyes) ?# end of copy template - be sure to search
>> for omtemplate to find everything!
>>
>>
>> +# SMJSON SUPPORT
>> +
>> +AC_ARG_ENABLE(smjson,
>> + ? ? ? ?[AS_HELP_STRING([--enable-smjson],[Compiles smjson strgen module
>> @<:@default=no@:>@])],
>> + ? ? ? ?[case "${enableval}" in
>> + ? ? ? ? yes) enable_smjson="yes" ;;
>> + ? ? ? ? ?no) enable_smjson="no" ;;
>> + ? ? ? ? ? *) AC_MSG_ERROR(bad value ${enableval} for --enable-smjson) ;;
>> + ? ? ? ? esac],
>> + ? ? ? ?[enable_smjson=no]
>> +)
>> +#
>> +# you may want to do some library checks here - see snmp, mysql, pgsql
>> +modules # for samples # AM_CONDITIONAL(ENABLE_SMJSON, test
>> +x$enable_smjson = xyes) # end of copy template - be sure to search for
>> +omtemplate to find everything!
>> +
>> +
>> ?AC_CONFIG_FILES([Makefile \
>> ? ? ? ? ? ? ? runtime/Makefile \
>> ? ? ? ? ? ? ? tools/Makefile \
>> @@ -1262,6 +1281,7 @@ AC_CONFIG_FILES([Makefile \
>> ? ? ? ? ? ? ? plugins/omoracle/Makefile \
>> ? ? ? ? ? ? ? plugins/omudpspoof/Makefile \
>> ? ? ? ? ? ? ? plugins/mmnormalize/Makefile \
>> + ? ? ? ? ? ? plugins/smjson/Makefile \
>> ? ? ? ? ? ? ? plugins/sm_cust_bindcdr/Makefile \
>> ? ? ? ? ? ? ? plugins/mmsnmptrapd/Makefile \
>> ? ? ? ? ? ? ? plugins/cust1/Makefile \
>> @@ -1317,6 +1337,7 @@ echo " ? ?mmsnmptrapd module will be compiled:
>> $enable_mmsnmptrapd"
>> ?echo
>> ?echo "---{ strgen modules }---"
>> ?echo " ? ?sm_cust_bindcdr module will be compiled:
>> $enable_sm_cust_bindcdr"
>> +echo " ? ?smjsonmodule will be compiled: ? ? ? ? ? ?$enable_smjson"
>> ?echo
>> ?echo "---{ database support }---"
>> ?echo " ? ?MySql support enabled: ? ? ? ? ? ? ? ? ? ?$enable_mysql"
>> diff --git a/doc/smjson.html b/doc/smjson.html new file mode 100644 index
>> 0000000..870163b
>> --- /dev/null
>> +++ b/doc/smjson.html
>> @@ -0,0 +1,64 @@
>> +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
>> +<html><head> <meta http-equiv="Content-Language" content="en">
>> +<title>JSON Strgen Module</title></head>
>> +
>> +<body>
>> +<a href="rsyslog_conf_modules.html">back</a>
>> +
>> +<h1>JSON Strgen Module</h1>
>> +<p><b>Module Name:&nbsp;&nbsp;&nbsp; smjson</b></p>
>> +<p><b>Author: Christian Brunner &lt;chb at muc.de&gt;</b></p>
>> +<p><b>Description</b>:</p> <p>Provides the ability to format syslog
>> +messages in JSON syntax.
>> +This module uses the <a target="_blank"
>> +href="http://libestr.adiscon.com/">
>> +libestr</a> and the <a target="_blank"
>> +href="http://www.libee.org/">libee</a> library. In order to compile
>> +this module, you will need to have the corresponding developer
>> +(headers) package installed. </p> <p>&nbsp;</p> <p><b>Configuration
>> +Directives</b>:</p> <ul>
>> + ? ? <li><strong>$smjsonadd key,value</strong><br>
>> + ? ? Add a key/value pair to the generated message.<br>
>> + ? ? transport types which are supported by NET-SNMP. <br>
>> + ? ? <strong>key</strong> is an arbitrary text string.<br>
>> + ? ? <strong>value</strong> can be a rsyslog template string (has to start
>> + ? ? ? ?and end with %) or a text string.<br>
>> + ? ? <br>
>> + ? ? Example: <strong>$smjsonadd date,%TIMESTAMP:::date-
>> rfc3339%<br>
>> + ? ? </strong></li>
>> +</ul>
>> +<p>&nbsp;</p>
>> +<p><b>Caveats/Known Bugs:</b></p><ul><li>Json output will be in
>> reverse
>> +order. </li></ul> <p><b>Sample:</b></p> <p>To generate a logfile with
>> +json messages like this:</p>
>> +
>> +<textarea rows="10" cols="60"> {"type": "SYSLOG", "cee":{"srcport":
>> +"54161", "srcip": "192.168.2.195", "user": "chb"}, "msg": "Accepted
>> +publickey for chb from 192.168.2.195 port 54161 ssh2", "hostname":
>> +"sir", "facility": "authpriv", "level": "info", "date":
>> +"2011-06-07T21:20:48+02:00"} </textarea>
>> +
>> +<p>You will need the following commands:</p> <textarea rows="10"
>> +cols="60">$ModLoad omsnmp
>> +
>> +$ModLoad smjson.so
>> +$template json,=SMJSON
>> +$smjsonadd date,%TIMESTAMP:::date-rfc3339% $smjsonadd
>> +level,%syslogpriority-text% $smjsonadd facility,%syslogfacility-text%
>> +$smjsonadd hostname,%hostname% $smjsonadd msg,%msg% $smjsonadd
>> +cee,%$!all-json% $smjsonadd type,SYSLOG
>> +*.* /var/tmp/json.log;json
>> +
>> +</textarea>
>> +
>> +<p>The example above is using mmnormalize to normalize the log message
>> +($!all-json).</p>
>> +
>> +<p>[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a
>> +href="manual.html">manual index</a>] [<a
>> +href="http://www.rsyslog.com/">rsyslog site</a>]</p> <p><font
>> +size="2">This documentation is part of the <a
>> +href="http://www.rsyslog.com/">rsyslog</a> project.<br> Copyright (c)
>> +2008 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a>
>> +and <a href="http://www.adiscon.com/">Adiscon</a>. Released under
>> the
>> +GNU GPL version 3 or higher.</font></p>
>> +
>> +</body></html>
>> diff --git a/plugins/smjson/Makefile.am b/plugins/smjson/Makefile.am new
>> file mode 100644 index 0000000..c0e9327
>> --- /dev/null
>> +++ b/plugins/smjson/Makefile.am
>> @@ -0,0 +1,6 @@
>> +pkglib_LTLIBRARIES = smjson.la
>> +
>> +smjson_la_SOURCES = smjson.c
>> +smjson_la_CPPFLAGS = -I$(top_srcdir) $(PTHREADS_CFLAGS)
>> $(RSRT_CFLAGS)
>> +$(LIBEE_CFLAGS) smjson_la_LDFLAGS = -module -avoid-version
>> +$(LIBEE_LIBS) smjson_la_LIBADD =
>> diff --git a/plugins/smjson/smjson.c b/plugins/smjson/smjson.c new file
>> mode 100644 index 0000000..9d0b7d3
>> --- /dev/null
>> +++ b/plugins/smjson/smjson.c
>> @@ -0,0 +1,262 @@
>> +/* smjson.c
>> + *
>> + * A strgen module to transform log messages into the json format.
>> + *
>> + * File begun on 2011-06-02 by Christian Brunner <chb at muc.de>
>> + *
>> + * Rsyslog is free software: you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation, either version 3 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * Rsyslog is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ?See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with Rsyslog. ?If not, see <http://www.gnu.org/licenses/>.
>> + *
>> + * A copy of the GPL can be found in the file "COPYING" in this
> distribution.
>> + */
>> +
>> +#include "config.h"
>> +#include "rsyslog.h"
>> +#include <stdlib.h>
>> +#include <ctype.h>
>> +#include <string.h>
>> +#include <assert.h>
>> +#include <errno.h>
>> +#include "conf.h"
>> +#include "syslogd-types.h"
>> +#include "cfsysline.h"
>> +#include "template.h"
>> +#include "msg.h"
>> +#include "module-template.h"
>> +#include "unicode-helper.h"
>> +#include "errmsg.h"
>> +
>> +#include <libee/libee.h>
>> +#include <libee/field.h>
>> +
>> +MODULE_TYPE_STRGEN
>> +MODULE_TYPE_NOKEEP
>> +STRGEN_NAME("SMJSON")
>> +
>> +/* internal structures
>> + */
>> +DEF_SMOD_STATIC_DATA
>> +DEFobjCurrIf(errmsg)
>> +
>> +typedef struct jsonentry_s {
>> + ? ? char *key;
>> + ? ? struct ee_field *constfield;
>> + ? ? ? ?struct templateEntry *pTpe;
>> + ? ? ? ?struct jsonentry_s *next;
>> +} jsonentry_t;
>> +
>> +static jsonentry_t *root;
>> +static ee_ctx ctx;
>> +
>> +static rsRetVal addJsonEntry(void __attribute__((unused)) *pVal, uchar
>> +*pNewVal) {
>> + ? ? jsonentry_t *pNew;
>> + ? ? DEFiRet;
>> +
>> + ? ? struct template *pTpl;
>> + ? ? ? ?struct templateEntry *pTpe;
>> +
>> + ? ? char *key, *value;
>> + ? ? es_str_t *estr;
>> +
>> + ? ? struct ee_value *eevalue;
>> +
>> + ? ? key = (char *) pNewVal;
>> + ? ? if (!(value = strchr((char *) pNewVal, ','))) {
>> + ? ? ? ? ? ? errmsg.LogError(0, NO_ERRCODE, "error: key/value separator
>> "
>> + ? ? ? ? ? ? ? ? ? ? "not found in '%s'", pNewVal);
>> + ? ? ? ? ? ? ? ?ABORT_FINALIZE(RS_RET_ERR);
>> + ? ? }
>> + ? ? *value = '\0';
>> + ? ? value++;
>> +
>> + ? ? CHKmalloc(pNew = malloc(sizeof(jsonentry_t)));
>> + ? ? pNew->key = key;
>> +
>> + ? ? if (value[0] == '%') {
>> + ? ? ? ? ? ? unsigned char *templateString;
>> + ? ? ? ? ? ? CHKmalloc(templateString = malloc(strlen(value) + 3));
>> + ? ? ? ? ? ? templateString[0] = '"';
>> + ? ? ? ? ? ? memcpy(templateString+1, value, strlen(value));
>> + ? ? ? ? ? ? memcpy(templateString+1+strlen(value), "\"", 2);
>> +
>> + ? ? ? ? ? ? pTpl = tplAddLine("smjson-intern", &templateString);
>> + ? ? ? ? ? ? pTpe = pTpl->pEntryRoot;
>> +
>> + ? ? ? ? ? ? pNew->constfield = NULL;
>> + ? ? ? ? ? ? pNew->pTpe = pTpe;
>> +
>> + ? ? ? ? ? ? free(templateString);
>> + ? ? } else {
>> + ? ? ? ? ? ? pNew->constfield = ee_newField(ctx);
>> + ? ? ? ? ? ? pNew->constfield->name = es_newStrFromBuf(key,
>> strlen(key));
>> + ? ? ? ? ? ? estr = es_newStrFromBuf(value, strlen(value));
>> + ? ? ? ? ? ? eevalue = ee_newValue(ctx);
>> + ? ? ? ? ? ? ee_setStrValue(eevalue, estr);
>> + ? ? ? ? ? ? ee_addValueToField(pNew->constfield, eevalue);
>> +
>> + ? ? ? ? ? ? pNew->pTpe = NULL;
>> + ? ? }
>> +
>> + ? ? pNew->next = root;
>> + ? ? root = pNew;
>> +
>> + ? ? DBGPRINTF("smjson: key/value '%s':'%s' added.\n", key, value);
>> +
>> +finalize_it:
>> + ? ? if(iRet != RS_RET_OK) {
>> + ? ? ? ? ? ? free(pNewVal);
>> + ? ? }
>> +
>> + ? ? RETiRet;
>> +}
>> +
>> +
>> +/* This strgen uses libee to generate the output string.
>> + */
>> +
>> +#define JSON_END "\n"
>> +BEGINstrgen
>> + ? ? es_str_t *json;
>> + ? ? es_str_t *estr;
>> + ? ? struct ee_field *field;
>> + ? ? struct ee_value *eevalue;
>> + ? ? propid_t propID;
>> + ? ? size_t propLen, jsonLen;
>> + ? ? uchar *pszProp = NULL;
>> + ? ? unsigned short bMustBeFreed = 0;
>> + ? ? char *jsonStr;
>> +
>> + ? ? char *key;
>> +
>> + ? ? jsonentry_t *iter;
>> + ? ? ? ?struct templateEntry *pTpe;
>> +
>> + ? ? if((json = es_newStr(256)) == NULL) goto finalize_it;
>> +
>> + ? ? ? ?es_addChar(&json, '{');
>> +
>> + ? ? for(iter = root; iter != NULL; iter = iter->next)
>> + ? ? {
>> + ? ? ? ? ? ? key = iter->key;
>> +
>> + ? ? ? ? ? ? if (iter->constfield) {
>> + ? ? ? ? ? ? ? ? ? ? ee_addField_JSON(iter->constfield, &json, 0);
>> +
>> + ? ? ? ? ? ? ? ? ? ? goto nextiter;
>> + ? ? ? ? ? ? }
>> +
>> + ? ? ? ? ? ? pTpe = iter->pTpe;
>> + ? ? ? ? ? ? propID = pTpe->data.field.propid;
>> +
>> + ? ? ? ? ? ? if(propID == PROP_CEE_ALL_JSON) {
>> + ? ? ? ? ? ? ? ? ? ? char *ceestr;
>> +
>> + ? ? ? ? ? ? ? ? ? ? es_addChar(&json, '"');
>> + ? ? ? ? ? ? ? ? ? ? es_addBuf(&json, key, strlen(key));
>> + ? ? ? ? ? ? ? ? ? ? es_addBuf(&json, "\":", 2);
>> + ? ? ? ? ? ? ? ? ? ? ee_fmtEventToJSON(pMsg->event, &estr);
>> + ? ? ? ? ? ? ? ? ? ? ceestr = es_str2cstr(estr, "#000");
>> + ? ? ? ? ? ? ? ? ? ? es_deleteStr(estr);
>> + ? ? ? ? ? ? ? ? ? ? es_addBuf(&json, ceestr, strlen(ceestr));
>> + ? ? ? ? ? ? ? ? ? ? free(ceestr);
>> +
>> + ? ? ? ? ? ? ? ? ? ? goto nextiter;
>> + ? ? ? ? ? ? }
>> +
>> + ? ? ? ? ? ? field = ee_newField(ctx);
>> + ? ? ? ? ? ? field->name = es_newStrFromBuf(key, strlen(key));
>> +
>> + ? ? ? ? ? ? pszProp = MsgGetProp(pMsg, pTpe, propID,
>> + ? ? ? ? ? ? ? ? ? ? pTpe->data.field.propName, &propLen,
>> &bMustBeFreed);
>> +
>> + ? ? ? ? ? ? if(propID == PROP_MSG && pszProp[0] == ' ') {
>> + ? ? ? ? ? ? ? ? ? ? estr = es_newStrFromBuf((char *) pszProp+1,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? strlen((char *) pszProp)-1);
>> + ? ? ? ? ? ? } else {
>> + ? ? ? ? ? ? ? ? ? ? estr = es_newStrFromBuf((char *) pszProp,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? strlen((char *) pszProp));
>> + ? ? ? ? ? ? }
>> +
>> + ? ? ? ? ? ? eevalue = ee_newValue(ctx);
>> + ? ? ? ? ? ? ee_setStrValue(eevalue, estr);
>> + ? ? ? ? ? ? ee_addValueToField(field, eevalue);
>> +
>> + ? ? ? ? ? ? ee_addField_JSON(field, &json, 0);
>> + ? ? ? ? ? ? ee_deleteField(field);
>> +
>> + ? ? ? ? ? ? if(bMustBeFreed) {
>> + ? ? ? ? ? ? ? ? ? ? free(pszProp);
>> + ? ? ? ? ? ? ? ? ? ? bMustBeFreed = 0;
>> + ? ? ? ? ? ? }
>> +
>> +nextiter:
>> + ? ? ? ? ? ? if (iter->next)
>> + ? ? ? ? ? ? ? ? ? ? es_addBuf(&json, ", ", ee_ctxIsEncUltraCompact(ctx)
>> ? 1 : 2);
>> + ? ? }
>> +
>> + ? ? ? ?es_addChar(&json, '}');
>> +
>> + ? ? jsonStr = (char*) es_str2cstr(json, "#000");
>> + ? ? jsonLen = strlen(jsonStr);
>> +
>> + ? ? es_deleteStr(json);
>> +
>> + ? ? ? ?if((jsonLen + sizeof(JSON_END)) >= *pLenBuf)
>> + ? ? ? ? ? ? ? ?CHKiRet(ExtendBuf(ppBuf, pLenBuf, jsonLen +
>> + sizeof(JSON_END)));
>> +
>> + ? ? memcpy(*ppBuf, jsonStr, jsonLen);
>> + ? ? memcpy(*ppBuf + jsonLen, JSON_END, sizeof(JSON_END));
>> +
>> + ? ? free(jsonStr);
>> + ? ? ee_exitCtx(ctx);
>> +
>> +finalize_it:
>> + ? ? ? ?if(bMustBeFreed)
>> + ? ? ? ? ? ? ? ?free(pszProp);
>> +ENDstrgen
>> +
>> +BEGINmodExit
>> + ? ? jsonentry_t *iter, *jdel;
>> +CODESTARTmodExit
>> + ? ? for(iter = root; iter != NULL; ) {
>> + ? ? ? ? ? ? jdel = iter;
>> + ? ? ? ? ? ? iter = iter->next;
>> + ? ? ? ? ? ? if (jdel->constfield) {
>> + ? ? ? ? ? ? ? ? ? ? ee_deleteField(jdel->constfield);
>> + ? ? ? ? ? ? }
>> + ? ? ? ? ? ? free(jdel->key);
>> + ? ? }
>> + ? ? objRelease(errmsg, CORE_COMPONENT);
>> +ENDmodExit
>> +
>> +
>> +BEGINqueryEtryPt
>> +CODESTARTqueryEtryPt
>> +CODEqueryEtryPt_STD_SMOD_QUERIES
>> +ENDqueryEtryPt
>> +
>> +BEGINmodInit()
>> +CODESTARTmodInit
>> + ? ? ? ?*ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the
>> +current interface specification */ CODEmodInit_QueryRegCFSLineHdlr
>> + ? ? ? ?CHKiRet(objUse(errmsg, CORE_COMPONENT));
>> +
>> + ? ? root = NULL;
>> + ? ? ctx = ee_initCtx();
>> +
>> + ? ? ? ?CHKiRet(omsdRegCFSLineHdlr((uchar *)"smjsonadd", 0,
>> eCmdHdlrGetWord,
>> + ? ? ? ? ? ? ? ?addJsonEntry, NULL, STD_LOADABLE_MODULE_ID,
>> + eConfObjGlobal));
>> +
>> + ? ? ? ?dbgprintf("rsyslog json strgen init called, compiled with
>> +version %s\n", VERSION); ENDmodInit
>> --
>> 1.7.1
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From rory at ooma.com  Wed Jun  8 22:16:56 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 08 Jun 2011 13:16:56 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DE7FE7C.3020604@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com>
	<4DE7FE7C.3020604@ooma.com>
Message-ID: <4DEFD8B8.6040506@ooma.com>

On 6/2/11 2:19 PM, Rory Toma wrote:
> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
>> Mmhhh.. can you post a complete debug log (maybe via a website like 
>> filebin)?
>>
>> Thx,
>> Rainer
>>
> Let me know when you've downloaded this file and I'll reset the ACL, thx.
>
> http://www.colinburns.com/downloads/darwin0.bz2
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
Any update on this?

thx

From david at lang.hm  Thu Jun  9 00:58:35 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 8 Jun 2011 15:58:35 -0700 (PDT)
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1106081557480.28905@asgard.lang.hm>

an option that we should have at some point is to escape newline 
characters inside a message (converting a multiline message to a single 
line)

David Lang


  On Wed, 8 Jun 2011, Rainer Gerhards wrote:

> Multi-line messages are not supported by legacy plain syslogd. But you can
> turn on the (o) option, which enables octect-counted framing, with which it
> works. However, non-rsyslog receivers probably do not understand that
> framing.
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> Sent: Wednesday, June 08, 2011 3:16 PM
>> To: rsyslog-users
>> Subject: [rsyslog] Problem with forwarded multiline message
>>
>> Hello,
>>
>> I set up two hosts to test rsyslogd, dns1 as client, z6 as server, and
> found that
>> the server interpreted a copy of forwarded multiline message(3rd entry in
>> the following raw messages) into multiple entries(3rd and 4th entry in
> actual
>> output), while locally generated multiline message was fine. What's the
>> problem?
>>
>> The client setting:
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>> *.*                @@10.3.254.106:514
>>
>> The server setting:
>> $InputPTCPServerRun 514
>> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format $template
>> rawfmt,"%rawmsg%\n"
>> *.*    /var/log/rawmessages;rawfmt
>> *.info;mail.none;authpriv.none;cron.none                /var/log/messages
>>
>> Other settings were action queue tuning, I guess they were irrelevant.
>>
>> The raw messages:
>> <6>Jun  8 20:52:48 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
> started.
>> <46>Jun  8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"] start
>> <43>Jun  8 20:52:48 dns1 rsyslogd-2066: could not load module
>> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
>> cannot open shared object file: No such file or directory  [try
>> http://www.rsyslog.com/e/2066 ] <43>Jun  8 20:52:48 dns1 rsyslogd: the last
>> error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> <43>Jun  8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not interpret
>> master config file '/etc/rsyslog.conf'. [try
>> http://www.rsyslog.com/e/2124 ]
>> imklog 5.8.1, log source = /proc/kmsg started.
>>  [origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> x-info="http://www.rsyslog.com"] start
>> could not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory  [try http://www.rsyslog.com/e/2066 ] the last error occured in
>> /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try
>> http://www.rsyslog.com/e/2124 ]
>>
>> Actual ouput:
>> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - -  imklog 5.8.1, log source
> =
>> /proc/kmsg started.
>> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  [origin
>> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>> x-info="http://www.rsyslog.com"] start
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - -  could not load
>> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory
>> <13>1 2011-06-08T20:52:48.251337+08:00 bogon  - - -  [try
>> http://www.rsyslog.com/e/2066 ]
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - -  the last error occured
> in
>> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - -  CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog 5.8.1, log
> source
>> = /proc/kmsg started.
>> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - -  [origin
>> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>> x-info="http://www.rsyslog.com"] start
>> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - - could not
> load
>> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
> or
>> directory  [try http://www.rsyslog.com/e/2066 ]
>> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last error
>> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - - CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>>
>>
>> Thanks,
>> Kaiwang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Thu Jun  9 10:17:09 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Thu, 9 Jun 2011 16:17:09 +0800
Subject: [rsyslog] Problem with forwarded multiline message
In-Reply-To: <BANLkTimECXNT=DYnOK6RZS5To4uTE1D-DQ@mail.gmail.com>
References: <BANLkTi=Zfo8XjQysk8D1jfUZ2b+htiaJKg@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E03@GRFEXC.intern.adiscon.com>
	<BANLkTikovSuimR6qhfz9qpuDiEFAgiU+nA@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E05@GRFEXC.intern.adiscon.com>
	<BANLkTinexXFtBMvXsHOVNC6p232hRt6_rQ@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E13@GRFEXC.intern.adiscon.com>
	<BANLkTi=9C5KKsObJT9NeBdhCsfC0ovdE2w@mail.gmail.com>
	<BANLkTimECXNT=DYnOK6RZS5To4uTE1D-DQ@mail.gmail.com>
Message-ID: <BANLkTim6sQrTS1Ls3eQ5QDe6XdEzf650eQ@mail.gmail.com>

Strangely the problem no longer occur.  I just remember sometimes the
multiline messages were not recorded even with (o) option.

Thanks,
Kaiwang

2011/6/9 Kaiwang Chen <kaiwang.chen at gmail.com>:
> Sorry, it's false alarm. I noticed that the problem occured regardless
> of (o) option. After a few check, I found myself was testing new
> directives:
>
> $ModLoad ommail
>
> $ActionMailSMTPServer ....
> $ActionMailFrom ....
> $ActionMailTo ....
> $template mailSubject,"test problem on %HOSTNAME%"
> $template mailBody,"RSYSLOG Alert\r\nmsg='%rawmsg%'"
> $ActionMailSubject mailSubject
> $ActionExecOnlyOnceEveryInterval 1
> if $msg contains 'started' then :ommail:;mailBody
>
> With directives related to ommail commented out, the (o) option worked
> well and recorded:
> <43>Jun ?9 02:06:48 dns1 rsyslogd-2066: could not load module
> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
> cannot open shared object file: No such file or directory#012 [try
> http://www.rsyslog.com/e/2066 ]
>
> However, I am not sure why ommail suppressed the recording of the
> second packet. The full configuration of receiver is
>
> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>
> $ModLoad imklog
> $ModLoad imuxsock
> $ModLoad imudp
> $ModLoad imptcp
> $ModLoad ommysql
> $ModLoad impstats
>
> #$ModLoad ommail
>
> #$ActionMailSMTPServer ....
> #$ActionMailFrom ....
> #$ActionMailTo ....
> #$template mailSubject,"test problem on %HOSTNAME%"
> #$template mailBody,"RSYSLOG Alert\r\nmsg='%rawmsg%'"
> #$ActionMailSubject mailSubject
> #$ActionExecOnlyOnceEveryInterval 1
> #if $msg contains 'started' then :ommail:;mailBody
>
> $PStatInterval 600
> $PStatSeverity 7
>
> $InputPTCPServerNotifyOnConnectionClose on
>
> $UDPServerRun 514
> $InputPTCPServerRun 514
>
> $MainMsgQueueSaveOnShutdown on
> $MainMsgQueueFileName mq
> $MainMsgQueueMaxFileSize 5m
>
> $template DynFile,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/rsyslog.log"
> *.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??DynFile
>
> $WorkDirectory /var/spool/rsyslog
>
> $ActionQueueType LinkedList
> $ActionQueueSize 200000
> $ActionQueueDiscardMark 190000
> $ActionQueueHighWaterMark 160000
> $ActionQueueLowWaterMark 40000
> $ActionQueueFileName dbq
> $ActionQueueSaveOnShutdown on
> $ActionQueueMaxFileSize 128m
> $ActionResumeRetryCount -1
> $actionommysqlserverport 6666
> *.* ? ? ? :ommysql:10.3.254.109,syslog,syslogwriter,tops3cr3t
>
> syslog.debug ?/var/log/rsyslog-stats
>
>
> $template rawfmt,"%rawmsg%\n"
> *.* ? ?/var/log/rawmessages;rawfmt
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none ? ? ? ? ? ? ? ?/var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure
>
> # Log all the mail messages in one place.
> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/maillog
>
>
> # Log cron stuff
> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/cron
>
> # Everybody gets emergency messages
> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? *
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/boot.log
>
>
> Thanks,
> Kaiwang
>
>
> 2011/6/9 Kaiwang Chen <kaiwang.chen at gmail.com>:
>> Not sure what "debug log" is. I try to represent the problem with
>> packets and raw messages.
>>
>> I repeated the steps with: 1) stopping sender, 2) starting tcpdump on
>> sender, 3) starting tcpdump on receiver, ?4) starting sender with bad
>> configuration so that it compained with multiline message, 5) stopping
>> tcpdump on sender, 6) stopping tcpdump on receiver.
>>
>> Two data packets reached receiver during step 2) and 5), with
>> printable text payload
>> (packet 1)
>> 78 <6>Jun ?9 01:23:53 dns1 kernel: imklog 5.8.1, log source =
>> /proc/kmsg started.
>> (packet 2)
>> 132 <46>Jun ?9 01:23:53 dns1 rsyslogd: [origin software="rsyslogd"
>> swVersion="5.8.1" x-pid="6632" x-info="http://www.rsyslog.com"]
>> start68 <6>Jun ?9 01:23:53 dns1 kernel: device eth1 entered
>> promiscuous mode227 <43>Jun ?9 01:23:53 dns1 rsyslogd-2066: could not
>> load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
>> file or directory
>> ?[try http://www.rsyslog.com/e/2066 ]104 <43>Jun ?9 01:23:53 dns1
>> rsyslogd: the last error occured in /etc/rsyslog.conf, line
>> 3:"$ModLoad omhdfs"150 <43>Jun ?9 01:23:53 dns1 rsyslogd-2124: CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>>
>> In the meanwhile, the sender recorded formated messages:
>> <6>1 2011-06-09T01:23:53.994948+08:00 dns1 kernel - - - imklog 5.8.1,
>> log source = /proc/kmsg started.
>> <46>1 2011-06-09T01:23:53.995068+08:00 dns1 rsyslogd - - - ?[origin
>> software="rsyslogd" swVersion="5.8.1" x-pid="6632"
>> x-info="http://www.rsyslog.com"] start
>> <6>1 2011-06-09T01:23:53.995255+08:00 dns1 kernel - - - device eth1
>> entered promiscuous mode
>> <43>1 2011-06-09T01:23:53.993615+08:00 dns1 rsyslogd-2066 - - - could
>> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such
>> file or directory
>> ?[try http://www.rsyslog.com/e/2066 ]
>> <43>1 2011-06-09T01:23:53.993692+08:00 dns1 rsyslogd - - - the last
>> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>> <43>1 2011-06-09T01:23:53.994860+08:00 dns1 rsyslogd-2124 - - - CONFIG
>> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>> [try http://www.rsyslog.com/e/2124 ]
>> <6>1 2011-06-09T01:24:41.727433+08:00 dns1 kernel - - - device eth1
>> left promiscuous mode
>>
>> The raw messages on receiver:
>> device eth1 entered promiscuous mode
>> <6>Jun ?9 01:23:53 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg started.
>> <6>Jun ?9 01:24:41 dns1 kernel: device eth1 left promiscuous mode
>> device eth1 left promiscuous mode
>>
>> The missing packet should be after 2nd raw message and before 3rd one.
>> Obviously the first packet(2nd raw message) reached the receiver and
>> was recorded. The second packet reached receiver and was not recorded.
>>
>>
>> Thanks,
>> Kaiwang
>>
>> 2011/6/9 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>> Lets keep focssed. Please provide me a debug log of the receiver.
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>> Sent: Wednesday, June 08, 2011 7:10 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>>>
>>>> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>> >
>>>> >> -----Original Message-----
>>>> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>> >> Sent: Wednesday, June 08, 2011 4:04 PM
>>>> >> To: rsyslog-users
>>>> >> Subject: Re: [rsyslog] Problem with forwarded multiline message
>>>> >>
>>>> >> So the (o) option just affects the receiver, and would be of no harm
>>>> >> being turned on at the terminal end of syslog message flow.
>>>> >> Actually, I am going
>>>> > to
>>>> >> use rsyslog on both ends, except bridge and router sources.
>>>> >
>>>> > Actually the sender. The receiver automatically handles both. The
>>>> > option is at the action, I think it is along the lines of
>>>> >
>>>> > @@(o)host
>>>>
>>>>
>>>> Yes, it's documented in "Remote Machine" section from
>>>> http://www.rsyslog.com/doc/rsyslog_conf_actions.html
>>>> However, when rsyslogd as sender was configured
>>>> *.* ? ? ? ? ? ? ? ?@@(o)10.3.254.106:514
>>>>
>>>> the receiver (same version 5.8.1) recorded nothing. I confirmed with
>>>> "tcpdump -i eth1 tcp port 514 -s0" that two data packets reached the
>>> receiver
>>>> with payload:
>>>>
>>>> 0000 ? 37 38 20 3c 36 3e 4a 75 6e 20 20 39 20 30 30 3a ?78 <6>Jun ?9 00:
>>>> 0010 ? 34 34 3a 31 39 20 64 6e 73 31 20 6b 65 72 6e 65 ?44:19 dns1 kerne
>>>> 0020 ? 6c 3a 20 69 6d 6b 6c 6f 67 20 35 2e 38 2e 31 2c ?l: imklog 5.8.1,
>>>> 0030 ? 20 6c 6f 67 20 73 6f 75 72 63 65 20 3d 20 2f 70 ? log source = /p
>>>> 0040 ? 72 6f 63 2f 6b 6d 73 67 20 73 74 61 72 74 65 64 ?roc/kmsg started
>>>> 0050 ? 2e ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? .
>>>>
>>>> 0000 ? 31 33 32 20 3c 34 36 3e 4a 75 6e 20 20 39 20 30 ?132 <46>Jun ?9 0
>>>> 0010 ? 30 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 ?0:44:19 dns1 rsy
>>>> 0020 ? 73 6c 6f 67 64 3a 20 5b 6f 72 69 67 69 6e 20 73 ?slogd: [origin s
>>>> 0030 ? 6f 66 74 77 61 72 65 3d 22 72 73 79 73 6c 6f 67 ?oftware="rsyslog
>>>> 0040 ? 64 22 20 73 77 56 65 72 73 69 6f 6e 3d 22 35 2e ?d" swVersion="5.
>>>> 0050 ? 38 2e 31 22 20 78 2d 70 69 64 3d 22 36 32 32 30 ?8.1" x-pid="6220
>>>> 0060 ? 22 20 78 2d 69 6e 66 6f 3d 22 68 74 74 70 3a 2f ?" x-info="http:/
>>>> 0070 ? 2f 77 77 77 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d ?/www.rsyslog.com
>>>> 0080 ? 22 5d 20 73 74 61 72 74 32 32 37 20 3c 34 33 3e ?"] start227 <43>
>>>> 0090 ? 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a 31 39 20 ?Jun ?9 00:44:19
>>>> 00a0 ? 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 2d 32 30 ?dns1 rsyslogd-20
>>>> 00b0 ? 36 36 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 6c 6f ?66: could not lo
>>>> 00c0 ? 61 64 20 6d 6f 64 75 6c 65 20 27 2f 75 73 72 2f ?ad module '/usr/
>>>> 00d0 ? 6c 69 62 36 34 2f 72 73 79 73 6c 6f 67 2f 6f 6d ?lib64/rsyslog/om
>>>> 00e0 ? 68 64 66 73 2e 73 6f 27 2c 20 64 6c 6f 70 65 6e ?hdfs.so', dlopen
>>>> 00f0 ? 3a 20 2f 75 73 72 2f 6c 69 62 36 34 2f 72 73 79 ?: /usr/lib64/rsy
>>>> 0100 ? 73 6c 6f 67 2f 6f 6d 68 64 66 73 2e 73 6f 3a 20 ?slog/omhdfs.so:
>>>> 0110 ? 63 61 6e 6e 6f 74 20 6f 70 65 6e 20 73 68 61 72 ?cannot open shar
>>>> 0120 ? 65 64 20 6f 62 6a 65 63 74 20 66 69 6c 65 3a 20 ?ed object file:
>>>> 0130 ? 4e 6f 20 73 75 63 68 20 66 69 6c 65 20 6f 72 20 ?No such file or
>>>> 0140 ? 64 69 72 65 63 74 6f 72 79 0a 20 5b 74 72 79 20 ?directory. [try
>>>> 0150 ? 68 74 74 70 3a 2f 2f 77 77 77 2e 72 73 79 73 6c ?http://www.rsysl
>>>> 0160 ? 6f 67 2e 63 6f 6d 2f 65 2f 32 30 36 36 20 5d 31 ?og.com/e/2066 ]1
>>>> 0170 ? 30 34 20 3c 34 33 3e 4a 75 6e 20 20 39 20 30 30 ?04 <43>Jun ?9 00
>>>> 0180 ? 3a 34 34 3a 31 39 20 64 6e 73 31 20 72 73 79 73 ?:44:19 dns1 rsys
>>>> 0190 ? 6c 6f 67 64 3a 20 74 68 65 20 6c 61 73 74 20 65 ?logd: the last e
>>>> 01a0 ? 72 72 6f 72 20 6f 63 63 75 72 65 64 20 69 6e 20 ?rror occured in
>>>> 01b0 ? 2f 65 74 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e ?/etc/rsyslog.con
>>>> 01c0 ? 66 2c 20 6c 69 6e 65 20 33 3a 22 24 4d 6f 64 4c ?f, line 3:"$ModL
>>>> 01d0 ? 6f 61 64 20 6f 6d 68 64 66 73 22 31 35 30 20 3c ?oad omhdfs"150 <
>>>> 01e0 ? 34 33 3e 4a 75 6e 20 20 39 20 30 30 3a 34 34 3a ?43>Jun ?9 00:44:
>>>> 01f0 ? 31 39 20 64 6e 73 31 20 72 73 79 73 6c 6f 67 64 ?19 dns1 rsyslogd
>>>> 0200 ? 2d 32 31 32 34 3a 20 43 4f 4e 46 49 47 20 45 52 ?-2124: CONFIG ER
>>>> 0210 ? 52 4f 52 3a 20 63 6f 75 6c 64 20 6e 6f 74 20 69 ?ROR: could not i
>>>> 0220 ? 6e 74 65 72 70 72 65 74 20 6d 61 73 74 65 72 20 ?nterpret master
>>>> 0230 ? 63 6f 6e 66 69 67 20 66 69 6c 65 20 27 2f 65 74 ?config file '/et
>>>> 0240 ? 63 2f 72 73 79 73 6c 6f 67 2e 63 6f 6e 66 27 2e ?c/rsyslog.conf'.
>>>> 0250 ? 20 5b 74 72 79 20 68 74 74 70 3a 2f 2f 77 77 77 ? [try http://www
>>>> 0260 ? 2e 72 73 79 73 6c 6f 67 2e 63 6f 6d 2f 65 2f 32 ?.rsyslog.com/e/2
>>>> 0270 ? 31 32 34 20 5d ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 124 ]
>>>>
>>>>
>>>> >
>>>> >>
>>>> >> I came across such a framing option days ago, and just can't locate
>>>> >> it. How
>>>> > to
>>>> >> turn the (o) option, is it a compilation flag or a configuration
>>> directive?
>>>> >>
>>>> >> Is "lagacy plain syslogd" referring to rsyslogd with imptcp? I
>>>> >> understand
>>>> > stock
>>>> >> sysklogd can't deal with multiline messages.
>>>> >
>>>> > Oh, sorry, not just a typo. It should read "legacy plain tcp syslog
>>>> > (protocol)". This is what most applications understand under "TCP
>>>> > syslog". It uses \n to end a message and start a new one. Usually this
>>>> > is not a problem, as control character escaping removes \n in any case.
>>>>
>>>> Will omrelp supress this problem, or is there any other way to get rid of
>>> it, if
>>>> plain tcp with (o) option does not work well?
>>>>
>>>>
>>>> Thanks,
>>>> Kaiwang
>>>>
>>>> >
>>>> > Rainer
>>>> >>
>>>> >> Thanks,
>>>> >> Kaiwang
>>>> >>
>>>> >> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
>>>> >> > Multi-line messages are not supported by legacy plain syslogd. But
>>>> >> > you can turn on the (o) option, which enables octect-counted
>>>> >> > framing, with which it works. However, non-rsyslog receivers
>>>> >> > probably do not understand that framing.
>>>> >> > Rainer
>>>> >> >
>>>> >> >> -----Original Message-----
>>>> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> >> >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>>>> >> >> Sent: Wednesday, June 08, 2011 3:16 PM
>>>> >> >> To: rsyslog-users
>>>> >> >> Subject: [rsyslog] Problem with forwarded multiline message
>>>> >> >>
>>>> >> >> Hello,
>>>> >> >>
>>>> >> >> I set up two hosts to test rsyslogd, dns1 as client, z6 as server,
>>>> >> >> and
>>>> >> > found that
>>>> >> >> the server interpreted a copy of forwarded multiline message(3rd
>>>> >> >> entry in the following raw messages) into multiple entries(3rd and
>>>> >> >> 4th entry in
>>>> >> > actual
>>>> >> >> output), while locally generated multiline message was fine.
>>>> >> >> What's the problem?
>>>> >> >>
>>>> >> >> The client setting:
>>>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>>> >> >> *.* ? ? ? ? ? ? ? ?@@10.3.254.106:514
>>>> >> >>
>>>> >> >> The server setting:
>>>> >> >> $InputPTCPServerRun 514
>>>> >> >> $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
>>>> >> >> $template rawfmt,"%rawmsg%\n"
>>>> >> >> *.* ? ?/var/log/rawmessages;rawfmt
>>>> >> >> *.info;mail.none;authpriv.none;cron.none
>>>> >> >> /var/log/messages
>>>> >> >>
>>>> >> >> Other settings were action queue tuning, I guess they were
>>> irrelevant.
>>>> >> >>
>>>> >> >> The raw messages:
>>>> >> >> <6>Jun ?8 20:52:48 dns1 kernel: imklog 5.8.1, log source =
>>>> >> >> /proc/kmsg
>>>> >> > started.
>>>> >> >> <46>Jun ?8 20:52:48 dns1 rsyslogd: [origin software="rsyslogd"
>>>> >> >> swVersion="5.8.1" x-pid="4152" x-info="http://www.rsyslog.com"]
>>>> >> >> start <43>Jun ?8 20:52:48 dns1 rsyslogd-2066: could not load
>>>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>>> /usr/lib64/rsyslog/omhdfs.so:
>>>> >> >> cannot open shared object file: No such file or directory ?[try
>>>> >> >> http://www.rsyslog.com/e/2066 ] <43>Jun ?8 20:52:48 dns1 rsyslogd:
>>>> >> >> the last error occured in /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>>>> >> >> <43>Jun ?8 20:52:48 dns1 rsyslogd-2124: CONFIG ERROR: could not
>>>> >> >> interpret master config file '/etc/rsyslog.conf'. [try
>>>> >> >> http://www.rsyslog.com/e/2124 ]
>>>> >> >> imklog 5.8.1, log source = /proc/kmsg started.
>>>> >> >> ?[origin software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>>>> >> >> x-info="http://www.rsyslog.com"] start could not load module
>>>> >> >> '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>>> >> >> such file
>>>> >> > or
>>>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ] the last error
>>>> >> >> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>>>> >> >> CONFIG ERROR: could not interpret master config file
>>>> > '/etc/rsyslog.conf'.
>>>> >> > [try
>>>> >> >> http://www.rsyslog.com/e/2124 ]
>>>> >> >>
>>>> >> >> Actual ouput:
>>>> >> >> <6>1 2011-06-08T20:52:48+08:00 dns1 kernel - - - ?imklog 5.8.1,
>>>> >> >> log source
>>>> >> > =
>>>> >> >> /proc/kmsg started.
>>>> >> >> <46>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?[origin
>>>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="4152"
>>>> >> >> x-info="http://www.rsyslog.com"] start
>>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2066 - - - ?could
>>>> >> >> not load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>>> >> >> such file
>>>> >> > or
>>>> >> >> directory
>>>> >> >> <13>1 2011-06-08T20:52:48.251337+08:00 bogon ?- - - ?[try
>>>> >> >> http://www.rsyslog.com/e/2066 ]
>>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd - - - ?the last
>>>> >> >> error occured
>>>> >> > in
>>>> >> >> /etc/rsyslog.conf, line 6:"$ModLoad omhdfs"
>>>> >> >> <43>1 2011-06-08T20:52:48+08:00 dns1 rsyslogd-2124 - - - ?CONFIG
>>>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>>>> >> >> [try http://www.rsyslog.com/e/2124 ]
>>>> >> >> <6>1 2011-06-08T20:59:33.791626+08:00 z6 kernel - - - imklog
>>>> >> >> 5.8.1, log
>>>> >> > source
>>>> >> >> = /proc/kmsg started.
>>>> >> >> <46>1 2011-06-08T20:59:33.791820+08:00 z6 rsyslogd - - - ?[origin
>>>> >> >> software="rsyslogd" swVersion="5.8.1" x-pid="11033"
>>>> >> >> x-info="http://www.rsyslog.com"] start
>>>> >> >> <43>1 2011-06-08T20:59:33.787162+08:00 z6 rsyslogd-2066 - - -
>>>> >> >> could not
>>>> >> > load
>>>> >> >> module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
>>>> >> >> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No
>>>> >> >> such file
>>>> >> > or
>>>> >> >> directory ?[try http://www.rsyslog.com/e/2066 ]
>>>> >> >> <43>1 2011-06-08T20:59:33.787221+08:00 z6 rsyslogd - - - the last
>>>> >> >> error occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
>>>> >> >> <43>1 2011-06-08T20:59:33.791571+08:00 z6 rsyslogd-2124 - - -
>>>> >> >> CONFIG
>>>> >> >> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
>>>> >> >> [try http://www.rsyslog.com/e/2124 ]
>>>> >> >>
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Kaiwang
>>>> >> >> _______________________________________________
>>>> >> >> rsyslog mailing list
>>>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> >> >> http://www.rsyslog.com
>>>> >> > _______________________________________________
>>>> >> > rsyslog mailing list
>>>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> >> > http://www.rsyslog.com
>>>> >> >
>>>> >> _______________________________________________
>>>> >> rsyslog mailing list
>>>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> >> http://www.rsyslog.com
>>>> > _______________________________________________
>>>> > rsyslog mailing list
>>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> > http://www.rsyslog.com
>>>> >
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>>>
>>
>

From rgerhards at hq.adiscon.com  Thu Jun  9 11:26:52 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 9 Jun 2011 11:26:52 +0200
Subject: [rsyslog] Problem with multiline messages or ommail
In-Reply-To: <BANLkTingiYgbHKj95ELymPP8ggnt-7c99A@mail.gmail.com>
References: <BANLkTingiYgbHKj95ELymPP8ggnt-7c99A@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E20@GRFEXC.intern.adiscon.com>

Sorry, accidently stripped the list when replying ;)

I just checked the log, the

$ActionExecOnlyOnceEveryInterval 1

Does not auto-reset, so you need to do that yourself. This is an artifact of
the old config system and will be solved with v6 as soon as I find enough
time to work on it ;)

RAiner

> -----Original Message-----
> From: Kaiwang Chen [mailto:kaiwang.chen at gmail.com]
> Sent: Thursday, June 09, 2011 11:19 AM
> To: Rainer Gerhards
> Subject: Problem with multiline messages or ommail
> 
> Hello Rainer,
> 
> I can't reliably reproduce the problem. Fortunately captured a debug log
> (enclosed rsyslog_mail.txt) with
> RSYSLOG_DEBUGLOG=/root/rsyslog_mail.txt RSYSLOG_DEBUG="debug
> NoStdOut"  rsyslogd -c5
> 
> The problem was that the multiline message was alerted via mail but not
> recorded, neither to raw message, nor to mysql backend.
> 
> The sender was configured so that when start it would complain dlopen error
> with a multiline message. It forwarded messages to the receiver with (o)
> option enabled. It was running on CentOS release 5.5 (Final) with static
> kernel 2.6.35.5.
> The receiver was configured(enclosed rsyslog.conf) to log raw messages and
> to save to mysql database, and to send alert mail on certain messages. It
was
> running on CentOS release 5.6 (Final) with static kernel 2.6.35.12.
> Both was running rsyslog 5.8.1 (spec enclosed), which was built on sender
> host. Patch0 was to add ProcessID field to SystemEvents in CreateDB.sql.
> 
> (sender log)
> <6>1 2011-06-09T16:48:34.167156+08:00 dns1 kernel - - - Kernel logging
> (proc) stopped.
> <46>1 2011-06-09T16:48:34.167320+08:00 dns1 rsyslogd - - -  [origin
> software="rsyslogd" swVersion="5.8.1" x-pid="16596"
> x-info="http://www.rsyslog.com"] exiting on signal 15.
> <6>1 2011-06-09T16:48:43.897599+08:00 dns1 kernel - - - imklog 5.8.1, log
> source = /proc/kmsg started.
> <46>1 2011-06-09T16:48:43.897716+08:00 dns1 rsyslogd - - -  [origin
> software="rsyslogd" swVersion="5.8.1" x-pid="16618"
> x-info="http://www.rsyslog.com"] start
> <43>1 2011-06-09T16:48:43.896184+08:00 dns1 rsyslogd-2066 - - - could not
> load module '/usr/lib64/rsyslog/omhdfs.so', dlopen:
> /usr/lib64/rsyslog/omhdfs.so: cannot open shared object file: No such file
or
> directory  [try http://www.rsyslog.com/e/2066 ]
> <43>1 2011-06-09T16:48:43.896261+08:00 dns1 rsyslogd - - - the last error
> occured in /etc/rsyslog.conf, line 3:"$ModLoad omhdfs"
> <43>1 2011-06-09T16:48:43.897512+08:00 dns1 rsyslogd-2124 - - - CONFIG
> ERROR: could not interpret master config file '/etc/rsyslog.conf'.
> [try http://www.rsyslog.com/e/2124 ]
> 
> (receiver raw message)
> <6>Jun  9 16:48:34 dns1 kernel: Kernel logging (proc) stopped.
> <6>Jun  9 16:48:43 dns1 kernel: imklog 5.8.1, log source = /proc/kmsg
started.
> Kernel logging (proc) stopped.
> 
> (mail reported)
> From 10.3.254.106 Thu Jun  9 16:48:43 2011 MAIL FROM: <root at kc.org> RCPT
> TO: <sa at kc.org> DATA
> Date: Thu,  9 Jun 2011  8:48:42 UT
> From: <root at kc.org>
> To: <sa at kc.org>
> Subject: test problem on dns1
> X-Mailer: rsyslog-immail
> 
> RSYSLOG Alert
> msg='<43>Jun  9 16:48:43 dns1 rsyslogd-2066: could not load module
> '/usr/lib64/rsyslog/omhdfs.so', dlopen: /usr/lib64/rsyslog/omhdfs.so:
> cannot open shared object file: No such file or directory#012 [try
> http://www.rsyslog.com/e/2066 ]'
> 
> ====================
> 
> The above mail was reported by the this program listenning on
> 10.3.254.110:25 use strict; use Carp; use Getopt::Long; use
> Net::SMTP::Server; use Net::SMTP::Server::Client;
> 
> my ($listen) = ('localhost');
> GetOptions("listen=s" => \$listen);
> 
> my ($listen_ip, $listen_port) = split /:/, $listen;
> 
> print "listening on $listen_ip:$listen_port\n"; my $server = new
> Net::SMTP::Server($listen_ip, $listen_port)
>     or croak("Unable to handle client connection: $!\n");
> 
> while(my $conn = $server->accept()) {
>     use Socket;
>     print STDERR "From ", inet_ntoa($conn->peeraddr()), " ", scalar
localtime,
> "\n";
> 
>     my $client = new Net::SMTP::Server::Client($conn)
>         or croak("Unable to handle client connection: $!\n");
> 
>     # Process the client.  This command will block until
>     # the connecting client completes the SMTP transaction.
>     $client->process || next;
> 
>     print "MAIL FROM: $client->{FROM}\n";
>     print "RCPT TO: ". join(";", @{$client->{TO}}) . "\n";
>     print "DATA\n";
>     print $client->{MSG};
>     print "\n====================\n\n";
> }
> 
> Would you please diagnose?
> 
> Thanks,
> Kaiwang

From kaiwang.chen at gmail.com  Thu Jun  9 18:35:20 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Fri, 10 Jun 2011 00:35:20 +0800
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
 doesn'tconform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280DFE@GRFEXC.intern.adiscon.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com>
	<E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com>
	<BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com>
	<E152A03EC5F8DF4D8567006AF38A282D1ED5E5@vmexch1.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280DFE@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTinvzGpwPJ4+87bGeQhP40_Y5zw6iQ@mail.gmail.com>

There's another missing field in CreateDB.sql, which would result in
empty "Syslogmessages consolidated per Host" section in a Syslog
Summary Report. The field checksum(mapped from misc_checksum) is
required by CreateSQLStatement in classes/logstreamdb.class.php:

1320                 if ( $includeFields && $this->_arrProperties != null )
1321                 {
1322                         // Loop through all requested fields
1323                         foreach ( $this->_arrProperties as $myproperty )
1324                         {
1325                                 // SYSLOG_UID already added!
1326                                 if ( $myproperty != SYSLOG_UID &&
isset($dbmapping[$szTableType]['DBMAPPINGS'][$myproperty]) )
1327                                 {
1328                                         // Append field!
1329                                         $sqlString .= ", " .
$dbmapping[$szTableType]['DBMAPPINGS'][$myproperty];
1330                                 }
1331                         }
1332                 }

Notice the _arrProperties member in lin 1323, it should have been assigned by
  84         public function Open($arrProperties)
  97                 $this->_arrProperties = $arrProperties;

and passed into LogStreamDB by (in report.syslog.syslogsummary.class.php)
118         public function startDataProcessing()
129                 $res = $this->_streamObj->Open(
$this->_arrProperties, true );

The startDataProcessing() method would be called by RunReport() in
cmdreportgen.php to generate reports.


The definition is in ./classes/reports/report.syslog.syslogsummary.class.php
 69                 $this->_arrProperties[] = SYSLOG_UID;
 70                 $this->_arrProperties[] = SYSLOG_DATE;
 71                 $this->_arrProperties[] = SYSLOG_HOST;
 72                 $this->_arrProperties[] = SYSLOG_MESSAGETYPE;
 73                 $this->_arrProperties[] = SYSLOG_FACILITY;
 74                 $this->_arrProperties[] = SYSLOG_SEVERITY;
 75                 $this->_arrProperties[] = SYSLOG_SYSLOGTAG;
 76                 $this->_arrProperties[] = SYSLOG_PROCESSID;
 77                 $this->_arrProperties[] = SYSLOG_MESSAGE;
 78                 $this->_arrProperties[] = MISC_CHECKSUM;

Notice line 78 introduces the dependency on a checksum field in data
sources of monitorware table type. Otherwise, the CreateSQLStatement
would generate a sql like

SELECT id, devicereportedtime, fromhost, infounitid, facility,
priority, syslogtag, processid, message, checksum FROM SystemEvents
WHERE devicereportedtime > '2011-06-08 23:52:48' AND infounitid IN (1)
 ORDER BY id LIMIT 100

The ReadNextRecordsFromDB to fetch data would fail silently.


I am not sure what purpose the summary field is meant to serve.
Leaving it NULL works fine for the summary report, with the checksum
varchar(100) field added.


Thanks,
Kaiwang

2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> This field is not populated and I am bit hesitant to change the default
> template. That will probably break a number of running configurations. Also,
> I can not reliably populate that field due to the variety of different ways a
> process ID is expressed...
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Andre Lorbach
>> Sent: Wednesday, June 08, 2011 11:43 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
> doesn'tconform
>> to table type monitorware in loganalyzer-3.2.1 ?
>>
>> The ProcessID field is more or less an optional, so having a NULL value in
> it is
>> fine.
>> Populating it with the ProcessID field will be useful for filtering within
>> LogAnalyzer.
>>
>> However as far as I know, the default template does not include the
>> ProcessID field, but it can be easily extended.
>>
>> Best regards,
>> Andre Lorbach
>>
>> > -----Original Message-----
>> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> > bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> > Sent: Dienstag, 7. Juni 2011 19:34
>> > To: rsyslog-users
>> > Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
>> > conform to table type monitorware in loganalyzer-3.2.1 ?
>> >
>> > Will the default output template of rsyslog fill the new procid field?
>> > Looks like leaving it NULL should work as well.
>> >
>> > Thanks,
>> > Kaiwang
>> >
>> > 2011/6/7 Andre Lorbach <alorbach at ro1.adiscon.com>:
>> > > Hi,
>> > >
>> > > the ProcessID field was added for LogAnalyzer. It wasn't in
>> > > MonitorWare either.
>> > > But LogAnalyzer will automatically add missing fields into the
>> > > logstream databases, if the database user has sufficient rights to
>> > > the table. So granting the database user sufficient rights would
>> > > solve the
>> > problem for now.
>> > >
>> > >
>> > > Apparently adding this field into the default database schema of
>> > > MonitorWare and RSyslog was lost in communication somewhere.
>> > >
>> > > Best regards,
>> > > Andre Lorbach
>> > >
>> > >> -----Original Message-----
>> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> > >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
>> > >> Sent: Dienstag, 7. Juni 2011 16:04
>> > >> To: rsyslog-users
>> > >> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1 doesn't
>> > >> conform
>> > > to
>> > >> table type monitorware in loganalyzer-3.2.1 ?
>> > >>
>> > >> Hello,
>> > >>
>> > >> In Step 7 of installation process, "Create the first source for
>> > >> syslog messages", selecting Table type: MonitorWare (the other is
>> > >> SyslogNG) would load $dbmapping['mnoitorware'] in
>> > >> include/constants_logstream.php, resulting in SQL like this:
>> > >>
>> > >> SELECT id, devicereportedtime, facility, priority, fromhost,
>> > >> syslogtag, processid, infounitid, message FROM SystemEvents ORDER
>> > >> BY id DESC LIMIT
>> > >> 100
>> > >>
>> > >> In the case of syslog, the fields are mapped from
>> > >> ./include/functions_config.php:
>> > >>
>> > >> ?501 ? ? ? ? $CFG['Views']['SYSLOG']= array(
>> > >> ?502
>> > >> ? ? ? 'ID' => ? ? ? ? ? ? ? ? "SYSLOG",
>> > >> ?503
>> > >> ? ? ? 'DisplayName' =>"Syslog Fields",
>> > >> ?504
>> > >> ? ? ? 'Columns' => ? ?array ( SYSLOG_DATE, SYSLOG_FACILITY, S
>> > >> ? ? ?YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
>> > SYSLOG_PROCESSID,
>> > >> SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
>> > >> ?505
>> > >> ? ? ? 'userid' => ? ? ? ? ? ? null,
>> > >> ?506
>> > >> ? ? ? 'groupid' => ? ?null,
>> > >> ?507
>> > >> );
>> > >>
>> > >> Columns array:
>> > >> ? ? ? ? ? ? [0] => timereported
>> > >> ? ? ? ? ? ? [1] => syslogfacility
>> > >> ? ? ? ? ? ? [2] => syslogseverity
>> > >> ? ? ? ? ? ? [3] => FROMHOST
>> > >> ? ? ? ? ? ? [4] => syslogtag
>> > >> ? ? ? ? ? ? [5] => procid
>> > >> ? ? ? ? ? ? [6] => IUT
>> > >> ? ? ? ? ? ? [7] => msg
>> > >>
>> > >>
>> > >> Finally, I got a error prompt like this:
>> > >>
>> > >> No syslog records found - Error Details:
>> > >>
>> > >> No syslog records found
>> > >>
>> > >>
>> > >> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
>> > >> processid filed is missing)
>> > >>
>> > >> CREATE TABLE SystemEvents
>> > >> (
>> > >> ? ? ? ? ID int unsigned not null auto_increment primary key,
>> > >> ? ? ? ? CustomerID bigint,
>> > >> ? ? ? ? ReceivedAt datetime NULL,
>> > >> ? ? ? ? DeviceReportedTime datetime NULL,
>> > >> ? ? ? ? Facility smallint NULL,
>> > >> ? ? ? ? Priority smallint NULL,
>> > >> ? ? ? ? FromHost varchar(60) NULL,
>> > >> ? ? ? ? Message text,
>> > >> ? ? ? ? NTSeverity int NULL,
>> > >> ? ? ? ? Importance int NULL,
>> > >> ? ? ? ? EventSource varchar(60),
>> > >> ? ? ? ? EventUser varchar(60) NULL,
>> > >> ? ? ? ? EventCategory int NULL,
>> > >> ? ? ? ? EventID int NULL,
>> > >> ? ? ? ? EventBinaryData text NULL,
>> > >> ? ? ? ? MaxAvailable int NULL,
>> > >> ? ? ? ? CurrUsage int NULL,
>> > >> ? ? ? ? MinUsage int NULL,
>> > >> ? ? ? ? MaxUsage int NULL,
>> > >> ? ? ? ? InfoUnitID int NULL ,
>> > >> ? ? ? ? SysLogTag varchar(60),
>> > >> ? ? ? ? EventLogType varchar(60),
>> > >> ? ? ? ? GenericFileName VarChar(60),
>> > >> ? ? ? ? SystemID int NULL
>> > >> );
>> > >>
>> > >>
>> > >> So, what should I do? I heard of monitorware schema, and assumed it
>> > >> to be what shipped with rsyslog.
>> > >>
>> > >>
>> > >> Thanks,
>> > >> Kaiwang
>> > >> _______________________________________________
>> > >> rsyslog mailing list
>> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > >> http://www.rsyslog.com
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com
>> > >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From kaiwang.chen at gmail.com  Fri Jun 10 09:03:38 2011
From: kaiwang.chen at gmail.com (Kaiwang Chen)
Date: Fri, 10 Jun 2011 15:03:38 +0800
Subject: [rsyslog] How to complie mmnormalize support in rsyslog-5.8.1?
Message-ID: <BANLkTin4z0KR3dZrRygRDcmgWZXVK2oqsg@mail.gmail.com>

Hello,

In this article, mmnormalize is said to work with rsyslog 5.8.0.
http://loganalyzer.adiscon.com/articles/using-rsyslog-mmnormalize-module-effectively-with-adiscon-loganalyzer

I am evaluating v5.8.1, and failed to locate any --enable-mmnormalize
configure option nor any plugins/mmnormalize subdiretory. So what am I
missing?


Thanks,
Kaiwang

From piavka at cs.bgu.ac.il  Sun Jun 12 21:36:57 2011
From: piavka at cs.bgu.ac.il (Piavlo)
Date: Sun, 12 Jun 2011 22:36:57 +0300
Subject: [rsyslog] how to put tab instead of space between properties in
	template format
Message-ID: <4DF51559.5000504@cs.bgu.ac.il>

I've the following
$template       ImpressionsFormat, "%hostname%:%msg:::drop-last-lf%\n"

This result in "<hostname>:<space> <msg>"
and I need it to be "<hostname>:<tab> <msg>"

could not figure out how do it myself with property replacer -  as it
documents how to change or extract values in/from properties
but not how to delimit the properties.

Tried stuff like
$template       ImpressionsFormat, "%hostname%:\t%msg:::drop-last-lf%\n"
result in t character recorded

If i just hit tab asis in template this results in
This results  in "<hostname>:<tab><space> <msg>"

There must be something trivial what I'm missing?

Thanks
Alex


From rgerhards at hq.adiscon.com  Mon Jun 13 11:10:12 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 13 Jun 2011 11:10:12 +0200
Subject: [rsyslog] how to put tab instead of space between properties
	intemplate format
In-Reply-To: <4DF51559.5000504@cs.bgu.ac.il>
References: <4DF51559.5000504@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E4C@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Piavlo
> Sent: Sunday, June 12, 2011 9:37 PM
> To: rsyslog-users
> Subject: [rsyslog] how to put tab instead of space between properties
> intemplate format
> 
> I've the following
> $template       ImpressionsFormat, "%hostname%:%msg:::drop-last-lf%\n"
> 
> This result in "<hostname>:<space> <msg>"
> and I need it to be "<hostname>:<tab> <msg>"
> 
> could not figure out how do it myself with property replacer -  as it
> documents how to change or extract values in/from properties but not how
> to delimit the properties.
> 
> Tried stuff like
> $template       ImpressionsFormat, "%hostname%:\t%msg:::drop-last-lf%\n"
> result in t character recorded
Indeed, I did not implement this (surprises me as well ;). Anyhow, you can
use numerical escapes: \9 is USASCII 9 -> HT

> 
> If i just hit tab asis in template this results in This results  in
> "<hostname>:<tab><space> <msg>"

Isn't that what you want?

Rainer
> 
> There must be something trivial what I'm missing?
> 
> Thanks
> Alex
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From piavka at cs.bgu.ac.il  Mon Jun 13 11:35:53 2011
From: piavka at cs.bgu.ac.il (Piavlo)
Date: Mon, 13 Jun 2011 12:35:53 +0300
Subject: [rsyslog] how to put tab instead of space between properties
 intemplate format
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E4C@GRFEXC.intern.adiscon.com>
References: <4DF51559.5000504@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E4C@GRFEXC.intern.adiscon.com>
Message-ID: <4DF5D9F9.4080705@cs.bgu.ac.il>


>> Tried stuff like
>> $template       ImpressionsFormat, "%hostname%:\t%msg:::drop-last-lf%\n"
>> result in t character recorded
>>     
> Indeed, I did not implement this (surprises me as well ;). Anyhow, you can
> use numerical escapes: \9 is USASCII 9 -> HT
>   
This does not help since this adds the tab - but the space is still added!!!
>   
>> If i just hit tab asis in template this results in This results  in
>> "<hostname>:<tab><space> <msg>"
>>     
> Isn't that what you want?
>   

No I want

"<hostname>:<tab><msg>"
and not
"<hostname>:<tab><space><msg>"


Thanks
Alex

From piavka at cs.bgu.ac.il  Tue Jun 14 10:58:55 2011
From: piavka at cs.bgu.ac.il (Piavlo)
Date: Tue, 14 Jun 2011 11:58:55 +0300
Subject: [rsyslog] failover and actionqueue combination problem
Message-ID: <4DF722CF.1080906@cs.bgu.ac.il>

 Hi,

I'm trying to implement the following scenario
Messages should be sent to remote server jobs1a
If message can't deliver message to jobs1a rsyslog should
send it to jobs2a, if delivering to jobs2a fails too then rsyslog should
again try to sending to jobs1a again this time using disk queue - so
that logs could be buffered and sent to jobs1a
later then it recovers.

so I've made the following config
--------------------------------------------
local3.info @@jobs1a.internal:10514

$ActionExecOnlyWhenPreviousIsSuspended on

& @@jobs2a.internal:10514

$ActionQueueType LinkedList
$ActionQueueFileName forward_access_log
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
& @@jobs1a.internal:10514

$ActionExecOnlyWhenPreviousIsSuspended off
--------------------------------------------

The problem is that it does not seem to work messages which are logged
while both jobs1a & jobs2a there down are lost.
Then jobs1a recovers it receives only new logs. I've also tried
simplifying the failure scenario by first stopping rsyslog on jobs2a and
then jobs1a
and the starting it on jobs2a with same results.

Not that the following works ok - and messages which were queued while
jobs1a was down are delivered to it then it recovers.
-----------------------------------------
$ActionQueueType LinkedList
$ActionQueueFileName forward_access_log
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
& @@jobs1a.internal:10514
-----------------------------------------

Any idea what I'm doing wrong?
Is it possible to achieve such scenario with rsyslog ?

Thanks
Alex

From rgerhards at hq.adiscon.com  Tue Jun 14 11:02:28 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 14 Jun 2011 11:02:28 +0200
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <4DF722CF.1080906@cs.bgu.ac.il>
References: <4DF722CF.1080906@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com>

It would be good to have a debug log with the failing failover scenario ;).
Just be warned in advance that I am currently deep inside another debugging
issue and there is one more log waiting in queue, so I will probably not be
able to look at it immediately (other than a quick glimpse).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Piavlo
> Sent: Tuesday, June 14, 2011 10:59 AM
> To: rsyslog-users
> Subject: [rsyslog] failover and actionqueue combination problem
> 
>  Hi,
> 
> I'm trying to implement the following scenario Messages should be sent to
> remote server jobs1a If message can't deliver message to jobs1a rsyslog
> should send it to jobs2a, if delivering to jobs2a fails too then rsyslog
should
> again try to sending to jobs1a again this time using disk queue - so that
logs
> could be buffered and sent to jobs1a later then it recovers.
> 
> so I've made the following config
> --------------------------------------------
> local3.info @@jobs1a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended on
> 
> & @@jobs2a.internal:10514
> 
> $ActionQueueType LinkedList
> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> $ActionResumeRetryCount -1
> & @@jobs1a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended off
> --------------------------------------------
> 
> The problem is that it does not seem to work messages which are logged
> while both jobs1a & jobs2a there down are lost.
> Then jobs1a recovers it receives only new logs. I've also tried simplifying
the
> failure scenario by first stopping rsyslog on jobs2a and then jobs1a and
the
> starting it on jobs2a with same results.
> 
> Not that the following works ok - and messages which were queued while
> jobs1a was down are delivered to it then it recovers.
> -----------------------------------------
> $ActionQueueType LinkedList
> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> $ActionResumeRetryCount -1
> & @@jobs1a.internal:10514
> -----------------------------------------
> 
> Any idea what I'm doing wrong?
> Is it possible to achieve such scenario with rsyslog ?
> 
> Thanks
> Alex
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Jun 14 11:38:35 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 14 Jun 2011 11:38:35 +0200
Subject: [rsyslog] how to put tab instead of space between properties
	intemplate format
In-Reply-To: <4DF5D9F9.4080705@cs.bgu.ac.il>
References: <4DF51559.5000504@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E4C@GRFEXC.intern.adiscon.com>
	<4DF5D9F9.4080705@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E5B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: Piavlo [mailto:piavka at cs.bgu.ac.il]
> Sent: Monday, June 13, 2011 11:36 AM
> To: rsyslog-users
> Cc: Rainer Gerhards
> Subject: Re: [rsyslog] how to put tab instead of space between properties
> intemplate format
> 
> 
> >> Tried stuff like
> >> $template       ImpressionsFormat, "%hostname%:\t%msg:::drop-last-
> lf%\n"
> >> result in t character recorded
> >>
> > Indeed, I did not implement this (surprises me as well ;). Anyhow, you
> > can use numerical escapes: \9 is USASCII 9 -> HT
> >
> This does not help since this adds the tab - but the space is still
added!!!
> >
> >> If i just hit tab asis in template this results in This results  in
> >> "<hostname>:<tab><space> <msg>"
> >>
> > Isn't that what you want?
> >
> 
> No I want
> 
> "<hostname>:<tab><msg>"
> and not
> "<hostname>:<tab><space><msg>"

I see your problem. <msg> CONTAINS the space, so you actually want to drop
the first space of the message! I assume that you deal with legacy/RFC3164
message. These usually contain a space as the first character (see the
RFC3164 description of the MSG field).

Rainer
> 
> 
> Thanks
> Alex

From rgerhards at hq.adiscon.com  Tue Jun 14 15:05:01 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 14 Jun 2011 15:05:01 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DEFD8B8.6040506@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>
	<4DEFD8B8.6040506@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>

Sorry, I have been swamped with another issue, which took up all my time. I
have now reviewed the log but it looks like it does not contain any instance
where the rules are actually triggered. Could you create such a one?

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, June 08, 2011 10:17 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> On 6/2/11 2:19 PM, Rory Toma wrote:
> > On 6/2/11 3:32 AM, Rainer Gerhards wrote:
> >> Mmhhh.. can you post a complete debug log (maybe via a website like
> >> filebin)?
> >>
> >> Thx,
> >> Rainer
> >>
> > Let me know when you've downloaded this file and I'll reset the ACL, thx.
> >
> > http://www.colinburns.com/downloads/darwin0.bz2
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> Any update on this?
> 
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From tbergfeld at hq.adiscon.com  Tue Jun 14 15:41:43 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Tue, 14 Jun 2011 15:41:43 +0200
Subject: [rsyslog] rsyslog 6.1.9 (v6-beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E6E@GRFEXC.intern.adiscon.com>

This is a maintenance release offering a set of bug fixes. It is suggested
that users of the v6-beta branch update to this release.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-9/

Download:

http://www.rsyslog.com/rsyslog-6-1-9-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From rory at ooma.com  Wed Jun 15 00:49:28 2011
From: rory at ooma.com (Rory Toma)
Date: Tue, 14 Jun 2011 15:49:28 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
Message-ID: <4DF7E578.7040303@ooma.com>

The rules should have been triggered. I blocked (via firewall) the 
primary syslog server, and tcpdumps shows that it just keeps sending to 
it, even though it could not connect. The fact that the rules are not 
being triggered means that either somehow the config is wrong, or there 
is a bug. How does rsyslog determine the "upness" of the server it tries 
to connect to?

On 6/14/11 6:05 AM, Rainer Gerhards wrote:
> Sorry, I have been swamped with another issue, which took up all my time. I
> have now reviewed the log but it looks like it does not contain any instance
> where the rules are actually triggered. Could you create such a one?
>
> Thanks,
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>> Sent: Wednesday, June 08, 2011 10:17 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> On 6/2/11 2:19 PM, Rory Toma wrote:
>>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
>>>> Mmhhh.. can you post a complete debug log (maybe via a website like
>>>> filebin)?
>>>>
>>>> Thx,
>>>> Rainer
>>>>
>>> Let me know when you've downloaded this file and I'll reset the ACL, thx.
>>>
>>> http://www.colinburns.com/downloads/darwin0.bz2
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> Any update on this?
>>
>> thx
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun 15 08:42:05 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 08:42:05 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DF7E578.7040303@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>

The "upness" is determined by connecting. But I really can't say much more
until I get a log that was recorded while the problem happened. The log you
provided contains just the startup but no message flow at all.

Rainer

> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Wednesday, June 15, 2011 12:49 AM
> To: rsyslog-users
> Cc: Rainer Gerhards
> Subject: Re: [rsyslog] Question on host failover
> 
> The rules should have been triggered. I blocked (via firewall) the
> primary syslog server, and tcpdumps shows that it just keeps sending to
> it, even though it could not connect. The fact that the rules are not
> being triggered means that either somehow the config is wrong, or there
> is a bug. How does rsyslog determine the "upness" of the server it
> tries
> to connect to?
> 
> On 6/14/11 6:05 AM, Rainer Gerhards wrote:
> > Sorry, I have been swamped with another issue, which took up all my
> time. I
> > have now reviewed the log but it looks like it does not contain any
> instance
> > where the rules are actually triggered. Could you create such a one?
> >
> > Thanks,
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >> Sent: Wednesday, June 08, 2011 10:17 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> On 6/2/11 2:19 PM, Rory Toma wrote:
> >>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
> >>>> Mmhhh.. can you post a complete debug log (maybe via a website
> like
> >>>> filebin)?
> >>>>
> >>>> Thx,
> >>>> Rainer
> >>>>
> >>> Let me know when you've downloaded this file and I'll reset the
> ACL, thx.
> >>>
> >>> http://www.colinburns.com/downloads/darwin0.bz2
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com
> >> Any update on this?
> >>
> >> thx
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com


From piavka at cs.bgu.ac.il  Wed Jun 15 09:47:11 2011
From: piavka at cs.bgu.ac.il (Piavlo)
Date: Wed, 15 Jun 2011 10:47:11 +0300
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com>
References: <4DF722CF.1080906@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com>
Message-ID: <4DF8637F.8040901@cs.bgu.ac.il>

On 06/14/2011 12:02 PM, Rainer Gerhards wrote:
> It would be good to have a debug log with the failing failover scenario ;).
>   
So basically you are saying that the config part looks correct?

Anyway I simplified the scenario to the following config by taking the
second redundant rsyslog out of picture

--------------------------------------------
local3.info @@jobs2a.internal:10514

$ActionExecOnlyWhenPreviousIsSuspended on

$ActionQueueType LinkedList
$ActionQueueFileName forward_access_log
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
& @@jobs2a.internal:10514

$ActionExecOnlyWhenPreviousIsSuspended off
--------------------------------------------

So that now it tries to send to jobs2a and if it fails on next try sends
it to jobs2a with disk queue buffering.
This also does not work then remote jobs2a rsyslog recovers - the
messages which were enqueued localy during failure are not recieved on
jobs2a
but only new ones.

I'm attaching the debug log - it's truncated - since it appears to
endlessly print the following lines in loop
------------
0364.703265000:42003940: we deleted 0 objects and enqueued 1 objects
0364.703276000:42003940: delete batch from store, new sizes: log 1, phys 1
0364.703287000:42003940: regular consumer finished, iret=0, szlog 0 sz
phys 1
0364.703298000:42003940: XXX: DeleteProcessedBatch re-enqueue 0 of 1,
state 0
0364.703309000:42003940: action 10 queue: entry added, size now log 1,
phys 2 entries
-----------
untill the DoDie is called - then I stop the rsyslog in debug mode on
the sending side.

Do you see anything interesting in the attached log?

Thanks
Alex

> Just be warned in advance that I am currently deep inside another debugging
> issue and there is one more log waiting in queue, so I will probably not be
> able to look at it immediately (other than a quick glimpse).
>
> Rainer
>
>   
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Piavlo
>> Sent: Tuesday, June 14, 2011 10:59 AM
>> To: rsyslog-users
>> Subject: [rsyslog] failover and actionqueue combination problem
>>
>>  Hi,
>>
>> I'm trying to implement the following scenario Messages should be sent to
>> remote server jobs1a If message can't deliver message to jobs1a rsyslog
>> should send it to jobs2a, if delivering to jobs2a fails too then rsyslog
>>     
> should
>   
>> again try to sending to jobs1a again this time using disk queue - so that
>>     
> logs
>   
>> could be buffered and sent to jobs1a later then it recovers.
>>
>> so I've made the following config
>> --------------------------------------------
>> local3.info @@jobs1a.internal:10514
>>
>> $ActionExecOnlyWhenPreviousIsSuspended on
>>
>> & @@jobs2a.internal:10514
>>
>> $ActionQueueType LinkedList
>> $ActionQueueFileName forward_access_log
>> $ActionQueueSaveOnShutdown on
>> $ActionResumeRetryCount -1
>> & @@jobs1a.internal:10514
>>
>> $ActionExecOnlyWhenPreviousIsSuspended off
>> --------------------------------------------
>>
>> The problem is that it does not seem to work messages which are logged
>> while both jobs1a & jobs2a there down are lost.
>> Then jobs1a recovers it receives only new logs. I've also tried simplifying
>>     
> the
>   
>> failure scenario by first stopping rsyslog on jobs2a and then jobs1a and
>>     
> the
>   
>> starting it on jobs2a with same results.
>>
>> Not that the following works ok - and messages which were queued while
>> jobs1a was down are delivered to it then it recovers.
>> -----------------------------------------
>> $ActionQueueType LinkedList
>> $ActionQueueFileName forward_access_log
>> $ActionQueueSaveOnShutdown on
>> $ActionResumeRetryCount -1
>> & @@jobs1a.internal:10514
>> -----------------------------------------
>>
>> Any idea what I'm doing wrong?
>> Is it possible to achieve such scenario with rsyslog ?
>>
>> Thanks
>> Alex
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
>>     
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>   

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rsyslog.debug1
URL: <http://lists.adiscon.net/pipermail/rsyslog/attachments/20110615/6373049d/attachment-0001.txt>

From roland.kamke at auconet.com  Wed Jun 15 09:55:09 2011
From: roland.kamke at auconet.com (Roland Kamke)
Date: Wed, 15 Jun 2011 09:55:09 +0200
Subject: [rsyslog] Special chars in database settings
Message-ID: <69DFC544026A844790C013C0FB16B9E0BC75DE@eta.firma.lan>

Hello,

I am using rsyslog 5.7.10 and redirect all messages to a database:

# Define the SQL statement that inserts syslog events to the data base
$template abcDefaultFormat,"insert into SyslogEvents(ReceivedAt,
DeviceReportedTime, Facility, Priority, FromHost, Tag, Message) values
('%timegenerated:::date-pgsql%', '%timereported:::date-pgsql%',
'%syslogfacility%', '%syslogpriority%', '%HOSTNAME%', '%syslogtag%',
'%msg%')\n",SQL

# Log everything to PostgreSQL
*.*
:ompgsql:localhost,syslogDB,syslogUSER,syslogPASSWORD;abcDefaultFormat
# And for debugging
*.*  /var/log/messages

One of our customers has a strict policy for user ids and passwords:
These pieces of data must contain special characters. Particularly the
comma (,), the semicolon (;) and the hash (#) are to be supported.
How would I add such characters to the configuration file?

I tried things like
*.*  :ompgsql:localhost,syslogDB,syslogUSER,syslog#PASSWORD;abcFormat
*.*  :ompgsql:localhost,syslogDB,syslogUSER,syslog\#PASSWORD;abcFormat
*.*  :ompgsql:localhost,syslogDB,syslogUSER,'syslog#PASSWORD';abcFormat

*.*  :ompgsql:localhost,syslogDB,syslog#USER,syslogPASSWORD;abcFormat
*.*  :ompgsql:localhost,syslogDB,syslog\#USER,syslogPASSWORD;abcFormat
*.*  :ompgsql:localhost,syslogDB,'syslog#USER',syslogPASSWORD;abcFormat

In each of these attempts to include a hash I did not receive any hint
that rsyslogd started (be the start successfully or with problems
reading the configuration) in the /var/log/messages (omitting the output
module configuration line or using "simpler" credentials I always
received a start up line there). Triggering a syslog message via the
logger command did also not produce any reaction in /var/log/messages.
And of course, there was no reaction in the database that receives all
the messages if special characters were used in the configuration.

Although rsyslogd was visible in the "ps ax" output it simply seemed to
be dead, silently irresponsive.

Any idea how to go on? Is it possible at all to include special
characters in configuration items such as database access data? If so,
how to escape them?

	Kind regards
	Roland


From rgerhards at hq.adiscon.com  Wed Jun 15 10:21:38 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 10:21:38 +0200
Subject: [rsyslog] Special chars in database settings
In-Reply-To: <69DFC544026A844790C013C0FB16B9E0BC75DE@eta.firma.lan>
References: <69DFC544026A844790C013C0FB16B9E0BC75DE@eta.firma.lan>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E79@GRFEXC.intern.adiscon.com>

I am sorry, but that's currently not supported. You need to modify the source
:(

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Roland Kamke
> Sent: Wednesday, June 15, 2011 9:55 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Special chars in database settings
> 
> Hello,
> 
> I am using rsyslog 5.7.10 and redirect all messages to a database:
> 
> # Define the SQL statement that inserts syslog events to the data base
> $template abcDefaultFormat,"insert into SyslogEvents(ReceivedAt,
> DeviceReportedTime, Facility, Priority, FromHost, Tag, Message) values
> ('%timegenerated:::date-pgsql%', '%timereported:::date-pgsql%',
> '%syslogfacility%', '%syslogpriority%', '%HOSTNAME%', '%syslogtag%',
> '%msg%')\n",SQL
> 
> # Log everything to PostgreSQL
> *.*
> :ompgsql:localhost,syslogDB,syslogUSER,syslogPASSWORD;abcDefaultFormat
> # And for debugging
> *.*  /var/log/messages
> 
> One of our customers has a strict policy for user ids and passwords:
> These pieces of data must contain special characters. Particularly the
> comma (,), the semicolon (;) and the hash (#) are to be supported.
> How would I add such characters to the configuration file?
> 
> I tried things like
> *.*  :ompgsql:localhost,syslogDB,syslogUSER,syslog#PASSWORD;abcFormat
> *.*  :ompgsql:localhost,syslogDB,syslogUSER,syslog\#PASSWORD;abcFormat
> *.*  :ompgsql:localhost,syslogDB,syslogUSER,'syslog#PASSWORD';abcFormat
> 
> *.*  :ompgsql:localhost,syslogDB,syslog#USER,syslogPASSWORD;abcFormat
> *.*  :ompgsql:localhost,syslogDB,syslog\#USER,syslogPASSWORD;abcFormat
> *.*  :ompgsql:localhost,syslogDB,'syslog#USER',syslogPASSWORD;abcFormat
> 
> In each of these attempts to include a hash I did not receive any hint
> that rsyslogd started (be the start successfully or with problems
> reading the configuration) in the /var/log/messages (omitting the
> output
> module configuration line or using "simpler" credentials I always
> received a start up line there). Triggering a syslog message via the
> logger command did also not produce any reaction in /var/log/messages.
> And of course, there was no reaction in the database that receives all
> the messages if special characters were used in the configuration.
> 
> Although rsyslogd was visible in the "ps ax" output it simply seemed to
> be dead, silently irresponsive.
> 
> Any idea how to go on? Is it possible at all to include special
> characters in configuration items such as database access data? If so,
> how to escape them?
> 
> 	Kind regards
> 	Roland
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun 15 11:38:51 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 11:38:51 +0200
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <4DF8637F.8040901@cs.bgu.ac.il>
References: <4DF722CF.1080906@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com>
	<4DF8637F.8040901@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E7E@GRFEXC.intern.adiscon.com>

I am looking at the debug log, but I see that you use 5.8.0. I remember I
have patched some things in regard to failover in 5.8.1. Could you retry with
that version (I still continue to analyze).

Rainer

> -----Original Message-----
> From: Piavlo [mailto:piavka at cs.bgu.ac.il]
> Sent: Wednesday, June 15, 2011 9:47 AM
> To: rsyslog-users
> Cc: Rainer Gerhards
> Subject: Re: [rsyslog] failover and actionqueue combination problem
> 
> On 06/14/2011 12:02 PM, Rainer Gerhards wrote:
> > It would be good to have a debug log with the failing failover scenario
;).
> >
> So basically you are saying that the config part looks correct?
> 
> Anyway I simplified the scenario to the following config by taking the
second
> redundant rsyslog out of picture
> 
> --------------------------------------------
> local3.info @@jobs2a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended on
> 
> $ActionQueueType LinkedList
> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> $ActionResumeRetryCount -1
> & @@jobs2a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended off
> --------------------------------------------
> 
> So that now it tries to send to jobs2a and if it fails on next try sends it
to
> jobs2a with disk queue buffering.
> This also does not work then remote jobs2a rsyslog recovers - the messages
> which were enqueued localy during failure are not recieved on jobs2a but
> only new ones.
> 
> I'm attaching the debug log - it's truncated - since it appears to
endlessly print
> the following lines in loop
> ------------
> 0364.703265000:42003940: we deleted 0 objects and enqueued 1 objects
> 0364.703276000:42003940: delete batch from store, new sizes: log 1, phys 1
> 0364.703287000:42003940: regular consumer finished, iret=0, szlog 0 sz phys
1
> 0364.703298000:42003940: XXX: DeleteProcessedBatch re-enqueue 0 of 1,
> state 0
> 0364.703309000:42003940: action 10 queue: entry added, size now log 1, phys
> 2 entries
> -----------
> untill the DoDie is called - then I stop the rsyslog in debug mode on the
> sending side.
> 
> Do you see anything interesting in the attached log?
> 
> Thanks
> Alex
> 
> > Just be warned in advance that I am currently deep inside another
> > debugging issue and there is one more log waiting in queue, so I will
> > probably not be able to look at it immediately (other than a quick
glimpse).
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Piavlo
> >> Sent: Tuesday, June 14, 2011 10:59 AM
> >> To: rsyslog-users
> >> Subject: [rsyslog] failover and actionqueue combination problem
> >>
> >>  Hi,
> >>
> >> I'm trying to implement the following scenario Messages should be
> >> sent to remote server jobs1a If message can't deliver message to
> >> jobs1a rsyslog should send it to jobs2a, if delivering to jobs2a
> >> fails too then rsyslog
> >>
> > should
> >
> >> again try to sending to jobs1a again this time using disk queue - so
> >> that
> >>
> > logs
> >
> >> could be buffered and sent to jobs1a later then it recovers.
> >>
> >> so I've made the following config
> >> --------------------------------------------
> >> local3.info @@jobs1a.internal:10514
> >>
> >> $ActionExecOnlyWhenPreviousIsSuspended on
> >>
> >> & @@jobs2a.internal:10514
> >>
> >> $ActionQueueType LinkedList
> >> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> >>
> >> $ActionExecOnlyWhenPreviousIsSuspended off
> >> --------------------------------------------
> >>
> >> The problem is that it does not seem to work messages which are
> >> logged while both jobs1a & jobs2a there down are lost.
> >> Then jobs1a recovers it receives only new logs. I've also tried
> >> simplifying
> >>
> > the
> >
> >> failure scenario by first stopping rsyslog on jobs2a and then jobs1a
> >> and
> >>
> > the
> >
> >> starting it on jobs2a with same results.
> >>
> >> Not that the following works ok - and messages which were queued
> >> while jobs1a was down are delivered to it then it recovers.
> >> -----------------------------------------
> >> $ActionQueueType LinkedList
> >> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> >> -----------------------------------------
> >>
> >> Any idea what I'm doing wrong?
> >> Is it possible to achieve such scenario with rsyslog ?
> >>
> >> Thanks
> >> Alex
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >


From rgerhards at hq.adiscon.com  Wed Jun 15 11:45:45 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 11:45:45 +0200
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <4DF8637F.8040901@cs.bgu.ac.il>
References: <4DF722CF.1080906@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com>
	<4DF8637F.8040901@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E81@GRFEXC.intern.adiscon.com>

OK, one thing: $WorkDirectory does not like trailing slashes. While no doubt
it should gracefully handle them, the current code runs into problems. This
could cause the symptoms you see. So remove the trailing slash while I work
on a patch to handle that situation more intelligently...

Rainer

> -----Original Message-----
> From: Piavlo [mailto:piavka at cs.bgu.ac.il]
> Sent: Wednesday, June 15, 2011 9:47 AM
> To: rsyslog-users
> Cc: Rainer Gerhards
> Subject: Re: [rsyslog] failover and actionqueue combination problem
> 
> On 06/14/2011 12:02 PM, Rainer Gerhards wrote:
> > It would be good to have a debug log with the failing failover scenario
;).
> >
> So basically you are saying that the config part looks correct?
> 
> Anyway I simplified the scenario to the following config by taking the
second
> redundant rsyslog out of picture
> 
> --------------------------------------------
> local3.info @@jobs2a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended on
> 
> $ActionQueueType LinkedList
> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> $ActionResumeRetryCount -1
> & @@jobs2a.internal:10514
> 
> $ActionExecOnlyWhenPreviousIsSuspended off
> --------------------------------------------
> 
> So that now it tries to send to jobs2a and if it fails on next try sends it
to
> jobs2a with disk queue buffering.
> This also does not work then remote jobs2a rsyslog recovers - the messages
> which were enqueued localy during failure are not recieved on jobs2a but
> only new ones.
> 
> I'm attaching the debug log - it's truncated - since it appears to
endlessly print
> the following lines in loop
> ------------
> 0364.703265000:42003940: we deleted 0 objects and enqueued 1 objects
> 0364.703276000:42003940: delete batch from store, new sizes: log 1, phys 1
> 0364.703287000:42003940: regular consumer finished, iret=0, szlog 0 sz phys
1
> 0364.703298000:42003940: XXX: DeleteProcessedBatch re-enqueue 0 of 1,
> state 0
> 0364.703309000:42003940: action 10 queue: entry added, size now log 1, phys
> 2 entries
> -----------
> untill the DoDie is called - then I stop the rsyslog in debug mode on the
> sending side.
> 
> Do you see anything interesting in the attached log?
> 
> Thanks
> Alex
> 
> > Just be warned in advance that I am currently deep inside another
> > debugging issue and there is one more log waiting in queue, so I will
> > probably not be able to look at it immediately (other than a quick
glimpse).
> >
> > Rainer
> >
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Piavlo
> >> Sent: Tuesday, June 14, 2011 10:59 AM
> >> To: rsyslog-users
> >> Subject: [rsyslog] failover and actionqueue combination problem
> >>
> >>  Hi,
> >>
> >> I'm trying to implement the following scenario Messages should be
> >> sent to remote server jobs1a If message can't deliver message to
> >> jobs1a rsyslog should send it to jobs2a, if delivering to jobs2a
> >> fails too then rsyslog
> >>
> > should
> >
> >> again try to sending to jobs1a again this time using disk queue - so
> >> that
> >>
> > logs
> >
> >> could be buffered and sent to jobs1a later then it recovers.
> >>
> >> so I've made the following config
> >> --------------------------------------------
> >> local3.info @@jobs1a.internal:10514
> >>
> >> $ActionExecOnlyWhenPreviousIsSuspended on
> >>
> >> & @@jobs2a.internal:10514
> >>
> >> $ActionQueueType LinkedList
> >> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> >>
> >> $ActionExecOnlyWhenPreviousIsSuspended off
> >> --------------------------------------------
> >>
> >> The problem is that it does not seem to work messages which are
> >> logged while both jobs1a & jobs2a there down are lost.
> >> Then jobs1a recovers it receives only new logs. I've also tried
> >> simplifying
> >>
> > the
> >
> >> failure scenario by first stopping rsyslog on jobs2a and then jobs1a
> >> and
> >>
> > the
> >
> >> starting it on jobs2a with same results.
> >>
> >> Not that the following works ok - and messages which were queued
> >> while jobs1a was down are delivered to it then it recovers.
> >> -----------------------------------------
> >> $ActionQueueType LinkedList
> >> $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> >> -----------------------------------------
> >>
> >> Any idea what I'm doing wrong?
> >> Is it possible to achieve such scenario with rsyslog ?
> >>
> >> Thanks
> >> Alex
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> >>
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >


From rgerhards at hq.adiscon.com  Wed Jun 15 12:31:22 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 12:31:22 +0200
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E81@GRFEXC.intern.adiscon.com>
References: <4DF722CF.1080906@cs.bgu.ac.il><9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com><4DF8637F.8040901@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280E81@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E82@GRFEXC.intern.adiscon.com>

The $WorkDirectory is patched, but there seems to be some other problem. I
can repro and am looking into it. So I currently do NOT need more information
from you.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, June 15, 2011 11:46 AM
> To: Piavlo; rsyslog-users
> Subject: Re: [rsyslog] failover and actionqueue combination problem
> 
> OK, one thing: $WorkDirectory does not like trailing slashes. While no
doubt it
> should gracefully handle them, the current code runs into problems. This
> could cause the symptoms you see. So remove the trailing slash while I work
> on a patch to handle that situation more intelligently...
> 
> Rainer
> 
> > -----Original Message-----
> > From: Piavlo [mailto:piavka at cs.bgu.ac.il]
> > Sent: Wednesday, June 15, 2011 9:47 AM
> > To: rsyslog-users
> > Cc: Rainer Gerhards
> > Subject: Re: [rsyslog] failover and actionqueue combination problem
> >
> > On 06/14/2011 12:02 PM, Rainer Gerhards wrote:
> > > It would be good to have a debug log with the failing failover
> > > scenario
> ;).
> > >
> > So basically you are saying that the config part looks correct?
> >
> > Anyway I simplified the scenario to the following config by taking the
> second
> > redundant rsyslog out of picture
> >
> > --------------------------------------------
> > local3.info @@jobs2a.internal:10514
> >
> > $ActionExecOnlyWhenPreviousIsSuspended on
> >
> > $ActionQueueType LinkedList
> > $ActionQueueFileName forward_access_log
> $ActionQueueSaveOnShutdown on
> > $ActionResumeRetryCount -1 & @@jobs2a.internal:10514
> >
> > $ActionExecOnlyWhenPreviousIsSuspended off
> > --------------------------------------------
> >
> > So that now it tries to send to jobs2a and if it fails on next try
> > sends it
> to
> > jobs2a with disk queue buffering.
> > This also does not work then remote jobs2a rsyslog recovers - the
> > messages which were enqueued localy during failure are not recieved on
> > jobs2a but only new ones.
> >
> > I'm attaching the debug log - it's truncated - since it appears to
> endlessly print
> > the following lines in loop
> > ------------
> > 0364.703265000:42003940: we deleted 0 objects and enqueued 1 objects
> > 0364.703276000:42003940: delete batch from store, new sizes: log 1,
> > phys 1
> > 0364.703287000:42003940: regular consumer finished, iret=0, szlog 0 sz
> > phys
> 1
> > 0364.703298000:42003940: XXX: DeleteProcessedBatch re-enqueue 0 of 1,
> > state 0
> > 0364.703309000:42003940: action 10 queue: entry added, size now log 1,
> > phys
> > 2 entries
> > -----------
> > untill the DoDie is called - then I stop the rsyslog in debug mode on
> > the sending side.
> >
> > Do you see anything interesting in the attached log?
> >
> > Thanks
> > Alex
> >
> > > Just be warned in advance that I am currently deep inside another
> > > debugging issue and there is one more log waiting in queue, so I
> > > will probably not be able to look at it immediately (other than a
> > > quick
> glimpse).
> > >
> > > Rainer
> > >
> > >
> > >> -----Original Message-----
> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > >> bounces at lists.adiscon.com] On Behalf Of Piavlo
> > >> Sent: Tuesday, June 14, 2011 10:59 AM
> > >> To: rsyslog-users
> > >> Subject: [rsyslog] failover and actionqueue combination problem
> > >>
> > >>  Hi,
> > >>
> > >> I'm trying to implement the following scenario Messages should be
> > >> sent to remote server jobs1a If message can't deliver message to
> > >> jobs1a rsyslog should send it to jobs2a, if delivering to jobs2a
> > >> fails too then rsyslog
> > >>
> > > should
> > >
> > >> again try to sending to jobs1a again this time using disk queue -
> > >> so that
> > >>
> > > logs
> > >
> > >> could be buffered and sent to jobs1a later then it recovers.
> > >>
> > >> so I've made the following config
> > >> --------------------------------------------
> > >> local3.info @@jobs1a.internal:10514
> > >>
> > >> $ActionExecOnlyWhenPreviousIsSuspended on
> > >>
> > >> & @@jobs2a.internal:10514
> > >>
> > >> $ActionQueueType LinkedList
> > >> $ActionQueueFileName forward_access_log
> > $ActionQueueSaveOnShutdown on
> > >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> > >>
> > >> $ActionExecOnlyWhenPreviousIsSuspended off
> > >> --------------------------------------------
> > >>
> > >> The problem is that it does not seem to work messages which are
> > >> logged while both jobs1a & jobs2a there down are lost.
> > >> Then jobs1a recovers it receives only new logs. I've also tried
> > >> simplifying
> > >>
> > > the
> > >
> > >> failure scenario by first stopping rsyslog on jobs2a and then
> > >> jobs1a and
> > >>
> > > the
> > >
> > >> starting it on jobs2a with same results.
> > >>
> > >> Not that the following works ok - and messages which were queued
> > >> while jobs1a was down are delivered to it then it recovers.
> > >> -----------------------------------------
> > >> $ActionQueueType LinkedList
> > >> $ActionQueueFileName forward_access_log
> > $ActionQueueSaveOnShutdown on
> > >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> > >> -----------------------------------------
> > >>
> > >> Any idea what I'm doing wrong?
> > >> Is it possible to achieve such scenario with rsyslog ?
> > >>
> > >> Thanks
> > >> Alex
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com
> > >>
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > >
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Wed Jun 15 12:49:02 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 12:49:02 +0200
Subject: [rsyslog] failover and actionqueue combination problem
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E82@GRFEXC.intern.adiscon.com>
References: <4DF722CF.1080906@cs.bgu.ac.il><9B6E2A8877C38245BFB15CC491A11DA7280E57@GRFEXC.intern.adiscon.com><4DF8637F.8040901@cs.bgu.ac.il><9B6E2A8877C38245BFB15CC491A11DA7280E81@GRFEXC.intern.adiscon.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E82@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E83@GRFEXC.intern.adiscon.com>

I have created a bug tracker, suggest to subscribe:

http://bugzilla.adiscon.com/show_bug.cgi?id=270

As can be seen there, doesn't even work with a further simplified config.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, June 15, 2011 12:31 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] failover and actionqueue combination problem
> 
> The $WorkDirectory is patched, but there seems to be some other problem. I
> can repro and am looking into it. So I currently do NOT need more
> information from you.
> 
> Rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> > Sent: Wednesday, June 15, 2011 11:46 AM
> > To: Piavlo; rsyslog-users
> > Subject: Re: [rsyslog] failover and actionqueue combination problem
> >
> > OK, one thing: $WorkDirectory does not like trailing slashes. While no
> doubt it
> > should gracefully handle them, the current code runs into problems.
> > This could cause the symptoms you see. So remove the trailing slash
> > while I work on a patch to handle that situation more intelligently...
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: Piavlo [mailto:piavka at cs.bgu.ac.il]
> > > Sent: Wednesday, June 15, 2011 9:47 AM
> > > To: rsyslog-users
> > > Cc: Rainer Gerhards
> > > Subject: Re: [rsyslog] failover and actionqueue combination problem
> > >
> > > On 06/14/2011 12:02 PM, Rainer Gerhards wrote:
> > > > It would be good to have a debug log with the failing failover
> > > > scenario
> > ;).
> > > >
> > > So basically you are saying that the config part looks correct?
> > >
> > > Anyway I simplified the scenario to the following config by taking
> > > the
> > second
> > > redundant rsyslog out of picture
> > >
> > > --------------------------------------------
> > > local3.info @@jobs2a.internal:10514
> > >
> > > $ActionExecOnlyWhenPreviousIsSuspended on
> > >
> > > $ActionQueueType LinkedList
> > > $ActionQueueFileName forward_access_log
> > $ActionQueueSaveOnShutdown on
> > > $ActionResumeRetryCount -1 & @@jobs2a.internal:10514
> > >
> > > $ActionExecOnlyWhenPreviousIsSuspended off
> > > --------------------------------------------
> > >
> > > So that now it tries to send to jobs2a and if it fails on next try
> > > sends it
> > to
> > > jobs2a with disk queue buffering.
> > > This also does not work then remote jobs2a rsyslog recovers - the
> > > messages which were enqueued localy during failure are not recieved
> > > on jobs2a but only new ones.
> > >
> > > I'm attaching the debug log - it's truncated - since it appears to
> > endlessly print
> > > the following lines in loop
> > > ------------
> > > 0364.703265000:42003940: we deleted 0 objects and enqueued 1 objects
> > > 0364.703276000:42003940: delete batch from store, new sizes: log 1,
> > > phys 1
> > > 0364.703287000:42003940: regular consumer finished, iret=0, szlog 0
> > > sz phys
> > 1
> > > 0364.703298000:42003940: XXX: DeleteProcessedBatch re-enqueue 0 of
> > > 1, state 0
> > > 0364.703309000:42003940: action 10 queue: entry added, size now log
> > > 1, phys
> > > 2 entries
> > > -----------
> > > untill the DoDie is called - then I stop the rsyslog in debug mode
> > > on the sending side.
> > >
> > > Do you see anything interesting in the attached log?
> > >
> > > Thanks
> > > Alex
> > >
> > > > Just be warned in advance that I am currently deep inside another
> > > > debugging issue and there is one more log waiting in queue, so I
> > > > will probably not be able to look at it immediately (other than a
> > > > quick
> > glimpse).
> > > >
> > > > Rainer
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > > >> bounces at lists.adiscon.com] On Behalf Of Piavlo
> > > >> Sent: Tuesday, June 14, 2011 10:59 AM
> > > >> To: rsyslog-users
> > > >> Subject: [rsyslog] failover and actionqueue combination problem
> > > >>
> > > >>  Hi,
> > > >>
> > > >> I'm trying to implement the following scenario Messages should be
> > > >> sent to remote server jobs1a If message can't deliver message to
> > > >> jobs1a rsyslog should send it to jobs2a, if delivering to jobs2a
> > > >> fails too then rsyslog
> > > >>
> > > > should
> > > >
> > > >> again try to sending to jobs1a again this time using disk queue -
> > > >> so that
> > > >>
> > > > logs
> > > >
> > > >> could be buffered and sent to jobs1a later then it recovers.
> > > >>
> > > >> so I've made the following config
> > > >> --------------------------------------------
> > > >> local3.info @@jobs1a.internal:10514
> > > >>
> > > >> $ActionExecOnlyWhenPreviousIsSuspended on
> > > >>
> > > >> & @@jobs2a.internal:10514
> > > >>
> > > >> $ActionQueueType LinkedList
> > > >> $ActionQueueFileName forward_access_log
> > > $ActionQueueSaveOnShutdown on
> > > >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> > > >>
> > > >> $ActionExecOnlyWhenPreviousIsSuspended off
> > > >> --------------------------------------------
> > > >>
> > > >> The problem is that it does not seem to work messages which are
> > > >> logged while both jobs1a & jobs2a there down are lost.
> > > >> Then jobs1a recovers it receives only new logs. I've also tried
> > > >> simplifying
> > > >>
> > > > the
> > > >
> > > >> failure scenario by first stopping rsyslog on jobs2a and then
> > > >> jobs1a and
> > > >>
> > > > the
> > > >
> > > >> starting it on jobs2a with same results.
> > > >>
> > > >> Not that the following works ok - and messages which were queued
> > > >> while jobs1a was down are delivered to it then it recovers.
> > > >> -----------------------------------------
> > > >> $ActionQueueType LinkedList
> > > >> $ActionQueueFileName forward_access_log
> > > $ActionQueueSaveOnShutdown on
> > > >> $ActionResumeRetryCount -1 & @@jobs1a.internal:10514
> > > >> -----------------------------------------
> > > >>
> > > >> Any idea what I'm doing wrong?
> > > >> Is it possible to achieve such scenario with rsyslog ?
> > > >>
> > > >> Thanks
> > > >> Alex
> > > >> _______________________________________________
> > > >> rsyslog mailing list
> > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > >> http://www.rsyslog.com
> > > >>
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From alorbach at ro1.adiscon.com  Wed Jun 15 13:24:14 2011
From: alorbach at ro1.adiscon.com (Andre Lorbach)
Date: Wed, 15 Jun 2011 13:24:14 +0200
Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
	doesn'tconform to table type monitorware in loganalyzer-3.2.1 ?
In-Reply-To: <BANLkTinvzGpwPJ4+87bGeQhP40_Y5zw6iQ@mail.gmail.com>
References: <BANLkTi=rAPg5dGvXf-8LwDp8pdwUiD6TCA@mail.gmail.com><E152A03EC5F8DF4D8567006AF38A282D1ED5E3@vmexch1.intern.adiscon.com><BANLkTi=O8Otv9PtQL7O++ryaSM+2iqbaRw@mail.gmail.com><E152A03EC5F8DF4D8567006AF38A282D1ED5E5@vmexch1.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7280DFE@GRFEXC.intern.adiscon.com>
	<BANLkTinvzGpwPJ4+87bGeQhP40_Y5zw6iQ@mail.gmail.com>
Message-ID: <E152A03EC5F8DF4D8567006AF38A282D1ED5F5@vmexch1.intern.adiscon.com>

Hi,

the checksum field is also automatically added by the HandleMissingField()
function in the  logstream source itself if not available. 
This of course requires ALTER TABLE rights for the used mysql user. 

The fields purpose is simple, a 32bit checksum generated from the message
itself is stored in this field when a report is being generated. The
intention of this field was to speed up future reports as the checksum can be
used to consolidate messages. As it is only used for the reports, it was not
added to the main database schema yet. 

Best regards,
Andre Lorbach

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> Sent: Donnerstag, 9. Juni 2011 18:35
> To: rsyslog-users
> Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
doesn'tconform
> to table type monitorware in loganalyzer-3.2.1 ?
> 
> There's another missing field in CreateDB.sql, which would result in empty
> "Syslogmessages consolidated per Host" section in a Syslog Summary Report.
> The field checksum(mapped from misc_checksum) is required by
> CreateSQLStatement in classes/logstreamdb.class.php:
> 
> 1320                 if ( $includeFields && $this->_arrProperties != null )
> 1321                 {
> 1322                         // Loop through all requested fields
> 1323                         foreach ( $this->_arrProperties as $myproperty
)
> 1324                         {
> 1325                                 // SYSLOG_UID already added!
> 1326                                 if ( $myproperty != SYSLOG_UID &&
> isset($dbmapping[$szTableType]['DBMAPPINGS'][$myproperty]) )
> 1327                                 {
> 1328                                         // Append field!
> 1329                                         $sqlString .= ", " .
> $dbmapping[$szTableType]['DBMAPPINGS'][$myproperty];
> 1330                                 }
> 1331                         }
> 1332                 }
> 
> Notice the _arrProperties member in lin 1323, it should have been assigned
> by
>   84         public function Open($arrProperties)
>   97                 $this->_arrProperties = $arrProperties;
> 
> and passed into LogStreamDB by (in report.syslog.syslogsummary.class.php)
> 118         public function startDataProcessing()
> 129                 $res = $this->_streamObj->Open(
> $this->_arrProperties, true );
> 
> The startDataProcessing() method would be called by RunReport() in
> cmdreportgen.php to generate reports.
> 
> 
> The definition is in
./classes/reports/report.syslog.syslogsummary.class.php
>  69                 $this->_arrProperties[] = SYSLOG_UID;
>  70                 $this->_arrProperties[] = SYSLOG_DATE;
>  71                 $this->_arrProperties[] = SYSLOG_HOST;
>  72                 $this->_arrProperties[] = SYSLOG_MESSAGETYPE;
>  73                 $this->_arrProperties[] = SYSLOG_FACILITY;
>  74                 $this->_arrProperties[] = SYSLOG_SEVERITY;
>  75                 $this->_arrProperties[] = SYSLOG_SYSLOGTAG;
>  76                 $this->_arrProperties[] = SYSLOG_PROCESSID;
>  77                 $this->_arrProperties[] = SYSLOG_MESSAGE;
>  78                 $this->_arrProperties[] = MISC_CHECKSUM;
> 
> Notice line 78 introduces the dependency on a checksum field in data
> sources of monitorware table type. Otherwise, the CreateSQLStatement
> would generate a sql like
> 
> SELECT id, devicereportedtime, fromhost, infounitid, facility, priority,
> syslogtag, processid, message, checksum FROM SystemEvents WHERE
> devicereportedtime > '2011-06-08 23:52:48' AND infounitid IN (1)  ORDER BY
> id LIMIT 100
> 
> The ReadNextRecordsFromDB to fetch data would fail silently.
> 
> 
> I am not sure what purpose the summary field is meant to serve.
> Leaving it NULL works fine for the summary report, with the checksum
> varchar(100) field added.
> 
> 
> Thanks,
> Kaiwang
> 
> 2011/6/8 Rainer Gerhards <rgerhards at hq.adiscon.com>:
> > This field is not populated and I am bit hesitant to change the
> > default template. That will probably break a number of running
> > configurations. Also, I can not reliably populate that field due to
> > the variety of different ways a process ID is expressed...
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Andre Lorbach
> >> Sent: Wednesday, June 08, 2011 11:43 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
> > doesn'tconform
> >> to table type monitorware in loganalyzer-3.2.1 ?
> >>
> >> The ProcessID field is more or less an optional, so having a NULL
> >> value in
> > it is
> >> fine.
> >> Populating it with the ProcessID field will be useful for filtering
> >> within LogAnalyzer.
> >>
> >> However as far as I know, the default template does not include the
> >> ProcessID field, but it can be easily extended.
> >>
> >> Best regards,
> >> Andre Lorbach
> >>
> >> > -----Original Message-----
> >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> > bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> > Sent: Dienstag, 7. Juni 2011 19:34
> >> > To: rsyslog-users
> >> > Subject: Re: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
> >> > doesn't conform to table type monitorware in loganalyzer-3.2.1 ?
> >> >
> >> > Will the default output template of rsyslog fill the new procid field?
> >> > Looks like leaving it NULL should work as well.
> >> >
> >> > Thanks,
> >> > Kaiwang
> >> >
> >> > 2011/6/7 Andre Lorbach <alorbach at ro1.adiscon.com>:
> >> > > Hi,
> >> > >
> >> > > the ProcessID field was added for LogAnalyzer. It wasn't in
> >> > > MonitorWare either.
> >> > > But LogAnalyzer will automatically add missing fields into the
> >> > > logstream databases, if the database user has sufficient rights
> >> > > to the table. So granting the database user sufficient rights
> >> > > would solve the
> >> > problem for now.
> >> > >
> >> > >
> >> > > Apparently adding this field into the default database schema of
> >> > > MonitorWare and RSyslog was lost in communication somewhere.
> >> > >
> >> > > Best regards,
> >> > > Andre Lorbach
> >> > >
> >> > >> -----Original Message-----
> >> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> > >> bounces at lists.adiscon.com] On Behalf Of Kaiwang Chen
> >> > >> Sent: Dienstag, 7. Juni 2011 16:04
> >> > >> To: rsyslog-users
> >> > >> Subject: [rsyslog] createDB.sql shipped with rsyslog-5.8.1
> >> > >> doesn't conform
> >> > > to
> >> > >> table type monitorware in loganalyzer-3.2.1 ?
> >> > >>
> >> > >> Hello,
> >> > >>
> >> > >> In Step 7 of installation process, "Create the first source for
> >> > >> syslog messages", selecting Table type: MonitorWare (the other
> >> > >> is
> >> > >> SyslogNG) would load $dbmapping['mnoitorware'] in
> >> > >> include/constants_logstream.php, resulting in SQL like this:
> >> > >>
> >> > >> SELECT id, devicereportedtime, facility, priority, fromhost,
> >> > >> syslogtag, processid, infounitid, message FROM SystemEvents
> >> > >> ORDER BY id DESC LIMIT
> >> > >> 100
> >> > >>
> >> > >> In the case of syslog, the fields are mapped from
> >> > >> ./include/functions_config.php:
> >> > >>
> >> > >> ?501 ? ? ? ? $CFG['Views']['SYSLOG']= array(
> >> > >> ?502
> >> > >> ? ? ? 'ID' => ? ? ? ? ? ? ? ? "SYSLOG",
> >> > >> ?503
> >> > >> ? ? ? 'DisplayName' =>"Syslog Fields",
> >> > >> ?504
> >> > >> ? ? ? 'Columns' => ? ?array ( SYSLOG_DATE, SYSLOG_FACILITY, S
> >> > >> ? ? ?YSLOG_SEVERITY, SYSLOG_HOST, SYSLOG_SYSLOGTAG,
> >> > SYSLOG_PROCESSID,
> >> > >> SYSLOG_MESSAGETYPE, SYSLOG_MESSAGE ),
> >> > >> ?505
> >> > >> ? ? ? 'userid' => ? ? ? ? ? ? null,
> >> > >> ?506
> >> > >> ? ? ? 'groupid' => ? ?null,
> >> > >> ?507
> >> > >> );
> >> > >>
> >> > >> Columns array:
> >> > >> ? ? ? ? ? ? [0] => timereported
> >> > >> ? ? ? ? ? ? [1] => syslogfacility
> >> > >> ? ? ? ? ? ? [2] => syslogseverity
> >> > >> ? ? ? ? ? ? [3] => FROMHOST
> >> > >> ? ? ? ? ? ? [4] => syslogtag
> >> > >> ? ? ? ? ? ? [5] => procid
> >> > >> ? ? ? ? ? ? [6] => IUT
> >> > >> ? ? ? ? ? ? [7] => msg
> >> > >>
> >> > >>
> >> > >> Finally, I got a error prompt like this:
> >> > >>
> >> > >> No syslog records found - Error Details:
> >> > >>
> >> > >> No syslog records found
> >> > >>
> >> > >>
> >> > >> The CreateDB.sql shipped with rsyslog-5.8.1 contains(Notice the
> >> > >> processid filed is missing)
> >> > >>
> >> > >> CREATE TABLE SystemEvents
> >> > >> (
> >> > >> ? ? ? ? ID int unsigned not null auto_increment primary key,
> >> > >> ? ? ? ? CustomerID bigint,
> >> > >> ? ? ? ? ReceivedAt datetime NULL,
> >> > >> ? ? ? ? DeviceReportedTime datetime NULL,
> >> > >> ? ? ? ? Facility smallint NULL,
> >> > >> ? ? ? ? Priority smallint NULL,
> >> > >> ? ? ? ? FromHost varchar(60) NULL,
> >> > >> ? ? ? ? Message text,
> >> > >> ? ? ? ? NTSeverity int NULL,
> >> > >> ? ? ? ? Importance int NULL,
> >> > >> ? ? ? ? EventSource varchar(60),
> >> > >> ? ? ? ? EventUser varchar(60) NULL,
> >> > >> ? ? ? ? EventCategory int NULL,
> >> > >> ? ? ? ? EventID int NULL,
> >> > >> ? ? ? ? EventBinaryData text NULL,
> >> > >> ? ? ? ? MaxAvailable int NULL,
> >> > >> ? ? ? ? CurrUsage int NULL,
> >> > >> ? ? ? ? MinUsage int NULL,
> >> > >> ? ? ? ? MaxUsage int NULL,
> >> > >> ? ? ? ? InfoUnitID int NULL ,
> >> > >> ? ? ? ? SysLogTag varchar(60),
> >> > >> ? ? ? ? EventLogType varchar(60),
> >> > >> ? ? ? ? GenericFileName VarChar(60),
> >> > >> ? ? ? ? SystemID int NULL
> >> > >> );
> >> > >>
> >> > >>
> >> > >> So, what should I do? I heard of monitorware schema, and assumed
> >> > >> it to be what shipped with rsyslog.
> >> > >>
> >> > >>
> >> > >> Thanks,
> >> > >> Kaiwang
> >> > >> _______________________________________________
> >> > >> rsyslog mailing list
> >> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > >> http://www.rsyslog.com
> >> > > _______________________________________________
> >> > > rsyslog mailing list
> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > > http://www.rsyslog.com
> >> > >
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From roland.kamke at auconet.com  Wed Jun 15 13:28:03 2011
From: roland.kamke at auconet.com (Roland Kamke)
Date: Wed, 15 Jun 2011 13:28:03 +0200
Subject: [rsyslog] Special chars in database settings
Message-ID: <69DFC544026A844790C013C0FB16B9E0BC7601@eta.firma.lan>

Hello Rainer,

> I am sorry, but that's currently not supported. You need to modify the
> source
> :(
> 
> Rainer

Thank you for your fast reply.
At least I learnt not to use hash or comma chars etc. in the appropriate
configuration parts. That's good to know. We'll find a solution with our
customer.

Modifying your C code... Honestly, I am afraid I might break your code
and make things worse. I do not dare to do so.

Nevertheless, thank you for rsyslog in general and this piece of
information in particular.

Regards,
Roland


From rory at ooma.com  Wed Jun 15 16:09:35 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 15 Jun 2011 07:09:35 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
Message-ID: <4DF8BD1F.2050306@ooma.com>

So, how do I make such a log? I turned on the debug option and generated 
messages. What else do I need to do?

On 6/14/11 11:42 PM, Rainer Gerhards wrote:
> The "upness" is determined by connecting. But I really can't say much more
> until I get a log that was recorded while the problem happened. The log you
> provided contains just the startup but no message flow at all.
>
> Rainer
>
>> -----Original Message-----
>> From: Rory Toma [mailto:rory at ooma.com]
>> Sent: Wednesday, June 15, 2011 12:49 AM
>> To: rsyslog-users
>> Cc: Rainer Gerhards
>> Subject: Re: [rsyslog] Question on host failover
>>
>> The rules should have been triggered. I blocked (via firewall) the
>> primary syslog server, and tcpdumps shows that it just keeps sending to
>> it, even though it could not connect. The fact that the rules are not
>> being triggered means that either somehow the config is wrong, or there
>> is a bug. How does rsyslog determine the "upness" of the server it
>> tries
>> to connect to?
>>
>> On 6/14/11 6:05 AM, Rainer Gerhards wrote:
>>> Sorry, I have been swamped with another issue, which took up all my
>> time. I
>>> have now reviewed the log but it looks like it does not contain any
>> instance
>>> where the rules are actually triggered. Could you create such a one?
>>>
>>> Thanks,
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>>>> Sent: Wednesday, June 08, 2011 10:17 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Question on host failover
>>>>
>>>> On 6/2/11 2:19 PM, Rory Toma wrote:
>>>>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
>>>>>> Mmhhh.. can you post a complete debug log (maybe via a website
>> like
>>>>>> filebin)?
>>>>>>
>>>>>> Thx,
>>>>>> Rainer
>>>>>>
>>>>> Let me know when you've downloaded this file and I'll reset the
>> ACL, thx.
>>>>> http://www.colinburns.com/downloads/darwin0.bz2
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com
>>>> Any update on this?
>>>>
>>>> thx
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Wed Jun 15 17:16:44 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 15 Jun 2011 17:16:44 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DF8BD1F.2050306@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Wednesday, June 15, 2011 4:10 PM
> To: Rainer Gerhards
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> So, how do I make such a log?

http://www.rsyslog.com/doc/troubleshoot.html

> I turned on the debug option and generated
> messages. What else do I need to do?

Just make sure that failover kicks in. But the good news is that I am
probably able to reproduce it myself. I guess this is the bug:

http://bugzilla.adiscon.com/show_bug.cgi?id=270

So I do not need anything at this moment. I suggest to subscribe yourself to
the bug tracker.

Rainer
> 
> On 6/14/11 11:42 PM, Rainer Gerhards wrote:
> > The "upness" is determined by connecting. But I really can't say much
> > more until I get a log that was recorded while the problem happened.
> > The log you provided contains just the startup but no message flow at
all.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: Rory Toma [mailto:rory at ooma.com]
> >> Sent: Wednesday, June 15, 2011 12:49 AM
> >> To: rsyslog-users
> >> Cc: Rainer Gerhards
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> The rules should have been triggered. I blocked (via firewall) the
> >> primary syslog server, and tcpdumps shows that it just keeps sending
> >> to it, even though it could not connect. The fact that the rules are
> >> not being triggered means that either somehow the config is wrong, or
> >> there is a bug. How does rsyslog determine the "upness" of the server
> >> it tries to connect to?
> >>
> >> On 6/14/11 6:05 AM, Rainer Gerhards wrote:
> >>> Sorry, I have been swamped with another issue, which took up all my
> >> time. I
> >>> have now reviewed the log but it looks like it does not contain any
> >> instance
> >>> where the rules are actually triggered. Could you create such a one?
> >>>
> >>> Thanks,
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >>>> Sent: Wednesday, June 08, 2011 10:17 PM
> >>>> To: rsyslog-users
> >>>> Subject: Re: [rsyslog] Question on host failover
> >>>>
> >>>> On 6/2/11 2:19 PM, Rory Toma wrote:
> >>>>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
> >>>>>> Mmhhh.. can you post a complete debug log (maybe via a website
> >> like
> >>>>>> filebin)?
> >>>>>>
> >>>>>> Thx,
> >>>>>> Rainer
> >>>>>>
> >>>>> Let me know when you've downloaded this file and I'll reset the
> >> ACL, thx.
> >>>>> http://www.colinburns.com/downloads/darwin0.bz2
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com
> >>>> Any update on this?
> >>>>
> >>>> thx
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com


From rory at ooma.com  Wed Jun 15 22:41:43 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 15 Jun 2011 13:41:43 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
Message-ID: <4DF91907.7010404@ooma.com>

On 6/15/11 8:16 AM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: Rory Toma [mailto:rory at ooma.com]
>> Sent: Wednesday, June 15, 2011 4:10 PM
>> To: Rainer Gerhards
>> Cc: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> So, how do I make such a log?
> http://www.rsyslog.com/doc/troubleshoot.html
>
>> I turned on the debug option and generated
>> messages. What else do I need to do?
> Just make sure that failover kicks in. But the good news is that I am
> probably able to reproduce it myself. I guess this is the bug:

Lol, I guess that goes back to the original issue... Failover doesn't 
kick in. I set up the config, which should failover, then block the port 
on the firewall, and it doesn't fail over, just keeps trying the 
original over and over.

I'll check out the bug. Thx.
> http://bugzilla.adiscon.com/show_bug.cgi?id=270
>
> So I do not need anything at this moment. I suggest to subscribe yourself to
> the bug tracker.


From rory at ooma.com  Wed Jun 15 22:58:20 2011
From: rory at ooma.com (Rory Toma)
Date: Wed, 15 Jun 2011 13:58:20 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
Message-ID: <4DF91CEC.9040107@ooma.com>

So, if I check out the bug, the syntax that you have marked as "works" 
is what does *not* work for me.

On 6/15/11 8:16 AM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: Rory Toma [mailto:rory at ooma.com]
>> Sent: Wednesday, June 15, 2011 4:10 PM
>> To: Rainer Gerhards
>> Cc: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> So, how do I make such a log?
> http://www.rsyslog.com/doc/troubleshoot.html
>
>> I turned on the debug option and generated
>> messages. What else do I need to do?
> Just make sure that failover kicks in. But the good news is that I am
> probably able to reproduce it myself. I guess this is the bug:
>
> http://bugzilla.adiscon.com/show_bug.cgi?id=270
>
> So I do not need anything at this moment. I suggest to subscribe yourself to
> the bug tracker.
>
> Rainer
>> On 6/14/11 11:42 PM, Rainer Gerhards wrote:
>>> The "upness" is determined by connecting. But I really can't say much
>>> more until I get a log that was recorded while the problem happened.
>>> The log you provided contains just the startup but no message flow at
> all.
>>> Rainer
>>>
>>>> -----Original Message-----
>>>> From: Rory Toma [mailto:rory at ooma.com]
>>>> Sent: Wednesday, June 15, 2011 12:49 AM
>>>> To: rsyslog-users
>>>> Cc: Rainer Gerhards
>>>> Subject: Re: [rsyslog] Question on host failover
>>>>
>>>> The rules should have been triggered. I blocked (via firewall) the
>>>> primary syslog server, and tcpdumps shows that it just keeps sending
>>>> to it, even though it could not connect. The fact that the rules are
>>>> not being triggered means that either somehow the config is wrong, or
>>>> there is a bug. How does rsyslog determine the "upness" of the server
>>>> it tries to connect to?
>>>>
>>>> On 6/14/11 6:05 AM, Rainer Gerhards wrote:
>>>>> Sorry, I have been swamped with another issue, which took up all my
>>>> time. I
>>>>> have now reviewed the log but it looks like it does not contain any
>>>> instance
>>>>> where the rules are actually triggered. Could you create such a one?
>>>>>
>>>>> Thanks,
>>>>> Rainer
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
>>>>>> Sent: Wednesday, June 08, 2011 10:17 PM
>>>>>> To: rsyslog-users
>>>>>> Subject: Re: [rsyslog] Question on host failover
>>>>>>
>>>>>> On 6/2/11 2:19 PM, Rory Toma wrote:
>>>>>>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
>>>>>>>> Mmhhh.. can you post a complete debug log (maybe via a website
>>>> like
>>>>>>>> filebin)?
>>>>>>>>
>>>>>>>> Thx,
>>>>>>>> Rainer
>>>>>>>>
>>>>>>> Let me know when you've downloaded this file and I'll reset the
>>>> ACL, thx.
>>>>>>> http://www.colinburns.com/downloads/darwin0.bz2
>>>>>>> _______________________________________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>> http://www.rsyslog.com
>>>>>> Any update on this?
>>>>>>
>>>>>> thx
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com


From rgerhards at hq.adiscon.com  Thu Jun 16 08:28:06 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 16 Jun 2011 08:28:06 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DF91907.7010404@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91907.7010404@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E94@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Wednesday, June 15, 2011 10:42 PM
> To: Rainer Gerhards
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> On 6/15/11 8:16 AM, Rainer Gerhards wrote:
> >> -----Original Message-----
> >> From: Rory Toma [mailto:rory at ooma.com]
> >> Sent: Wednesday, June 15, 2011 4:10 PM
> >> To: Rainer Gerhards
> >> Cc: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> So, how do I make such a log?
> > http://www.rsyslog.com/doc/troubleshoot.html
> >
> >> I turned on the debug option and generated
> >> messages. What else do I need to do?
> > Just make sure that failover kicks in. But the good news is that I am
> > probably able to reproduce it myself. I guess this is the bug:
> 
> Lol, I guess that goes back to the original issue... Failover doesn't
> kick in.

OK, I should have written "should kick in". But the bottom line is that you
can not diagnose something if you do not have any information on the
situation that you shall diagnose...

 I set up the config, which should failover, then block the
> port
> on the firewall, and it doesn't fail over, just keeps trying the
> original over and over.
> 
> I'll check out the bug. Thx.
> > http://bugzilla.adiscon.com/show_bug.cgi?id=270
> >
> > So I do not need anything at this moment. I suggest to subscribe
> yourself to
> > the bug tracker.


From rgerhards at hq.adiscon.com  Thu Jun 16 08:28:47 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 16 Jun 2011 08:28:47 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DF91CEC.9040107@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91CEC.9040107@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Wednesday, June 15, 2011 10:58 PM
> To: Rainer Gerhards
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> So, if I check out the bug, the syntax that you have marked as "works"
> is what does *not* work for me.

Wait until that bug is fixed and let's see if what you see is a side-effect
or not. If it doesn't sole the issue, we can always look at the next one...

Rainer
> 
> On 6/15/11 8:16 AM, Rainer Gerhards wrote:
> >> -----Original Message-----
> >> From: Rory Toma [mailto:rory at ooma.com]
> >> Sent: Wednesday, June 15, 2011 4:10 PM
> >> To: Rainer Gerhards
> >> Cc: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> So, how do I make such a log?
> > http://www.rsyslog.com/doc/troubleshoot.html
> >
> >> I turned on the debug option and generated
> >> messages. What else do I need to do?
> > Just make sure that failover kicks in. But the good news is that I am
> > probably able to reproduce it myself. I guess this is the bug:
> >
> > http://bugzilla.adiscon.com/show_bug.cgi?id=270
> >
> > So I do not need anything at this moment. I suggest to subscribe
> yourself to
> > the bug tracker.
> >
> > Rainer
> >> On 6/14/11 11:42 PM, Rainer Gerhards wrote:
> >>> The "upness" is determined by connecting. But I really can't say
> much
> >>> more until I get a log that was recorded while the problem
> happened.
> >>> The log you provided contains just the startup but no message flow
> at
> > all.
> >>> Rainer
> >>>
> >>>> -----Original Message-----
> >>>> From: Rory Toma [mailto:rory at ooma.com]
> >>>> Sent: Wednesday, June 15, 2011 12:49 AM
> >>>> To: rsyslog-users
> >>>> Cc: Rainer Gerhards
> >>>> Subject: Re: [rsyslog] Question on host failover
> >>>>
> >>>> The rules should have been triggered. I blocked (via firewall) the
> >>>> primary syslog server, and tcpdumps shows that it just keeps
> sending
> >>>> to it, even though it could not connect. The fact that the rules
> are
> >>>> not being triggered means that either somehow the config is wrong,
> or
> >>>> there is a bug. How does rsyslog determine the "upness" of the
> server
> >>>> it tries to connect to?
> >>>>
> >>>> On 6/14/11 6:05 AM, Rainer Gerhards wrote:
> >>>>> Sorry, I have been swamped with another issue, which took up all
> my
> >>>> time. I
> >>>>> have now reviewed the log but it looks like it does not contain
> any
> >>>> instance
> >>>>> where the rules are actually triggered. Could you create such a
> one?
> >>>>>
> >>>>> Thanks,
> >>>>> Rainer
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >>>>>> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> >>>>>> Sent: Wednesday, June 08, 2011 10:17 PM
> >>>>>> To: rsyslog-users
> >>>>>> Subject: Re: [rsyslog] Question on host failover
> >>>>>>
> >>>>>> On 6/2/11 2:19 PM, Rory Toma wrote:
> >>>>>>> On 6/2/11 3:32 AM, Rainer Gerhards wrote:
> >>>>>>>> Mmhhh.. can you post a complete debug log (maybe via a website
> >>>> like
> >>>>>>>> filebin)?
> >>>>>>>>
> >>>>>>>> Thx,
> >>>>>>>> Rainer
> >>>>>>>>
> >>>>>>> Let me know when you've downloaded this file and I'll reset the
> >>>> ACL, thx.
> >>>>>>> http://www.colinburns.com/downloads/darwin0.bz2
> >>>>>>> _______________________________________________
> >>>>>>> rsyslog mailing list
> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>> http://www.rsyslog.com
> >>>>>> Any update on this?
> >>>>>>
> >>>>>> thx
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com
> >>>>> _______________________________________________
> >>>>> rsyslog mailing list
> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>> http://www.rsyslog.com


From rory at ooma.com  Fri Jun 17 00:50:34 2011
From: rory at ooma.com (Rory Toma)
Date: Thu, 16 Jun 2011 15:50:34 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91CEC.9040107@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>
Message-ID: <4DFA88BA.1050208@ooma.com>

On 6/15/11 11:28 PM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: Rory Toma [mailto:rory at ooma.com]
>> Sent: Wednesday, June 15, 2011 10:58 PM
>> To: Rainer Gerhards
>> Cc: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> So, if I check out the bug, the syntax that you have marked as "works"
>> is what does *not* work for me.
> Wait until that bug is fixed and let's see if what you see is a side-effect
> or not. If it doesn't sole the issue, we can always look at the next one...
>
> Rainer

I tried out the two patches that you posted on the bug. No change for me.

From rgerhards at hq.adiscon.com  Fri Jun 17 08:09:25 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 17 Jun 2011 08:09:25 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DFA88BA.1050208@ooma.com>
References: <4DD459C3.9020301@ooma.com><9B6E2A8877C38245BFB15CC491A11DA71DE198@GRFEXC.intern.adiscon.com>	<4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91CEC.9040107@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>
	<4DFA88BA.1050208@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EB2@GRFEXC.intern.adiscon.com>

Thanks for the feedback. So I now need a debug log. Please make sure that it
includes both the startup as well as a sample message flow where the problem
occurs.

Rainer

> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Friday, June 17, 2011 12:51 AM
> To: Rainer Gerhards
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> On 6/15/11 11:28 PM, Rainer Gerhards wrote:
> >> -----Original Message-----
> >> From: Rory Toma [mailto:rory at ooma.com]
> >> Sent: Wednesday, June 15, 2011 10:58 PM
> >> To: Rainer Gerhards
> >> Cc: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> So, if I check out the bug, the syntax that you have marked as
> "works"
> >> is what does *not* work for me.
> > Wait until that bug is fixed and let's see if what you see is a side-
> effect
> > or not. If it doesn't sole the issue, we can always look at the next
> one...
> >
> > Rainer
> 
> I tried out the two patches that you posted on the bug. No change for
> me.

From josu.lazkano at barcelonamedia.org  Fri Jun 17 11:34:10 2011
From: josu.lazkano at barcelonamedia.org (Josu Lazkano)
Date: Fri, 17 Jun 2011 11:34:10 +0200
Subject: [rsyslog] Send Apache2 logs
Message-ID: <A55193C141A0254B8D28E8E7C9D3952C26263768EB@FBMEC01.corp.barcelonamedia.org>

Hello list, this is my first post on the list. I am using rsyslog to send all logs to a syslog-ng server this way:

$ cat /etc/rsyslog.d/99-rsyslog.conf

auth.*,authpriv.*       @logserver
kern.warn               @logserver
kern.err                @logserver
mail.*                  @logserver

I have some Apache logs on different path and I want to send all of them:

/var/www/domain1/log/
/var/www/domain2/log/
/var/www/domain3/log/
...

How could  I send all logs to my logserver?

Thanks for all your help and best regards.

From marcin at mejor.pl  Fri Jun 17 13:27:03 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Fri, 17 Jun 2011 13:27:03 +0200
Subject: [rsyslog] Send Apache2 logs
In-Reply-To: <A55193C141A0254B8D28E8E7C9D3952C26263768EB@FBMEC01.corp.barcelonamedia.org>
References: <A55193C141A0254B8D28E8E7C9D3952C26263768EB@FBMEC01.corp.barcelonamedia.org>
Message-ID: <4DFB3A07.8010502@mejor.pl>

W dniu 17.06.2011 11:34, Josu Lazkano pisze:
> Hello list, this is my first post on the list. I am using rsyslog to send all logs to a syslog-ng server this way:
> 
> $ cat /etc/rsyslog.d/99-rsyslog.conf
> 
> auth.*,authpriv.*       @logserver
> kern.warn               @logserver
> kern.err                @logserver
> mail.*                  @logserver
> 
> I have some Apache logs on different path and I want to send all of them:
> 
> /var/www/domain1/log/
> /var/www/domain2/log/
> /var/www/domain3/log/
> ...
> 
> How could  I send all logs to my logserver?

Hello!
I'm using something like this:
$ModLoad imfile
[...]
$InputFileName /var/log/exim/exim_panic.log
$InputFileTag hoscik.exim_panic:
$InputFileStateFile hermes.exim_panic
$InputFileFacility mail
$InputFileSeverity error
$InputRunFileMonitor

Regards.

From rory at ooma.com  Sat Jun 18 00:20:31 2011
From: rory at ooma.com (Rory Toma)
Date: Fri, 17 Jun 2011 15:20:31 -0700
Subject: [rsyslog] Question on host failover
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280EB2@GRFEXC.intern.adiscon.com>
References: <4DD459C3.9020301@ooma.com><4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91CEC.9040107@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>
	<4DFA88BA.1050208@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280EB2@GRFEXC.intern.adiscon.com>
Message-ID: <4DFBD32F.2020800@ooma.com>

The one I gave you was set up to fail over.

I blocked port 110, so it should failover to 143. It's not. It's as if 
it never gets to the rule, or it isn't parsing it. I watch my tcpdump 
and it just keeps retrying on port 110. I gave you the debug log with 
this setup and it never gets to the failover rule, which, is probably 
the problem.

So, given that, what do I do?

On 6/16/11 11:09 PM, Rainer Gerhards wrote:
> Thanks for the feedback. So I now need a debug log. Please make sure that it
> includes both the startup as well as a sample message flow where the problem
> occurs.
>
> Rainer
>
>> -----Original Message-----
>> From: Rory Toma [mailto:rory at ooma.com]
>> Sent: Friday, June 17, 2011 12:51 AM
>> To: Rainer Gerhards
>> Cc: rsyslog-users
>> Subject: Re: [rsyslog] Question on host failover
>>
>> On 6/15/11 11:28 PM, Rainer Gerhards wrote:
>>>> -----Original Message-----
>>>> From: Rory Toma [mailto:rory at ooma.com]
>>>> Sent: Wednesday, June 15, 2011 10:58 PM
>>>> To: Rainer Gerhards
>>>> Cc: rsyslog-users
>>>> Subject: Re: [rsyslog] Question on host failover
>>>>
>>>> So, if I check out the bug, the syntax that you have marked as
>> "works"
>>>> is what does *not* work for me.
>>> Wait until that bug is fixed and let's see if what you see is a side-
>> effect
>>> or not. If it doesn't sole the issue, we can always look at the next
>> one...
>>> Rainer
>> I tried out the two patches that you posted on the bug. No change for
>> me.


From rgerhards at hq.adiscon.com  Sat Jun 18 10:54:53 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 18 Jun 2011 10:54:53 +0200
Subject: [rsyslog] Question on host failover
In-Reply-To: <4DFBD32F.2020800@ooma.com>
References: <4DD459C3.9020301@ooma.com><4DE57634.5030901@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280D95@GRFEXC.intern.adiscon.com><4DE5E0D4.5050409@ooma.com>	<4DE5E12B.3060600@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D98@GRFEXC.intern.adiscon.com>	<4DE5E22B.9060001@ooma.com><9B6E2A8877C38245BFB15CC491A11DA7280D99@GRFEXC.intern.adiscon.com>	<4DE6AB5F.9040805@ooma.com>	<9B6E2A8877C38245BFB15CC491A11DA7280DBA@GRFEXC.intern.adiscon.com><4DE7FE7C.3020604@ooma.com>	<4DEFD8B8.6040506@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E69@GRFEXC.intern.adiscon.com>
	<4DF7E578.7040303@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E76@GRFEXC.intern.adiscon.com>
	<4DF8BD1F.2050306@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E90@GRFEXC.intern.adiscon.com>
	<4DF91CEC.9040107@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280E95@GRFEXC.intern.adiscon.com>
	<4DFA88BA.1050208@ooma.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280EB2@GRFEXC.intern.adiscon.com>
	<4DFBD32F.2020800@ooma.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EBE@GRFEXC.intern.adiscon.com>


> -----Original Message-----
> From: Rory Toma [mailto:rory at ooma.com]
> Sent: Saturday, June 18, 2011 12:21 AM
> To: Rainer Gerhards
> Cc: rsyslog-users
> Subject: Re: [rsyslog] Question on host failover
> 
> The one I gave you was set up to fail over.
> 
> I blocked port 110, so it should failover to 143. It's not. It's as if
> it never gets to the rule, or it isn't parsing it. I watch my tcpdump
> and it just keeps retrying on port 110. I gave you the debug log with
> this setup and it never gets to the failover rule, which, is probably
> the problem.

Sorry, I did not see the message, so I did a re-check. You are right, there
was one, which I overlooked :(

The debug instrumentation for that case is bad in 5.8.1. I think I have
improved it with my recent patch series.
> 
> So, given that, what do I do?

Please apply the recent patches (or even better use the git v5-stable branch)
and submit a new debug log. Hopefully this will have sufficient debug
information. Besides that, I need the new version to see where the patch
series still fails.

Thanks,
Rainer
> 
> On 6/16/11 11:09 PM, Rainer Gerhards wrote:
> > Thanks for the feedback. So I now need a debug log. Please make sure
> that it
> > includes both the startup as well as a sample message flow where the
> problem
> > occurs.
> >
> > Rainer
> >
> >> -----Original Message-----
> >> From: Rory Toma [mailto:rory at ooma.com]
> >> Sent: Friday, June 17, 2011 12:51 AM
> >> To: Rainer Gerhards
> >> Cc: rsyslog-users
> >> Subject: Re: [rsyslog] Question on host failover
> >>
> >> On 6/15/11 11:28 PM, Rainer Gerhards wrote:
> >>>> -----Original Message-----
> >>>> From: Rory Toma [mailto:rory at ooma.com]
> >>>> Sent: Wednesday, June 15, 2011 10:58 PM
> >>>> To: Rainer Gerhards
> >>>> Cc: rsyslog-users
> >>>> Subject: Re: [rsyslog] Question on host failover
> >>>>
> >>>> So, if I check out the bug, the syntax that you have marked as
> >> "works"
> >>>> is what does *not* work for me.
> >>> Wait until that bug is fixed and let's see if what you see is a
> side-
> >> effect
> >>> or not. If it doesn't sole the issue, we can always look at the
> next
> >> one...
> >>> Rainer
> >> I tried out the two patches that you posted on the bug. No change
> for
> >> me.


From josu.lazkano at barcelonamedia.org  Mon Jun 20 10:18:20 2011
From: josu.lazkano at barcelonamedia.org (Josu Lazkano)
Date: Mon, 20 Jun 2011 10:18:20 +0200
Subject: [rsyslog] Send Apache2 logs
In-Reply-To: <4DFB3A07.8010502@mejor.pl>
References: <A55193C141A0254B8D28E8E7C9D3952C26263768EB@FBMEC01.corp.barcelonamedia.org>
	<4DFB3A07.8010502@mejor.pl>
Message-ID: <A55193C141A0254B8D28E8E7C9D3952C26263768FD@FBMEC01.corp.barcelonamedia.org>

-----Mensaje original-----
De: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] En nombre de Marcin Miroslaw
Enviado el: viernes, 17 de junio de 2011 13:27
Para: rsyslog at lists.adiscon.com
Asunto: Re: [rsyslog] Send Apache2 logs

W dniu 17.06.2011 11:34, Josu Lazkano pisze:
> Hello list, this is my first post on the list. I am using rsyslog to send all logs to a syslog-ng server this way:
> 
> $ cat /etc/rsyslog.d/99-rsyslog.conf
> 
> auth.*,authpriv.*       @logserver
> kern.warn               @logserver
> kern.err                @logserver
> mail.*                  @logserver
> 
> I have some Apache logs on different path and I want to send all of them:
> 
> /var/www/domain1/log/
> /var/www/domain2/log/
> /var/www/domain3/log/
> ...
> 
> How could  I send all logs to my logserver?

Hello!
I'm using something like this:
$ModLoad imfile
[...]
$InputFileName /var/log/exim/exim_panic.log
$InputFileTag hoscik.exim_panic:
$InputFileStateFile hermes.exim_panic
$InputFileFacility mail
$InputFileSeverity error
$InputRunFileMonitor

Regards.

Thanks for the reply, this is my complete configuration:

# cat /etc/rsyslog.conf

$ModLoad imuxsock
$ModLoad imklog
$KLogPath /proc/kmsg
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$IncludeConfig /etc/rsyslog.d/*.conf

How can I configure to send the apache access logs?

This is the Apache site config:

CustomLog /var/www/domain1/log/access.log combined

Thanks for all your help.

Best regards.

From tbergfeld at hq.adiscon.com  Tue Jun 21 14:52:08 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Tue, 21 Jun 2011 14:52:08 +0200
Subject: [rsyslog] rsyslog 5.8.2 (v5-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EE1@GRFEXC.intern.adiscon.com>

This is a maintenance release containing only stability updates.
Note that the mutex-related bug can have quite fatal consequences. So it is
highly recommended to upgrade to this version, even if you did not yet
experience any problems.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-8-2-v5-stable/

Download:

http://www.rsyslog.com/rsyslog-5-8-2-v5-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From dfaulkner at pobox.com  Tue Jun 21 19:01:21 2011
From: dfaulkner at pobox.com (Don Faulkner)
Date: Tue, 21 Jun 2011 12:01:21 -0500
Subject: [rsyslog] Dynamic filenames built from filter regex submatch?
Message-ID: <BANLkTikpbkt+AoUWofBdWe-jRLpjSLpS+Q@mail.gmail.com>

Hi all, I'm trying to do something, but I'm not sure if it's possible.

I'm trying to build a template to specify a dynamic filename, where part of
the filename is built from a matching filter condition.

Here's a real-world (or almost) example:

multiple DHCP servers, sending logs on disparate facilities. I'd like to
have them all in a single place like dhcp.log or dhcp/%HOSTNAME%.log

Sample line:
Jun 21 00:00:00 192.168.1.1 dhcpd: added reverse map from
5.5.168.192.in-addr.arpa. to somename.ddns.example.com

Now, in situations like this, I can just use sylogtag like so:

%template
SyslogTag,"/var/log/%YEAR%/%MONTH%/%DAY%/%SYSLOGTAG%/%HOSTNAME%.log
:syslogtag, contains, "dhcpd" -?SyslogTag;TraditionalFormat

and continue with additional syslogtag filters for each service. But, what
if my condition can't rely upon the syslogtag? If I need to do a generic
match like this:

:msg, eregex, "this is (what) I'm looking for"
-?DestinationTemplate,TraditionalFormat

I'd like to see the output here:
/var/log/2011/06/21/what/%HOSTNAME%.log

I can do this:
$template
MatchWhat,"/var/log/%YEAR%/%MONTH%/%DAY%/%msg:R,ERE,0,FIELD:what--end%/%HOSTNAME%.log

But I have to build a separate template for each thing I'm matching. Is
there a way to build a template that takes its match from the filter
expression? Something like the following:

$template Example,"/var/log/%YEAR%/%MONTH%/%DAY%/%1%/%HOSTNAME%.log
:msg, eregex, "this is (what) I'm looking for" -?Example,TraditionalFormat

Obviously, the %1% isn't legal. It's just an example of what I want to do.
Is carrying a submatch from a regex filter into the file template possible?

-- 
Don Faulkner, KB5WPM
"All that is gold does not glitter. Not all those who wander are lost."

From mike.forbes at koordinates.com  Tue Jun 21 23:56:06 2011
From: mike.forbes at koordinates.com (Mike Forbes)
Date: Wed, 22 Jun 2011 09:56:06 +1200
Subject: [rsyslog] TCP receiving and TLS fowarding on the same host.
Message-ID: <BANLkTik1eYq5549ROL7eHGqXY8GErGgwEw@mail.gmail.com>

Hi there,

I need to set up some remote log forwarding - a bunch of dev machines
behind a firewall forward logs to a logserver, which in turn forwards
the logs it gets to an external log server.

I've successfully got log forwarding working from the dev hosts to the
first log server using plain TCP, but i'd like to use TLS to get the
logs from the first to the external log server.

When I try with my current configs I get a segfault! :(

Debug files are available if they'd help, but they are huge.

config files:
tcp listener on the first logserver

#Load tcp module
$modload imtcp
# and run it, listening on port 10514
$InputTCPServerRun 10514

tls sending config:

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog.d/client-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/client-key.pem

$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.domainname.com
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
*.* @@external.logserver.domainname.com:10514 # send (all) messages


So far I've been able to get these working independently, but not together.

rsyslog version:
rsyslog 4.2.0-2ubuntu8.1 (lucid)

$ rsyslogd -v
rsyslogd 4.2.0, compiled with:
   FEATURE_REGEXP:            Yes
   FEATURE_LARGEFILE:         Yes
   FEATURE_NETZIP (message compression):   Yes
   GSSAPI Kerberos 5 support:      Yes
   FEATURE_DEBUG (debug build, slow code):   No
   Atomic operations supported:      Yes
   Runtime Instrumentation (slow code):   No

(I've posted this to the forums, but haven't had much in the way of response)

Any suggestions or help is appreciated!


-- 
// Mike
GPG: BFC7 3F32 2CCF D91F 53E1 ?DF88 1578 B2E4 1399 6844

From tbergfeld at hq.adiscon.com  Wed Jun 22 10:46:34 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Wed, 22 Jun 2011 10:46:34 +0200
Subject: [rsyslog] rsyslog 6.1.10 (v6-beta) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EEA@GRFEXC.intern.adiscon.com>

This is a stability update that imports the recent set of patches developed
for version 5 and above. Note that users are strongly advised to upgrade to
this version due to a potentially fatal failure caused by one bug.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-1-10-beta/

Download:

http://www.rsyslog.com/rsyslog-6-1-10-beta/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From tbergfeld at hq.adiscon.com  Fri Jun 24 09:08:13 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Fri, 24 Jun 2011 09:08:13 +0200
Subject: [rsyslog] rsyslog 4.6.6 (v4-stable) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EED@GRFEXC.intern.adiscon.com>

We have just released rsyslog 4.6.6, the new v4-stable.

This is a maintenance release for the current v4-stable branch. It contains
some important bug fixes. It is highly recommended to upgrade to this
version.

Please see the ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-4-6-6-v4-stable/

Download:

http://www.rsyslog.com/rsyslog-4-6-6-v4-stable/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From maseda at stanford.edu  Sat Jun 25 00:13:18 2011
From: maseda at stanford.edu (Mike Seda)
Date: Fri, 24 Jun 2011 15:13:18 -0700
Subject: [rsyslog] /var/log/boot.log
Message-ID: <4E050BFE.3060103@stanford.edu>

All,
It looks like RHEL 6 no longer uses initlog, which used to log to local7 
by default.

Instead, it looks like plymouth is used, which seems to be hardcoded to 
log to /var/log/boot.log.

I've grep'ed through most of the filesystem on a RHEL 6 box, and can't 
seem to find anything that is set to log to local7.

This seems to be a good reason to remove to following from 
/etc/rsyslog.conf on my RHEL 6 boxes:
local7.*                                                /var/log/boot.log

Doesn't this sound reasonable?

Mike

From rgerhards at hq.adiscon.com  Sat Jun 25 10:31:50 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sat, 25 Jun 2011 10:31:50 +0200
Subject: [rsyslog] /var/log/boot.log
In-Reply-To: <4E050BFE.3060103@stanford.edu>
References: <4E050BFE.3060103@stanford.edu>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280EFB@GRFEXC.intern.adiscon.com>

Hi Mike,

the rsyslog project provides the base daemon. The config files are out of the
project scope, every distro does it differently. So I suggest to post that to
some RHEL-specific place.

HTH
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Mike Seda
> Sent: Saturday, June 25, 2011 12:13 AM
> To: rsyslog-users
> Subject: [rsyslog] /var/log/boot.log
> 
> All,
> It looks like RHEL 6 no longer uses initlog, which used to log to
> local7
> by default.
> 
> Instead, it looks like plymouth is used, which seems to be hardcoded to
> log to /var/log/boot.log.
> 
> I've grep'ed through most of the filesystem on a RHEL 6 box, and can't
> seem to find anything that is set to log to local7.
> 
> This seems to be a good reason to remove to following from
> /etc/rsyslog.conf on my RHEL 6 boxes:
> local7.*
> /var/log/boot.log
> 
> Doesn't this sound reasonable?
> 
> Mike
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Mon Jun 27 10:23:27 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Mon, 27 Jun 2011 10:23:27 +0200
Subject: [rsyslog] Request for Comments: config format
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>

Hi all,

I am still thinking about a better config format. With the recent changes in
v6, the necessary plumbing is now (mostly) present. However there is one
question that really puzzles me:

Do you think that it is vital or at least useful to have the ability to use
old style and new style config format *together*?

Note that old-style (current) format will of course be supported in the
future. The question is if it would be useful to be able to mix both of them.
If a mix is not possible, the config file would need to be written in either
the old or the new format. That would most probably extend to included files.
So at startup one would need to select one format over another.

I personally tend to think that the capability to mix config formats would be
useful. HOWEVER, it seems to be quite complex and a lot of work. So I am not
sure if it is worth it. I am not even sure if it is desirable at all to have
this capability.

Feedback is appreciated.

Thanks,
Rainer

From taotetek at gmail.com  Mon Jun 27 12:32:25 2011
From: taotetek at gmail.com (Brian Knox)
Date: Mon, 27 Jun 2011 06:32:25 -0400
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com>

Rainer -

Off the top of my head I don't see it as a vital feature.  My initial
thought is that allowing a mixture of old and new config files is going to
be far from trivial and probably open the door up for a lot of confusion and
errors.

Brian

On Mon, Jun 27, 2011 at 4:23 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com>wrote:

> Hi all,
>
> I am still thinking about a better config format. With the recent changes
> in
> v6, the necessary plumbing is now (mostly) present. However there is one
> question that really puzzles me:
>
> Do you think that it is vital or at least useful to have the ability to use
> old style and new style config format *together*?
>
> Note that old-style (current) format will of course be supported in the
> future. The question is if it would be useful to be able to mix both of
> them.
> If a mix is not possible, the config file would need to be written in
> either
> the old or the new format. That would most probably extend to included
> files.
> So at startup one would need to select one format over another.
>
> I personally tend to think that the capability to mix config formats would
> be
> useful. HOWEVER, it seems to be quite complex and a lot of work. So I am
> not
> sure if it is worth it. I am not even sure if it is desirable at all to
> have
> this capability.
>
> Feedback is appreciated.
>
> Thanks,
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From mbiebl at gmail.com  Mon Jun 27 14:09:40 2011
From: mbiebl at gmail.com (Michael Biebl)
Date: Mon, 27 Jun 2011 14:09:40 +0200
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>
	<BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com>
Message-ID: <BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com>

2011/6/27 Brian Knox <taotetek at gmail.com>:
> Rainer -
>
> Off the top of my head I don't see it as a vital feature. ?My initial
> thought is that allowing a mixture of old and new config files is going to
> be far from trivial and probably open the door up for a lot of confusion and
> errors.

I kinda agree here. Mixing the two config formats smells like it opens
a can of worms.
The only reason why I think it could be useful, is the
$IncludeConfig support. In Debian we use

$IncludeConfig /etc/rsyslog.d/*.conf

which is used by packages and administrators to extend the rsyslog
configuration with custom rules.
So, while being able to mix the config file format in one file doesn't
sound like a vital feature, being able to include files of different
formats might be.

That said, having a tool, which allows to read the old format and
convert it to the new one, would be a very valuable feature imho.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

From epiphani at gmail.com  Mon Jun 27 15:33:45 2011
From: epiphani at gmail.com (Aaron Wiebe)
Date: Mon, 27 Jun 2011 09:33:45 -0400
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com>
Message-ID: <BANLkTikvVEwwfzke7fi=sjum+vqBBoE5_A@mail.gmail.com>

On Mon, Jun 27, 2011 at 4:23 AM, Rainer Gerhards
<rgerhards at hq.adiscon.com> wrote:
>
> Do you think that it is vital or at least useful to have the ability to use
> old style and new style config format *together*?
>

I generally agree with what's been said on the thread so far - I can't
see a reason why I'd want to use both together.  Either we switch to
the new config format or we don't.

-Aaron

From lanas at securenet.net  Tue Jun 28 02:59:18 2011
From: lanas at securenet.net (lanas)
Date: Mon, 27 Jun 2011 20:59:18 -0400
Subject: [rsyslog] Remote syslogging through a (broken) VPN
Message-ID: <20110627205918.0cc9ce80@mistral.stie>

Hello,

  I would like to know the behaviour of rsyslog in the following
scenario where remote syslogging through a VPN is used.  The VPN
mechanism (IPsec) logs to the system's syslog while establishing
a connection.  In this scenario, since the remote syslogging os
done via the VPN and the VPN is not yet up, the syslog
daemon (syslogd for the time being) receives the log messages
from IPsec.  And cannot send them to the remote log server.

  What happens is that the syslogd queue or cache becomes full
and at that moment IPsec waits on syslogd.  And syslogd cannot
send its accumulated messages.  A deadlock happens. And because
of that, the VPN cannot cannot be established.

  If syslogd in that scenario is replaced by rsyslog, how would
rsyslog behave ?  Is it possible to tell rsyslog to drop log
messages meant to be sent if too many are accumulated ?  Is there
any provision within rsyslog to handle such situation that may
not happen only at the startup of a VPN session, but at any time
the VPN tunnel would break and needs to be brought up again.

  I reckon that rsyslog can already send encrypted logs to a
remote server although in this scenario this capability would not
be used.

  Thanks for any comments - it's appreciated.


From marcin at mejor.pl  Tue Jun 28 16:14:13 2011
From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=)
Date: Tue, 28 Jun 2011 16:14:13 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <20110627205918.0cc9ce80@mistral.stie>
References: <20110627205918.0cc9ce80@mistral.stie>
Message-ID: <4E09E1B5.7040704@mejor.pl>

Hi!
I'm using rsyslog overvpn. And i;ve got problems described here:
http://www.mail-archive.com/rsyslog at lists.adiscon.com/msg04860.html .
Regards.

From rgerhards at hq.adiscon.com  Tue Jun 28 16:41:04 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 28 Jun 2011 16:41:04 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <4E09E1B5.7040704@mejor.pl>
References: <20110627205918.0cc9ce80@mistral.stie> <4E09E1B5.7040704@mejor.pl>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>

Are you re-posting or do you say, you, too have this problem as well?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw
> Sent: Tuesday, June 28, 2011 4:14 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Remote syslogging through a (broken) VPN
> 
> Hi!
> I'm using rsyslog overvpn. And i;ve got problems described here:
> http://www.mail-archive.com/rsyslog at lists.adiscon.com/msg04860.html .
> Regards.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From rgerhards at hq.adiscon.com  Tue Jun 28 16:46:15 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 28 Jun 2011 16:46:15 +0200
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com><BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com>
	<BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com>

Thanks all for the helpful comments!

More inline below...

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> Sent: Monday, June 27, 2011 2:10 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Request for Comments: config format
> 
> 2011/6/27 Brian Knox <taotetek at gmail.com>:
> > Rainer -
> >
> > Off the top of my head I don't see it as a vital feature. ?My initial
> > thought is that allowing a mixture of old and new config files is
> going to
> > be far from trivial and probably open the door up for a lot of
> confusion and
> > errors.
> 
> I kinda agree here. Mixing the two config formats smells like it opens
> a can of worms.
> The only reason why I think it could be useful, is the
> $IncludeConfig support. In Debian we use
> 
> $IncludeConfig /etc/rsyslog.d/*.conf
> 
> which is used by packages and administrators to extend the rsyslog
> configuration with custom rules.
> So, while being able to mix the config file format in one file doesn't
> sound like a vital feature, being able to include files of different
> formats might be.
> 

That's a very valid case. Even though I have to admit it probably complicates
things to the same level. In essence, this boils down to a third option:

Permit new-style config blocks within old style config (or vice versa,
doesn't matter)

I'll think more about this.

> That said, having a tool, which allows to read the old format and
> convert it to the new one, would be a very valuable feature imho.

Good idea! For old-style, I need a lot of internal plumbing, so it probably
should be an rsyslogd option (and rsyslogd being the tool ;)). It would also
be good to see how actual configs look in some of the potential new formats.

Thanks again to everyone!
Rainer

From lanas at securenet.net  Tue Jun 28 23:58:31 2011
From: lanas at securenet.net (lanas)
Date: Tue, 28 Jun 2011 17:58:31 -0400
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <20110627205918.0cc9ce80@mistral.stie>
References: <20110627205918.0cc9ce80@mistral.stie>
Message-ID: <20110628175831.0dd62ef3@mistral.stie>

Le Lundi, 27 Juin 2011 20:59:18 -0400,
lanas <lanas at securenet.net> a ?crit :

>   What happens is that the syslogd queue or cache becomes full
> and at that moment IPsec waits on syslogd.  And syslogd cannot
> send its accumulated messages.  A deadlock happens. And because
> of that, the VPN cannot cannot be established.
> 
>   If syslogd in that scenario is replaced by rsyslog, how would
> rsyslog behave ?  

Some kind of provision for when the remote syslog link is unavailable,
log messages are dumped locally until the link is re-established.  Along
with a log msg from rsyslog itself about the situation.  Then when the
link becomes available again, the backlog of log msgs could (optional)
be pumped slowly (as to not overcome the link) to the remote log
server.  Hmmm. this would mean keeping the original timestamp and
adding a new one.  Perhaps even a mention that this is backlog.  

Hmmm. And if log rotation happens while the link is down...  There
would be a need to keep track of the log files and perhaps to even have
some mandatory limit of a certain number of msgs to 'keep' while the
link is down from the time the link comes back up.  Eg. when the link
comes back up only n number of log msgs from that time are pumped back
to the remote log server.  Like 300-400 hundred or so.

Anyways, how would rsyslog currently handle a broken VPN link when
remote syslogging via that link is configured ?

Thanks for any comments !

From david at lang.hm  Wed Jun 29 00:17:23 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 28 Jun 2011 15:17:23 -0700 (PDT)
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <20110628175831.0dd62ef3@mistral.stie>
References: <20110627205918.0cc9ce80@mistral.stie>
	<20110628175831.0dd62ef3@mistral.stie>
Message-ID: <alpine.DEB.2.02.1106281504330.24638@asgard.lang.hm>

On Tue, 28 Jun 2011, lanas wrote:

> Le Lundi, 27 Juin 2011 20:59:18 -0400,
> lanas <lanas at securenet.net> a ?crit :
>
>>   What happens is that the syslogd queue or cache becomes full
>> and at that moment IPsec waits on syslogd.  And syslogd cannot
>> send its accumulated messages.  A deadlock happens. And because
>> of that, the VPN cannot cannot be established.
>>
>>   If syslogd in that scenario is replaced by rsyslog, how would
>> rsyslog behave ? 
>
> Some kind of provision for when the remote syslog link is unavailable,
> log messages are dumped locally until the link is re-established.  Along
> with a log msg from rsyslog itself about the situation.

this much is possible with rsyslog today.

sysklogd processes messages one at a time if one message blocks, 
everything else stalls waiting for it.

rsyslog has configurable size (and type) queues inside it, messages will 
wait in the queue until they can be delivered. you just need to configure 
the queue to be large enough to handle the time when you are not 
connected (and configure the retry limit and interval to keep trying to 
forward the logs)

one thing you have to decide, what do you want to do when you run out of 
space to queue messages? do you want to block (which is what syslogd did, 
and the cause of your deadlock), or do you want to loose messages? 
eventually you will run out of memory/disk to queue messages in.

If the intent is that the VPN should be up almost all the time, you may be 
able to just use memory queues (possibly making them a bit large than 
normal, depending on your message volume)

if you don't have enough memory, or you want to have the log messages 
survive a reboot of the server where they are queued, you can configure 
rsyslog with disk assisted queues. With it set this way, messages that 
cannot be sent will be queued to be sent later, and when the connection is 
established all the messages will be sent.

>  Then when the
> link becomes available again, the backlog of log msgs could (optional)
> be pumped slowly (as to not overcome the link) to the remote log
> server.  Hmmm. this would mean keeping the original timestamp and
> adding a new one.  Perhaps even a mention that this is backlog.

I would not try to do throttleing with rsyslog (it does have some 
capibilities in this area, but not great ones), do throttling at the 
network layer for that port instead.

the original timestamp will be preserved (as per the RFC) and the 
receiving system has the ability to record when it received the message. 
you just need to decide exactly what format you want to log (do you 
_really_ want each message to have multiple timestamps on it? if so, how 
many if it's relayed multiple times? how do you deal with different logs 
getting relayed different numbers of hops, and therefor having different 
number of timestamps on them? ...)

> Hmmm. And if log rotation happens while the link is down...  There
> would be a need to keep track of the log files and perhaps to even have
> some mandatory limit of a certain number of msgs to 'keep' while the
> link is down from the time the link comes back up.  Eg. when the link
> comes back up only n number of log msgs from that time are pumped back
> to the remote log server.  Like 300-400 hundred or so.
>
> Anyways, how would rsyslog currently handle a broken VPN link when
> remote syslogging via that link is configured ?

it would handle just as it would any other reason for having a broken 
connection.

Be aware, there have been some reports recently that the handling of this 
has problems, and I'm not sure if the releases last week fixed this or not 
(I do know that all the releases prior to last week appear to have 
problems), Rainer would need to comment on this.

David Lang

From david at lang.hm  Wed Jun 29 00:25:53 2011
From: david at lang.hm (david at lang.hm)
Date: Tue, 28 Jun 2011 15:25:53 -0700 (PDT)
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com><BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com>
	<BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com>
	<9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1106281520070.24638@asgard.lang.hm>

On Tue, 28 Jun 2011, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
>> Sent: Monday, June 27, 2011 2:10 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Request for Comments: config format
>>
>> 2011/6/27 Brian Knox <taotetek at gmail.com>:
>>> Rainer -
>>>
>>> Off the top of my head I don't see it as a vital feature. ?My initial
>>> thought is that allowing a mixture of old and new config files is
>> going to
>>> be far from trivial and probably open the door up for a lot of
>> confusion and
>>> errors.
>>
>> I kinda agree here. Mixing the two config formats smells like it opens
>> a can of worms.
>> The only reason why I think it could be useful, is the
>> $IncludeConfig support. In Debian we use
>>
>> $IncludeConfig /etc/rsyslog.d/*.conf
>>
>> which is used by packages and administrators to extend the rsyslog
>> configuration with custom rules.
>> So, while being able to mix the config file format in one file doesn't
>> sound like a vital feature, being able to include files of different
>> formats might be.
>>
>
> That's a very valid case. Even though I have to admit it probably complicates
> things to the same level. In essence, this boils down to a third option:
>
> Permit new-style config blocks within old style config (or vice versa,
> doesn't matter)
>
> I'll think more about this.
>
>> That said, having a tool, which allows to read the old format and
>> convert it to the new one, would be a very valuable feature imho.
>
> Good idea! For old-style, I need a lot of internal plumbing, so it probably
> should be an rsyslogd option (and rsyslogd being the tool ;)). It would also
> be good to see how actual configs look in some of the potential new formats.

If you have the ability to read the old-style configs and output 
functionally identical new style configs you have well over 90% of the 
problem solved (remember the need to convert snippits of configs, the 
things that would be in the include file)

while it would be nice to not have to worry about the includes, I don't 
think it's really worth it.

the one issue with automated conversion is that I would expect comments to 
be lost, if there is a way to make a fair stab at keeping the comments it 
would greatly simplify things

you may even consider having it try to read the file as the new format, 
and if it can't try reading it as the old format (and if you want to get 
_really_ fancy if the file is an old format file, try saving the new 
format as a new file, logging that you have done so and suggest that they 
use the new file instead)

David Lang

From pgollucci at p6m7g8.com  Wed Jun 29 02:26:01 2011
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 29 Jun 2011 00:26:01 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog6-devel Makefile
 distinfo pkg-plist ports/sysutils/rsyslog6-devel/files patch-Makefile.in
Message-ID: <4E0A7119.20308@p6m7g8.com>



-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog6-devel Makefile distinfo
     pkg-plist ports/sysutils/rsyslog6-devel/files patch-Makefile.in
Date: Wed, 29 Jun 2011 00:01:38 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2011-06-29 00:01:38 UTC

  FreeBSD ports repository

  Modified files:
    sysutils/rsyslog6-devel Makefile distinfo pkg-plist
  Removed files:
    sysutils/rsyslog6-devel/files patch-Makefile.in
  Log:
  - Update to 6.1.9

  PR:             ports/158346
  Submitted by:   Jim Riggs <ports at christianserving.org>

  Revision  Changes    Path
  1.31      +7 -4      ports/sysutils/rsyslog6-devel/Makefile
  1.20      +2 -2      ports/sysutils/rsyslog6-devel/distinfo
  1.3       +0 -19
ports/sysutils/rsyslog6-devel/files/patch-Makefile.in (dead)
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/files/patch-Makefile.in?rev=1.2&content-type=text/x-cvsweb-markup
  1.15      +5 -1      ports/sysutils/rsyslog6-devel/pkg-plist

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/Makefile.diff?r1=1.30&r2=1.31&f=h
| --- ports/sysutils/rsyslog6-devel/Makefile	2011/06/05 18:43:14	1.30
| +++ ports/sysutils/rsyslog6-devel/Makefile	2011/06/29 00:01:38	1.31
| @@ -2,12 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog6-devel/Makefile,v
1.30 2011/06/05 18:43:14 novel Exp $
| +# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog6-devel/Makefile,v
1.31 2011/06/29 00:01:38 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	6.1.1
| -PORTREVISION?=	1
| +PORTVERSION=	6.1.9
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://www.rsyslog.com/files/download/rsyslog/
|  .ifdef MNAME
| @@ -17,11 +16,15 @@ PKGNAMESUFFIX?=	-${MNAME}
|  MAINTAINER=	pgollucci at FreeBSD.org
|  COMMENT?=	Syslogd supporting SQL, TCP and TLS
|
| +USE_LDCONFIG=	yes
| +
|  .ifdef MNAME
|  RUN_DEPENDS=	rsyslog>=${PORTVERSION}:${PORTSDIR}/sysutils/rsyslog6-devel
|  PLIST=		${.CURDIR}/pkg-plist
|  LATEST_LINK=	rsyslog6-devel-${MNAME}
|  .else
| +LIB_DEPENDS=	ee:${PORTSDIR}/devel/libee \
| +		estr:${PORTSDIR}/devel/libestr
|  LATEST_LINK=	rsyslog6-devel
|  .endif
|
| @@ -35,7 +38,7 @@ EXTRA_PATCHES+=	${FILESDIR}/extra-patch-
|  EXTRA_PATCHES+=	${FILESDIR}/extra-patch-sane-hostname
|  .endif
|
| -CONFLICTS=	rsyslog-devel-[!6].[0-9]*
| +CONFLICTS=	rsyslog-devel-[!6].[0-9]*
|  CPPFLAGS+=	-I${LOCALBASE}/include
|  LDFLAGS+=	-L${LOCALBASE}/lib
|  GNU_CONFIGURE=	yes
| @@ -48,7 +51,7 @@ CONFIGURE_ARGS+=--enable-rtinst --enable
|  .if ${ARCH} == "i386"
|  CPPFLAGS+=-march=i686
|  .endif
| -CONFIGURE_ENV+=	CPPFLAGS="${CPPFLAGS}" LDFLAGS="${LDFLAGS}"
| +CONFIGURE_ENV+=	CPPFLAGS="${CPPFLAGS}" LDFLAGS="${LDFLAGS}"
LIBESTR_CFLAGS="${CPPFLAGS}" LIBESTR_LIBS="${LDFLAGS} -lestr"
LIBEE_CFLAGS="${CPPFLAGS}" LIBEE_LIBS="${LDFLAGS} -lee"
|
|  .ifndef MNAME
|  MAN8=	rsyslogd.8
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/distinfo.diff?r1=1.19&r2=1.20&f=h
| --- ports/sysutils/rsyslog6-devel/distinfo	2010/12/10 06:44:20	1.19
| +++ ports/sysutils/rsyslog6-devel/distinfo	2011/06/29 00:01:38	1.20
| @@ -1,2 +1,2 @@
| -SHA256 (rsyslog-6.1.1.tar.gz) =
970d1ae88ea544ff916a61c572afbe7a4617ef675b6a017812bb67a78265f00f
| -SIZE (rsyslog-6.1.1.tar.gz) = 2312617
| +SHA256 (rsyslog-6.1.9.tar.gz) =
b6672a95ada4946e4e7caa8cb163f3bcb271cf838b39801736019ea7bb1f034e
| +SIZE (rsyslog-6.1.9.tar.gz) = 2407312
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/pkg-plist.diff?r1=1.14&r2=1.15&f=h
| --- ports/sysutils/rsyslog6-devel/pkg-plist	2010/12/10 06:44:20	1.14
| +++ ports/sysutils/rsyslog6-devel/pkg-plist	2011/06/29 00:01:38	1.15
| @@ -6,6 +6,8 @@ lib/rsyslog/immark.la
|  lib/rsyslog/immark.so
|  lib/rsyslog/imtcp.la
|  lib/rsyslog/imtcp.so
| +lib/rsyslog/imtemplate.la
| +lib/rsyslog/imtemplate.so
|  lib/rsyslog/imudp.la
|  lib/rsyslog/imudp.so
|  lib/rsyslog/imuxsock.la
| @@ -49,8 +51,8 @@ sbin/rsyslogd
|  %%PORTDOCS%%%%DOCSDIR%%/imfile.html
|  %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html
|  %%PORTDOCS%%%%DOCSDIR%%/imklog.html
| -%%PORTDOCS%%%%DOCSDIR%%/imptcp.html
|  %%PORTDOCS%%%%DOCSDIR%%/impstats.html
| +%%PORTDOCS%%%%DOCSDIR%%/imptcp.html
|  %%PORTDOCS%%%%DOCSDIR%%/imrelp.html
|  %%PORTDOCS%%%%DOCSDIR%%/imsolaris.html
|  %%PORTDOCS%%%%DOCSDIR%%/imtcp.html
| @@ -61,6 +63,8 @@ sbin/rsyslogd
|  %%PORTDOCS%%%%DOCSDIR%%/licensing.html
|  %%PORTDOCS%%%%DOCSDIR%%/log_rotation_fix_size.html
|  %%PORTDOCS%%%%DOCSDIR%%/manual.html
| +%%PORTDOCS%%%%DOCSDIR%%/mmnormalize.html
| +%%PORTDOCS%%%%DOCSDIR%%/mmsnmptrapd.html
|  %%PORTDOCS%%%%DOCSDIR%%/modules.html
|  %%PORTDOCS%%%%DOCSDIR%%/multi_ruleset.html
|  %%PORTDOCS%%%%DOCSDIR%%/netstream.html

From pgollucci at p6m7g8.com  Wed Jun 29 02:26:37 2011
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 29 Jun 2011 00:26:37 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog3 bsd.rsyslog.mk
Message-ID: <4E0A713D.8010808@p6m7g8.com>



-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog3 bsd.rsyslog.mk
Date: Sat, 25 Jun 2011 03:50:06 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2011-06-25 03:50:06 UTC

  FreeBSD ports repository

  Modified files:
    sysutils/rsyslog3    bsd.rsyslog.mk
  Log:
  unsupported upstream

  http://www.rsyslog.com/project-status/

  Revision  Changes    Path
  1.22      +3 -0      ports/sysutils/rsyslog3/bsd.rsyslog.mk

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog3/bsd.rsyslog.mk.diff?r1=1.21&r2=1.22&f=h
| --- ports/sysutils/rsyslog3/bsd.rsyslog.mk	2011/06/05 18:43:13	1.21
| +++ ports/sysutils/rsyslog3/bsd.rsyslog.mk	2011/06/25 03:50:06	1.22
| @@ -23,6 +23,9 @@ PLIST=	""
|  DESCR?=		${.CURDIR}/../rsyslog3/pkg-descr
|  DISTINFO_FILE?=	${.CURDIR}/../rsyslog3/distinfo
|
| +DEPRECATED=	unsupported upstream
| +EXPIRATION_DATE=	2011-07-25
| +
|  do-install:
|  	@${INSTALL} -d ${PREFIX}/lib/rsyslog/
|  .for _T in ${MTYPES}

From pgollucci at p6m7g8.com  Wed Jun 29 02:26:43 2011
From: pgollucci at p6m7g8.com (Philip M. Gollucci)
Date: Wed, 29 Jun 2011 00:26:43 +0000
Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog4 Makefile
 distinfo ports/sysutils/rsyslog5 Makefile distinfo
Message-ID: <4E0A7143.1050503@p6m7g8.com>



-------- Original Message --------
Subject: cvs commit: ports/sysutils/rsyslog4 Makefile distinfo
ports/sysutils/rsyslog5 Makefile distinfo
Date: Sat, 25 Jun 2011 03:55:23 +0000 (UTC)
From: Philip M. Gollucci <pgollucci at FreeBSD.org>
To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org

pgollucci    2011-06-25 03:55:23 UTC

  FreeBSD ports repository

  Modified files:
    sysutils/rsyslog4    Makefile distinfo
    sysutils/rsyslog5    Makefile distinfo
  Log:
  Update to 5.8.2
  Update to 4.6.6
  6.x update is pending submission from another contributor

  Revision  Changes    Path
  1.22      +1 -2      ports/sysutils/rsyslog4/Makefile
  1.12      +2 -2      ports/sysutils/rsyslog4/distinfo
  1.33      +2 -6      ports/sysutils/rsyslog5/Makefile
  1.19      +2 -2      ports/sysutils/rsyslog5/distinfo

http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4/Makefile.diff?r1=1.21&r2=1.22&f=h
| --- ports/sysutils/rsyslog4/Makefile	2011/06/05 18:43:14	1.21
| +++ ports/sysutils/rsyslog4/Makefile	2011/06/25 03:55:23	1.22
| @@ -2,12 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4/Makefile,v 1.21
2011/06/05 18:43:14 novel Exp $
| +# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4/Makefile,v 1.22
2011/06/25 03:55:23 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	4.6.5
| -PORTREVISION?=	1
| +PORTVERSION=	4.6.6
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://www.rsyslog.com/files/download/rsyslog/
|  .ifdef MNAME
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4/distinfo.diff?r1=1.11&r2=1.12&f=h
| --- ports/sysutils/rsyslog4/distinfo	2010/12/10 06:44:19	1.11
| +++ ports/sysutils/rsyslog4/distinfo	2011/06/25 03:55:23	1.12
| @@ -1,2 +1,2 @@
| -SHA256 (rsyslog-4.6.5.tar.gz) =
577b35340d4d0ba95c5d444d90282dfc938e2db8cf7c9ac6996d5ac111961195
| -SIZE (rsyslog-4.6.5.tar.gz) = 2080355
| +SHA256 (rsyslog-4.6.6.tar.gz) =
b6e853bdaa8bf04168ba1448696c33514e38cc39c5d4f26b310c751954f6b3a7
| +SIZE (rsyslog-4.6.6.tar.gz) = 2088319
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.32&r2=1.33&f=h
| --- ports/sysutils/rsyslog5/Makefile	2011/04/14 22:02:25	1.32
| +++ ports/sysutils/rsyslog5/Makefile	2011/06/25 03:55:23	1.33
| @@ -2,11 +2,11 @@
|  # Date created:     29 December 2008
|  # Whom:             Cristiano Rolim Pereira <cristianorolim at hotmail.com>
|  #
| -# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.32
2011/04/14 22:02:25 pgollucci Exp $
| +# $FreeBSD:
/usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.33
2011/06/25 03:55:23 pgollucci Exp $
|  #
|
|  PORTNAME=	rsyslog
| -PORTVERSION=	5.8.0
| +PORTVERSION=	5.8.2
|  CATEGORIES=	sysutils
|  MASTER_SITES=	http://www.rsyslog.com/files/download/rsyslog/
|  .ifdef MNAME
| @@ -39,11 +39,7 @@ CPPFLAGS+=	-I${LOCALBASE}/include
|  LDFLAGS+=	-L${LOCALBASE}/lib
|  GNU_CONFIGURE=	yes
|
| -# XXX: 5.5.6+ seem to crash frequently with low-mid load
| -# on FreeBSD, temporailiy enable debugging by default.
| -# Now we can send gdb backtraces into the list:
| -# rsyslog-users <rsyslog at lists.adiscon.com>
| -OPTIONS=	DEBUG	"Enable debugging"	on
| +OPTIONS=	DEBUG	"Enable debugging"	off
|
|  .ifdef WITH_DEBUG
|  CONFIGURE_ARGS+=--enable-rtinst --enable-debug
http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.18&r2=1.19&f=h
| --- ports/sysutils/rsyslog5/distinfo	2011/04/14 22:02:25	1.18
| +++ ports/sysutils/rsyslog5/distinfo	2011/06/25 03:55:23	1.19
| @@ -1,2 +1,2 @@
| -SHA256 (rsyslog-5.8.0.tar.gz) =
e034b02473fd7e5313522173ec2f6c57e6fbcaec2c6b289edb968b1d64ae3ffd
| -SIZE (rsyslog-5.8.0.tar.gz) = 2336866
| +SHA256 (rsyslog-5.8.2.tar.gz) =
8cdf3531370b6231dae6095f7c09d72e00b4508d7f985fde472a9741818d6ff4
| +SIZE (rsyslog-5.8.2.tar.gz) = 2366785

From piavka at cs.bgu.ac.il  Wed Jun 29 08:14:27 2011
From: piavka at cs.bgu.ac.il (Piavlo)
Date: Wed, 29 Jun 2011 09:14:27 +0300
Subject: [rsyslog] imfile feature request
Message-ID: <4E0AC2C3.8010205@cs.bgu.ac.il>

  Hi,

Currently the $InputFileTag option is mandatory.
I have a case there the messages in $InputFileName already have the 
ident field (programname) as the first field in the messages - I'd like 
imfile to get the messages as-is without adding the $InputFileTag to the 
messages - so that I could use the original first field as programname 
to automatically split the messages to files  on remote system.

Is there chance such feature can be implemented.

Otherwise I need to do some ugly things with property replacer which on 
the my first trials does seems to have enough features to do that I need.

Thanks
Alex

From rgerhards at hq.adiscon.com  Wed Jun 29 10:01:04 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 29 Jun 2011 10:01:04 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <alpine.DEB.2.02.1106281504330.24638@asgard.lang.hm>
References: <20110627205918.0cc9ce80@mistral.stie><20110628175831.0dd62ef3@mistral.stie>
	<alpine.DEB.2.02.1106281504330.24638@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F23@GRFEXC.intern.adiscon.com>

> Be aware, there have been some reports recently that the handling of
> this
> has problems, and I'm not sure if the releases last week fixed this or
> not
> (I do know that all the releases prior to last week appear to have
> problems), Rainer would need to comment on this.

V4 is fine, for v5 and v6 the latest releases are needed. They fix all known
and reproducible issues (there is one issue left in the bug tracker, but I
don'T get any more feedback nor can I reproduce -- not sure if it is valid).

Rainer

From rgerhards at hq.adiscon.com  Wed Jun 29 10:03:58 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 29 Jun 2011 10:03:58 +0200
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <alpine.DEB.2.02.1106281520070.24638@asgard.lang.hm>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com><BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com><BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1106281520070.24638@asgard.lang.hm>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F24@GRFEXC.intern.adiscon.com>

Thanks for the additional feedback. I am right now thinking about the
complexities, and it is useful to have it :-). I don't come up yet with
details, because I don't have them. I probably need to wrangle a couple of
days with flex (which I prefer, but I may hand-code if that's more useful).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Wednesday, June 29, 2011 12:26 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Request for Comments: config format
> 
> On Tue, 28 Jun 2011, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl
> >> Sent: Monday, June 27, 2011 2:10 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Request for Comments: config format
> >>
> >> 2011/6/27 Brian Knox <taotetek at gmail.com>:
> >>> Rainer -
> >>>
> >>> Off the top of my head I don't see it as a vital feature. ?My
> initial
> >>> thought is that allowing a mixture of old and new config files is
> >> going to
> >>> be far from trivial and probably open the door up for a lot of
> >> confusion and
> >>> errors.
> >>
> >> I kinda agree here. Mixing the two config formats smells like it
> opens
> >> a can of worms.
> >> The only reason why I think it could be useful, is the
> >> $IncludeConfig support. In Debian we use
> >>
> >> $IncludeConfig /etc/rsyslog.d/*.conf
> >>
> >> which is used by packages and administrators to extend the rsyslog
> >> configuration with custom rules.
> >> So, while being able to mix the config file format in one file
> doesn't
> >> sound like a vital feature, being able to include files of different
> >> formats might be.
> >>
> >
> > That's a very valid case. Even though I have to admit it probably
> complicates
> > things to the same level. In essence, this boils down to a third
> option:
> >
> > Permit new-style config blocks within old style config (or vice
> versa,
> > doesn't matter)
> >
> > I'll think more about this.
> >
> >> That said, having a tool, which allows to read the old format and
> >> convert it to the new one, would be a very valuable feature imho.
> >
> > Good idea! For old-style, I need a lot of internal plumbing, so it
> probably
> > should be an rsyslogd option (and rsyslogd being the tool ;)). It
> would also
> > be good to see how actual configs look in some of the potential new
> formats.
> 
> If you have the ability to read the old-style configs and output
> functionally identical new style configs you have well over 90% of the
> problem solved (remember the need to convert snippits of configs, the
> things that would be in the include file)
> 
> while it would be nice to not have to worry about the includes, I don't
> think it's really worth it.
> 
> the one issue with automated conversion is that I would expect comments
> to
> be lost, if there is a way to make a fair stab at keeping the comments
> it
> would greatly simplify things
> 
> you may even consider having it try to read the file as the new format,
> and if it can't try reading it as the old format (and if you want to
> get
> _really_ fancy if the file is an old format file, try saving the new
> format as a new file, logging that you have done so and suggest that
> they
> use the new file instead)
> 
> David Lang

From rgerhards at hq.adiscon.com  Wed Jun 29 12:36:25 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 29 Jun 2011 12:36:25 +0200
Subject: [rsyslog] imfile feature request
In-Reply-To: <4E0AC2C3.8010205@cs.bgu.ac.il>
References: <4E0AC2C3.8010205@cs.bgu.ac.il>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F28@GRFEXC.intern.adiscon.com>

There currently is a refactor/rewrite of imfile underway. Please see here:

http://kb.monitorware.com/imfile-refactor-t10622.html

I am hesitant to add non-trivial things (like this is) until this is
finished. I suggest to add that feature request to the thread (note that I am
NOT the author of the refactoring, so I don't know if the author has seen
your list posting).

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Piavlo
> Sent: Wednesday, June 29, 2011 8:14 AM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] imfile feature request
> 
>   Hi,
> 
> Currently the $InputFileTag option is mandatory.
> I have a case there the messages in $InputFileName already have the
> ident field (programname) as the first field in the messages - I'd like
> imfile to get the messages as-is without adding the $InputFileTag to
> the
> messages - so that I could use the original first field as programname
> to automatically split the messages to files  on remote system.
> 
> Is there chance such feature can be implemented.
> 
> Otherwise I need to do some ugly things with property replacer which on
> the my first trials does seems to have enough features to do that I
> need.
> 
> Thanks
> Alex
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From igor.sverkos at googlemail.com  Wed Jun 29 13:36:46 2011
From: igor.sverkos at googlemail.com (Igor Sverkos)
Date: Wed, 29 Jun 2011 13:36:46 +0200
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F24@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com><BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com><BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com>
	<alpine.DEB.2.02.1106281520070.24638@asgard.lang.hm>
	<9B6E2A8877C38245BFB15CC491A11DA7280F24@GRFEXC.intern.adiscon.com>
Message-ID: <4E0B0E4E.709@googlemail.com>

Hi,

did I understand you correctly? You want to support different
configuration syntax in one file? That sounds very bad to me.

Have you thought about adding a version tag to the configuration file?
For example, when the file starts with

"@version 5"

everyone would know that the used syntax version is "v5" (whatever that
will mean).

This would allow us to keep old configuration files working, so packages
which will drop a configuration file in /etc/rsyslog.d doesn't need to
be updated... but new files can use the latest syntax.


-- 
Regards,
Igor



From rgerhards at hq.adiscon.com  Wed Jun 29 14:10:43 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 29 Jun 2011 14:10:43 +0200
Subject: [rsyslog] Request for Comments: config format
In-Reply-To: <4E0B0E4E.709@googlemail.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7280F0A@GRFEXC.intern.adiscon.com><BANLkTim3+EMYEYfK20kN+ifRpHnbYFbjwQ@mail.gmail.com><BANLkTinf7-17RzcX4gjCq5_a=D6G23JUmA@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA7280F1E@GRFEXC.intern.adiscon.com><alpine.DEB.2.02.1106281520070.24638@asgard.lang.hm><9B6E2A8877C38245BFB15CC491A11DA7280F24@GRFEXC.intern.adiscon.com>
	<4E0B0E4E.709@googlemail.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F2B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Igor Sverkos
> Sent: Wednesday, June 29, 2011 1:37 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Request for Comments: config format
> 
> Hi,
> 
> did I understand you correctly? You want to support different
> configuration syntax in one file? That sounds very bad to me.
> 
> Have you thought about adding a version tag to the configuration file?
> For example, when the file starts with
> 
> "@version 5"
> 
> everyone would know that the used syntax version is "v5" (whatever that
> will mean).
> 
> This would allow us to keep old configuration files working, so
> packages
> which will drop a configuration file in /etc/rsyslog.d doesn't need to
> be updated... but new files can use the latest syntax.

Actually, that doesn't help so much. Think from that perspective: The main
rsyslog.conf and all includes actually make up one big config. You can think
of include as being a copy&paste into the main config file at the spot where
the include directive is. So as far as the config text stream is involved,
all is inside one stream (even if you would split of streams to different
handlers, you'd need to merge the configs, and this can become even more
ugly). So what this means is that rsyslog needs to be able to support
different config languages (or call them versions) inside a single config
stream.

So indicating which language to use for what part is indeed a question. A
version marker as you suggest may be an option. Other options also exist.

However, with these version markers, we actually get a single config
language, which support multiple formats at the same time. Think about the
need to merge all this to a *single* in-memory config.

If you disagree, please keep posting, I may overlook something important
(actually, I'd wish it were so).

Thanks,
Rainer

From david at lang.hm  Wed Jun 29 22:49:35 2011
From: david at lang.hm (david at lang.hm)
Date: Wed, 29 Jun 2011 13:49:35 -0700 (PDT)
Subject: [rsyslog] imfile feature request
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F28@GRFEXC.intern.adiscon.com>
References: <4E0AC2C3.8010205@cs.bgu.ac.il>
	<9B6E2A8877C38245BFB15CC491A11DA7280F28@GRFEXC.intern.adiscon.com>
Message-ID: <alpine.DEB.2.02.1106291344380.24638@asgard.lang.hm>

Rainer,
   one of the things that I think needs to be done with imfile is to have 
it run through the 'normal' parser process. If you do that, then 
$InputFileTag essentially becomes a custom parser module, but if it's not 
provided, you would fall through to the traditional parser, which would 
handle files containing syslog-type data (as opposed to the message-only 
data that you are expecting now)

I think this would probably simplify imfile as well.

this would also handle the issue of escaping control characters in the 
message as a side effect.

David Lang

  On Wed, 29 Jun 2011, Rainer Gerhards wrote:

> There currently is a refactor/rewrite of imfile underway. Please see here:
>
> http://kb.monitorware.com/imfile-refactor-t10622.html
>
> I am hesitant to add non-trivial things (like this is) until this is
> finished. I suggest to add that feature request to the thread (note that I am
> NOT the author of the refactoring, so I don't know if the author has seen
> your list posting).
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Piavlo
>> Sent: Wednesday, June 29, 2011 8:14 AM
>> To: rsyslog at lists.adiscon.com
>> Subject: [rsyslog] imfile feature request
>>
>>   Hi,
>>
>> Currently the $InputFileTag option is mandatory.
>> I have a case there the messages in $InputFileName already have the
>> ident field (programname) as the first field in the messages - I'd like
>> imfile to get the messages as-is without adding the $InputFileTag to
>> the
>> messages - so that I could use the original first field as programname
>> to automatically split the messages to files  on remote system.
>>
>> Is there chance such feature can be implemented.
>>
>> Otherwise I need to do some ugly things with property replacer which on
>> the my first trials does seems to have enough features to do that I
>> need.
>>
>> Thanks
>> Alex
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>

From tbergfeld at hq.adiscon.com  Thu Jun 30 10:55:27 2011
From: tbergfeld at hq.adiscon.com (Tom Bergfeld)
Date: Thu, 30 Jun 2011 10:55:27 +0200
Subject: [rsyslog] rsyslog 5.9.1 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F3B@GRFEXC.intern.adiscon.com>

This release offers better systemd integration as well as better support for
chrooted environments by supporting better timestamps in these environments.
It also contains a number of important bug fixes.
Upgrading to this release is strongly recommended for users of the v5-devel
branch.

Remeber that v5-devel receives mostly minor and/or very urgent improvements.
The "real" development takes place in v6, which already offers features not
found in v5.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-1-v5-devel/

Download:

http://www.rsyslog.com/rsyslog-5-9-1-devel/

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

--

Support
=======
Improving rsyslog is costly, but you can help!  We are looking for
organizations that find rsyslog useful and wish to contribute back.  You can
contribute by reporting bugs, improve the software, or donate money or
equipment.

Commercial support contracts for rsyslog are available, and they help finance
continued maintenance.  Adiscon GmbH, a privately held German company, is
currently funding rsyslog development. We are always looking for interesting
development projects. For details on how to help, please see
http://www.rsyslog.com/doc-how2help.html.

From marcin at mejor.pl  Thu Jun 30 13:46:48 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Thu, 30 Jun 2011 13:46:48 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>
References: <20110627205918.0cc9ce80@mistral.stie> <4E09E1B5.7040704@mejor.pl>
	<9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>
Message-ID: <4E0C6228.4000203@mejor.pl>

W dniu 28.06.2011 16:41, Rainer Gerhards pisze:
> Are you re-posting or do you say, you, too have this problem as well?

Hi!
I'm informing Ianas that there can be a problem with broken VPN, because
i have such problem (described some time ago, by myself).

Regards,
Marcin

From rgerhards at hq.adiscon.com  Thu Jun 30 14:49:08 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 30 Jun 2011 14:49:08 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <4E0C6228.4000203@mejor.pl>
References: <20110627205918.0cc9ce80@mistral.stie>
	<4E09E1B5.7040704@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>
	<4E0C6228.4000203@mejor.pl>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F47@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw
> Sent: Thursday, June 30, 2011 1:47 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Remote syslogging through a (broken) VPN
> 
> W dniu 28.06.2011 16:41, Rainer Gerhards pisze:
> > Are you re-posting or do you say, you, too have this problem as well?
> 
> Hi!
> I'm informing Ianas that there can be a problem with broken VPN, because i
> have such problem (described some time ago, by myself).

Ah, OK. I really did not understand that part. Looked much like a (very
early) "bump" to me ;)

That said: does David's answer address your problem as well? Or is it
different?

Rainer
> Regards,
> Marcin
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From marcin at mejor.pl  Thu Jun 30 16:34:02 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Thu, 30 Jun 2011 16:34:02 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7280F47@GRFEXC.intern.adiscon.com>
References: <20110627205918.0cc9ce80@mistral.stie>	<4E09E1B5.7040704@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>	<4E0C6228.4000203@mejor.pl>
	<9B6E2A8877C38245BFB15CC491A11DA7280F47@GRFEXC.intern.adiscon.com>
Message-ID: <4E0C895A.7010809@mejor.pl>

W dniu 30.06.2011 14:49, Rainer Gerhards pisze:
> That said: does David's answer address your problem as well? Or is it
> different?

David said how rsyslog should theoretically work, but i've got issue
with such workflow. I've tried today git version. Rsyslog still doesn't
terminate after SIGTERM. My conf is:
$ModLoad imudp.so
$ModLoad imrelp.so
$ModLoad omrelp.so
$ModLoad imtcp.so
$ModLoad imfile
$WorkDirectory /var/spool/rsyslog
$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1
$MainMsgQueueMaxFileSize 512M
MainMsgQueueSaveOnShutdown on
$MainMsgQueueCheckpointInterval 600
$MainMsgQueueSyncQueueFiles on
$InputRELPServerRun 20514
$InputFileName /var/log/exim/exim_main.log
$InputFileTag hermes.exim_main:
$InputFileStateFile hermes.exim_main
$InputFileFacility mail
$InputFilePollInterval 10
$InputRunFileMonitor
$InputFileName /var/log/exim/exim_panic.log
$InputFileTag hermes.exim_panic:
$InputFileStateFile hermes.exim_panic
$InputFileFacility mail
$InputFileSeverity error
$InputRunFileMonitor
$InputFileName /var/log/apache2/access_log
$InputFileTag hermes.apache_access:
$InputFileStateFile hermes.apache_access
$InputFileFacility mail
$InputRunFileMonitor
$InputFileName /var/log/apache2/error_log
$InputFileTag hermes.apache_error:
$InputFileStateFile hermes.apache_error
$InputFileSeverity notice
$InputRunFileMonitor
*.* :omrelp:10.10.10.25:20514
:inputname, isequal, "imfile" ~
:inputname, isequal, "imrelp" ~
kern.*                          /var/log/kern.log
:programname, isequal, "named" /var/log/named/named.log
& ~
:programname, isequal, "dovecot" /var/log/dovecot.log
& ~
:msg, contains, "PWD=/ ; USER=root ;
COMMAND=/usr/lib/nagios/plugins/check_mailq" ~
:programname, isequal, "ntpd" /var/log/ntp/ntpd.log
& ~
*.info;mail.none;authpriv.none;cron.none    -/var/log/messages.log
authpriv.*              /var/log/secure.log
[cut all remains /var/log/foo.log]

P.S. I've got messages:
rsyslogd3: activation of module imudp.so failed [try
http://www.rsyslog.com/e/-3 ]
rsyslogd3: activation of module imudp.so failed [try
http://www.rsyslog.com/e/-3 ]
rsyslogd-2040: fatal error on disk queue 'action 1 queue[DA]', emergency
switch to direct mode [try http://www.rsyslog.com/e/2
040 ]

probably i missed changes of configuration format...

Regards,
Marcin

From rgerhards at hq.adiscon.com  Thu Jun 30 16:37:38 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 30 Jun 2011 16:37:38 +0200
Subject: [rsyslog] Remote syslogging through a (broken) VPN
In-Reply-To: <4E0C895A.7010809@mejor.pl>
References: <20110627205918.0cc9ce80@mistral.stie>	<4E09E1B5.7040704@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA7280F1D@GRFEXC.intern.adiscon.com>	<4E0C6228.4000203@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA7280F47@GRFEXC.intern.adiscon.com>
	<4E0C895A.7010809@mejor.pl>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7280F4C@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw
> Sent: Thursday, June 30, 2011 4:34 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Remote syslogging through a (broken) VPN
> 
> W dniu 30.06.2011 14:49, Rainer Gerhards pisze:
> > That said: does David's answer address your problem as well? Or is it
> > different?
> 
> David said how rsyslog should theoretically work, but i've got issue with
such
> workflow. I've tried today git version. Rsyslog still doesn't terminate
after
> SIGTERM. My conf is:
> $ModLoad imudp.so
> $ModLoad imrelp.so
> $ModLoad omrelp.so
> $ModLoad imtcp.so
> $ModLoad imfile
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueType LinkedList
> $ActionQueueFileName dbq
> $ActionResumeRetryCount -1
> $MainMsgQueueMaxFileSize 512M
> MainMsgQueueSaveOnShutdown on
> $MainMsgQueueCheckpointInterval 600
> $MainMsgQueueSyncQueueFiles on
> $InputRELPServerRun 20514
> $InputFileName /var/log/exim/exim_main.log $InputFileTag
> hermes.exim_main:
> $InputFileStateFile hermes.exim_main
> $InputFileFacility mail
> $InputFilePollInterval 10
> $InputRunFileMonitor
> $InputFileName /var/log/exim/exim_panic.log $InputFileTag
> hermes.exim_panic:
> $InputFileStateFile hermes.exim_panic
> $InputFileFacility mail
> $InputFileSeverity error
> $InputRunFileMonitor
> $InputFileName /var/log/apache2/access_log $InputFileTag
> hermes.apache_access:
> $InputFileStateFile hermes.apache_access $InputFileFacility mail
> $InputRunFileMonitor $InputFileName /var/log/apache2/error_log
> $InputFileTag hermes.apache_error:
> $InputFileStateFile hermes.apache_error
> $InputFileSeverity notice
> $InputRunFileMonitor
> *.* :omrelp:10.10.10.25:20514
> :inputname, isequal, "imfile" ~
> :inputname, isequal, "imrelp" ~
> kern.*                          /var/log/kern.log
> :programname, isequal, "named" /var/log/named/named.log & ~
> :programname, isequal, "dovecot" /var/log/dovecot.log & ~ :msg, contains,
> "PWD=/ ; USER=root ; COMMAND=/usr/lib/nagios/plugins/check_mailq" ~
> :programname, isequal, "ntpd" /var/log/ntp/ntpd.log & ~
> *.info;mail.none;authpriv.none;cron.none    -/var/log/messages.log
> authpriv.*              /var/log/secure.log
> [cut all remains /var/log/foo.log]
> 
> P.S. I've got messages:
> rsyslogd3: activation of module imudp.so failed [try
> http://www.rsyslog.com/e/-3 ]
> rsyslogd3: activation of module imudp.so failed [try
> http://www.rsyslog.com/e/-3 ]

Interesting. Can you provide a debug log? Startup is sufficient.

> rsyslogd-2040: fatal error on disk queue 'action 1 queue[DA]', emergency
> switch to direct mode [try http://www.rsyslog.com/e/2
> 040 ]

The on-disk data structures for the queue are corrupt. This results in direct
mode. And, now I see it, if the main queue is in direct mode, the input can
probably not be terminated. This is an extreme border case that I did not yet
consider (nor do I know how to handle that, I have to admit).

Rainer
> 
> probably i missed changes of configuration format...
> 
> Regards,
> Marcin
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com