From david at lang.hm Wed Jul 1 00:42:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:42:14 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > ubuntu 9.04 correction, debian 5.0, not ubuntu (forgot which machine I was compiling on) master also fails with the same error, v4-stable has no problem. David Lang > zlib version > /* zlib.h -- interface of the 'zlib' general purpose compression library > version 1.2.3.3, October 2nd, 2006 > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > compression library - development > > > error > > In file included from zlibw.h:28, > from stream.h:72, > from obj.h:50, > from rsyslog.h:482, > from stringbuf.c:39: > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gzseek64' > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gztell64' > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > before 'off64_t' > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > before 'off64_t' > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 00:50:13 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:50:13 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > On Tue, 30 Jun 2009, david at lang.hm wrote: > >> ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. and the error remains the same with configure --disable-zlib David Lang > David Lang > > >> zlib version >> /* zlib.h -- interface of the 'zlib' general purpose compression library >> version 1.2.3.3, October 2nd, 2006 >> >> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >> compression library - development >> >> >> error >> >> In file included from zlibw.h:28, >> from stream.h:72, >> from obj.h:50, >> from rsyslog.h:482, >> from stringbuf.c:39: >> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gzseek64' >> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gztell64' >> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >> before 'off64_t' >> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >> before 'off64_t' >> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 09:13:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 09:13:29 +0200 Subject: [rsyslog] v5-devel won't compile References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Looks like a problem "with" Debian. I guess I defined some types that zlib also defines to the same names. I have to admit that Michael Biebl told me, but it somehow slipped my mind... Will test on Debian today and fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 12:42 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Tue, 30 Jun 2009, david at lang.hm wrote: > > > ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. > > David Lang > > > > zlib version > > /* zlib.h -- interface of the 'zlib' general purpose compression library > > version 1.2.3.3, October 2nd, 2006 > > > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > compression library - development > > > > > > error > > > > In file included from zlibw.h:28, > > from stream.h:72, > > from obj.h:50, > > from rsyslog.h:482, > > from stringbuf.c:39: > > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gzseek64' > > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gztell64' > > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > > before 'off64_t' > > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > > before 'off64_t' > > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory `/usr/src/rsyslog' > > make: *** [all] Error 2 > > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:10:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:10:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Looks like a problem "with" Debian. I guess I defined some types that zlib > also defines to the same names. I have to admit that Michael Biebl told me, > but it somehow slipped my mind... Will test on Debian today and fix. master and v5-devel also complained that I didn't have the zlib development libraries installed (unable to find zlib.h), after I installed that it gave me the error I posted below. v4 didn't complain about the missing library to start with as I posted in another message, I was suprised the the problem didn't go away when I configured with --disable-zlib David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 12:42 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> On Tue, 30 Jun 2009, david at lang.hm wrote: >> >>> ubuntu 9.04 >> >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling >> on) >> >> master also fails with the same error, v4-stable has no problem. >> >> David Lang >> >> >>> zlib version >>> /* zlib.h -- interface of the 'zlib' general purpose compression library >>> version 1.2.3.3, October 2nd, 2006 >>> >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>> compression library - development >>> >>> >>> error >>> >>> In file included from zlibw.h:28, >>> from stream.h:72, >>> from obj.h:50, >>> from rsyslog.h:482, >>> from stringbuf.c:39: >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gzseek64' >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gztell64' >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:14:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:14:37 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:11 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Looks like a problem "with" Debian. I guess I defined some types that zlib > > also defines to the same names. I have to admit that Michael Biebl told me, > > but it somehow slipped my mind... Will test on Debian today and fix. > > master and v5-devel also complained that I didn't have the zlib > development libraries installed (unable to find zlib.h), after I installed > that it gave me the error I posted below. I have begun to look into it. Debian has a slightly newer version of zlib than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem occurs. I now need to look at the root cause, but it is well hidden ;) > > v4 didn't complain about the missing library to start with > > as I posted in another message, I was suprised the the problem didn't go > away when I configured with --disable-zlib That's a bug in omfile, it obviously does not yet properly reflect this switch. Will fix that one, too. Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Wednesday, July 01, 2009 12:42 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > >> > >>> ubuntu 9.04 > >> > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > >> on) > >> > >> master also fails with the same error, v4-stable has no problem. > >> > >> David Lang > >> > >> > >>> zlib version > >>> /* zlib.h -- interface of the 'zlib' general purpose compression library > >>> version 1.2.3.3, October 2nd, 2006 > >>> > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>> compression library - development > >>> > >>> > >>> error > >>> > >>> In file included from zlibw.h:28, > >>> from stream.h:72, > >>> from obj.h:50, > >>> from rsyslog.h:482, > >>> from stringbuf.c:39: > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gzseek64' > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gztell64' > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>> make[1]: *** [all-recursive] Error 1 > >>> make[1]: Leaving directory `/usr/src/rsyslog' > >>> make: *** [all] Error 2 > >>> > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 10:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:35:56 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Of course (and as always), the problem rooted in rsyslog. The commit says it all: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 308f383bec1b8be8 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, July 01, 2009 10:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, July 01, 2009 10:11 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > > > Looks like a problem "with" Debian. I guess I defined some types that > zlib > > > also defines to the same names. I have to admit that Michael Biebl told > me, > > > but it somehow slipped my mind... Will test on Debian today and fix. > > > > master and v5-devel also complained that I didn't have the zlib > > development libraries installed (unable to find zlib.h), after I installed > > that it gave me the error I posted below. > > I have begun to look into it. Debian has a slightly newer version of zlib > than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > occurs. I now need to look at the root cause, but it is well hidden ;) > > > > > v4 didn't complain about the missing library to start with > > > > as I posted in another message, I was suprised the the problem didn't go > > away when I configured with --disable-zlib > > That's a bug in omfile, it obviously does not yet properly reflect this > switch. Will fix that one, too. > > Rainer > > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > >> Sent: Wednesday, July 01, 2009 12:42 AM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] v5-devel won't compile > > >> > > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > > >> > > >>> ubuntu 9.04 > > >> > > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > > >> on) > > >> > > >> master also fails with the same error, v4-stable has no problem. > > >> > > >> David Lang > > >> > > >> > > >>> zlib version > > >>> /* zlib.h -- interface of the 'zlib' general purpose compression > library > > >>> version 1.2.3.3, October 2nd, 2006 > > >>> > > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > >>> compression library - development > > >>> > > >>> > > >>> error > > >>> > > >>> In file included from zlibw.h:28, > > >>> from stream.h:72, > > >>> from obj.h:50, > > >>> from rsyslog.h:482, > > >>> from stringbuf.c:39: > > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gzseek64' > > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gztell64' > > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > >>> make[1]: *** [all-recursive] Error 1 > > >>> make[1]: Leaving directory `/usr/src/rsyslog' > > >>> make: *** [all] Error 2 > > >>> > > >>> > > >>> David Lang > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:53:15 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:53:15 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 I'll test it when I get into the office. when I tried to do it on my laptop I ran into the testbench failure again, even with --disable-testbench CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/bash: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 config.log: $ ./configure --disable-testbench David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:56:08 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAA0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Of course (and as always), the problem rooted in rsyslog. The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > > 308f383bec1b8be8 > > I'll test it when I get into the office. when I tried to do it on my > laptop I ran into the testbench failure again, even with > --disable-testbench > > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/bash: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > Mmhhh... I did my testing on a debian 5.0 machine without javadev and with --enable-testbench=no. Seems to make a difference (I am still not deep inside autoconf...). > > config.log: $ ./configure --disable-testbench > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined some types that > >> zlib > >>>> also defines to the same names. I have to admit that Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > >> occurs. I now need to look at the root cause, but it is well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 15:18:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 15:18:35 +0200 Subject: [rsyslog] performance optimization (milestone) done Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have reached a milestone with my optimizations. My goal of reducing malloc/free calls has now been reached. I need to revisit priorities, but I will probably move away from performance optimization for a while now (maybe some smaller things that I stumble across, but not a real effort). As I have written recently, there is still ample potential for optimization. I will "just" probably not explore it at this point in time. However, I would like to ask one question that could lead to a new effort (in v5). Looking at the queue engine, I see that managing the queue, especially in ultra-reliable mode, requires a lot of effort, what also means a lot of CPU time. If we relax reliability conditions, we could definitely save some time here (probably a real lot!). So my question is what would be the least reliability that you need for a non-audit, high-performance scenario. I can envision that it is sufficient to have this: All messages handed over to the queue will be processed if no system failure occurs and if there is space available inside the queue. All messages are queued in main memory. If messages are tried to be enqueued when the queue is full, these messages will be lost. Message loss victims will be strictly based on order of enqueue (queue full condition) and not severity or any other property (evaluating that would take time, again). Also, it is acceptable to lose unprocessed messages during rsyslogd shutdown, if they cannot be processed within the (configurable) shutdown intervals. Would this make sense for sufficiently important use cases? If so, I could see that (over time!) I implement a very fast queue driver based on that relaxed criteria. With it, I think I could also create a lock-free version (but not immediately), which would definitely be a big performance gain. Feedback is appreciated. The optimized builds are currently in the git master and v5-devel branches, new releases are due soon (but will see that I do at least a bit more doc work before releasing them). Rainer From david at lang.hm Wed Jul 1 18:58:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 09:58:55 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I just wanted to let you know that I have reached a milestone with my > optimizations. My goal of reducing malloc/free calls has now been reached. I > need to revisit priorities, but I will probably move away from performance > optimization for a while now (maybe some smaller things that I stumble > across, but not a real effort). > > As I have written recently, there is still ample potential for optimization. > I will "just" probably not explore it at this point in time. > > However, I would like to ask one question that could lead to a new effort (in > v5). Looking at the queue engine, I see that managing the queue, especially > in ultra-reliable mode, requires a lot of effort, what also means a lot of > CPU time. > > If we relax reliability conditions, we could definitely save some time here > (probably a real lot!). So my question is what would be the least reliability > that you need for a non-audit, high-performance scenario. I can envision that > it is sufficient to have this: > > All messages handed over to the queue will be processed if no system failure > occurs and if there is space available inside the queue. All messages are > queued in main memory. If messages are tried to be enqueued when the queue is > full, these messages will be lost. Message loss victims will be strictly > based on order of enqueue (queue full condition) and not severity or any > other property (evaluating that would take time, again). Also, it is > acceptable to lose unprocessed messages during rsyslogd shutdown, if they > cannot be processed within the (configurable) shutdown intervals. > > Would this make sense for sufficiently important use cases? If so, I could > see that (over time!) I implement a very fast queue driver based on that > relaxed criteria. With it, I think I could also create a lock-free version > (but not immediately), which would definitely be a big performance gain. how is this different than what happens today? David Lang > Feedback is appreciated. > > The optimized builds are currently in the git master and v5-devel branches, > new releases are due soon (but will see that I do at least a bit more doc > work before releasing them). > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 19:37:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 10:37:27 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: back on the debian 5 system I get the following error with v5-devel commit 6322343eca1084b509386a94c1bcf2ca819f1220 gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o rsyslogd-omfwd.o rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread ../runtime/.libs/librsyslog.a -ldl -lrt rsyslogd-syslogd.o: In function `mainloop': /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to `execScheduled' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtExit': /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference to `rulesetClassExit' /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference to `ruleClassExit' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtInit': /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference to `propClassInit' /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference to `ruleClassInit' /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference to `rulesetClassInit' ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In function `objClassInit': /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to `apcClassInit' collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/rsyslog/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Date: Wed, 1 Jul 2009 10:35:56 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:24:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:24:17 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, runtime/apc.c) are missing. I have just checked, they are in the git branch. Also I have no problems compiling that version, I tried it on Debian this morning during the zlib fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 7:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > back on the debian 5 system I get the following error with v5-devel > commit 6322343eca1084b509386a94c1bcf2ca819f1220 > > > gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > rsyslogd-omfwd.o > rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > ../runtime/.libs/librsyslog.a -ldl -lrt > rsyslogd-syslogd.o: In function `mainloop': > /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > `execScheduled' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtExit': > /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > to `rulesetClassExit' > /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > to `ruleClassExit' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtInit': > /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > to `propClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > to `ruleClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > to `rulesetClassInit' > ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > function `objClassInit': > /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > `apcClassInit' > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > > On Wed, 1 Jul 2009, Rainer > Gerhards wrote: > > > Date: Wed, 1 Jul 2009 10:35:56 +0200 > > From: Rainer Gerhards > > Reply-To: rsyslog-users > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > Of course (and as always), the problem rooted in rsyslog. > The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > 39b46630d95a8d8 > > 308f383bec1b8be8 > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined > some types that > >> zlib > >>>> also defines to the same names. I have to admit that > Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian > today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer > version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > header the problem > >> occurs. I now need to look at the root cause, but it is > well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the > problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly > reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose > compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:29:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:29:08 +0200 Subject: [rsyslog] performance optimization (milestone) done References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 6:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] performance optimization (milestone) done > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I just wanted to let you know that I have reached a > milestone with my > > optimizations. My goal of reducing malloc/free calls has > now been reached. I > > need to revisit priorities, but I will probably move away > from performance > > optimization for a while now (maybe some smaller things > that I stumble > > across, but not a real effort). > > > > As I have written recently, there is still ample potential > for optimization. > > I will "just" probably not explore it at this point in time. > > > > However, I would like to ask one question that could lead > to a new effort (in > > v5). Looking at the queue engine, I see that managing the > queue, especially > > in ultra-reliable mode, requires a lot of effort, what also > means a lot of > > CPU time. > > > > If we relax reliability conditions, we could definitely > save some time here > > (probably a real lot!). So my question is what would be the > least reliability > > that you need for a non-audit, high-performance scenario. I > can envision that > > it is sufficient to have this: > > > > All messages handed over to the queue will be processed if > no system failure > > occurs and if there is space available inside the queue. > All messages are > > queued in main memory. If messages are tried to be enqueued > when the queue is > > full, these messages will be lost. Message loss victims > will be strictly > > based on order of enqueue (queue full condition) and not > severity or any > > other property (evaluating that would take time, again). Also, it is > > acceptable to lose unprocessed messages during rsyslogd > shutdown, if they > > cannot be processed within the (configurable) shutdown intervals. > > > > Would this make sense for sufficiently important use cases? > If so, I could > > see that (over time!) I implement a very fast queue driver > based on that > > relaxed criteria. With it, I think I could also create a > lock-free version > > (but not immediately), which would definitely be a big > performance gain. > > how is this different than what happens today? Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, persistence, reliability, discard messages based on priority, slow down sender, rate limiting ... stuff. All nice and useful, but takes time... What I described above would be the sole capability, (almost) none of these bells and whistles (except those that are handled in the action engine, I do not know out of my head which these are, but a low subset of what I said). Rainer > > David Lang > > > Feedback is appreciated. > > > > The optimized builds are currently in the git master and > v5-devel branches, > > new releases are due soon (but will see that I do at least > a bit more doc > > work before releasing them). > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:51:36 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, > runtime/apc.c) are missing. I have just checked, they are in the git branch. > Also I have no problems compiling that version, I tried it on Debian this > morning during the zlib fix. hmm, those files were definantly missing. I had done a git checkout origin/v5-devel, I then did a git checkout -f origin/v5-devel and the files are there. now it is dieing in the configure step :-( checking for FSSTND support... yes ./configure: line 10307: syntax error near unexpected token `GNUTLS,' ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= 1.4.0)' make: *** [config.status] Error 2 I tried doing the autoreconf command, and it isn't happy #autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 my next step is going to be to nuke the entire directory and try a checkout again. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 7:37 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> back on the debian 5 system I get the following error with v5-devel >> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >> >> >> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >> rsyslogd-omfwd.o >> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >> ../runtime/.libs/librsyslog.a -ldl -lrt >> rsyslogd-syslogd.o: In function `mainloop': >> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >> `execScheduled' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtExit': >> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >> to `rulesetClassExit' >> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >> to `ruleClassExit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtInit': >> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >> to `propClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >> to `ruleClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >> to `rulesetClassInit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >> function `objClassInit': >> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >> `apcClassInit' >> collect2: ld returned 1 exit status >> make[2]: *** [rsyslogd] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/tools' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> >> On Wed, 1 Jul 2009, Rainer >> Gerhards wrote: >> >>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> Of course (and as always), the problem rooted in rsyslog. >> The commit says it >>> all: >>> >>> >> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >> 39b46630d95a8d8 >>> 308f383bec1b8be8 >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>> >>>>>> Looks like a problem "with" Debian. I guess I defined >> some types that >>>> zlib >>>>>> also defines to the same names. I have to admit that >> Michael Biebl told >>>> me, >>>>>> but it somehow slipped my mind... Will test on Debian >> today and fix. >>>>> >>>>> master and v5-devel also complained that I didn't have the zlib >>>>> development libraries installed (unable to find zlib.h), after I >>> installed >>>>> that it gave me the error I posted below. >>>> >>>> I have begun to look into it. Debian has a slightly newer >> version of zlib >>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >> header the problem >>>> occurs. I now need to look at the root cause, but it is >> well hidden ;) >>>> >>>>> >>>>> v4 didn't complain about the missing library to start with >>>>> >>>>> as I posted in another message, I was suprised the the >> problem didn't go >>>>> away when I configured with --disable-zlib >>>> >>>> That's a bug in omfile, it obviously does not yet properly >> reflect this >>>> switch. Will fix that one, too. >>>> >>>> Rainer >>>> >>>>> >>>>> David Lang >>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>> >>>>>>>> ubuntu 9.04 >>>>>>> >>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>> compiling >>>>>>> on) >>>>>>> >>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>>> zlib version >>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >> compression >>>> library >>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>> >>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>> compression library - development >>>>>>>> >>>>>>>> >>>>>>>> error >>>>>>>> >>>>>>>> In file included from zlibw.h:28, >>>>>>>> from stream.h:72, >>>>>>>> from obj.h:50, >>>>>>>> from rsyslog.h:482, >>>>>>>> from stringbuf.c:39: >>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gztell64' >>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>> make: *** [all] Error 2 >>>>>>>> >>>>>>>> >>>>>>>> David Lang >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:52:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:52:57 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 6:59 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] performance optimization (milestone) done >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> Hi all, >>> >>> I just wanted to let you know that I have reached a >> milestone with my >>> optimizations. My goal of reducing malloc/free calls has >> now been reached. I >>> need to revisit priorities, but I will probably move away >> from performance >>> optimization for a while now (maybe some smaller things >> that I stumble >>> across, but not a real effort). >>> >>> As I have written recently, there is still ample potential >> for optimization. >>> I will "just" probably not explore it at this point in time. >>> >>> However, I would like to ask one question that could lead >> to a new effort (in >>> v5). Looking at the queue engine, I see that managing the >> queue, especially >>> in ultra-reliable mode, requires a lot of effort, what also >> means a lot of >>> CPU time. >>> >>> If we relax reliability conditions, we could definitely >> save some time here >>> (probably a real lot!). So my question is what would be the >> least reliability >>> that you need for a non-audit, high-performance scenario. I >> can envision that >>> it is sufficient to have this: >>> >>> All messages handed over to the queue will be processed if >> no system failure >>> occurs and if there is space available inside the queue. >> All messages are >>> queued in main memory. If messages are tried to be enqueued >> when the queue is >>> full, these messages will be lost. Message loss victims >> will be strictly >>> based on order of enqueue (queue full condition) and not >> severity or any >>> other property (evaluating that would take time, again). Also, it is >>> acceptable to lose unprocessed messages during rsyslogd >> shutdown, if they >>> cannot be processed within the (configurable) shutdown intervals. >>> >>> Would this make sense for sufficiently important use cases? >> If so, I could >>> see that (over time!) I implement a very fast queue driver >> based on that >>> relaxed criteria. With it, I think I could also create a >> lock-free version >>> (but not immediately), which would definitely be a big >> performance gain. >> >> how is this different than what happens today? > > Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, > persistence, reliability, discard messages based on priority, slow down > sender, rate limiting ... stuff. All nice and useful, but takes time... > > What I described above would be the sole capability, (almost) none of these > bells and whistles (except those that are handled in the action engine, I do > not know out of my head which these are, but a low subset of what I said). what you described as the new, limited capabilities sounds like what I would expect from a syslog daemon. David Lang From david at lang.hm Wed Jul 1 22:58:59 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:58:59 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. still no luck, secdev:/usr/src/rsyslog# rm -r * secdev:/usr/src/rsyslog# ls -a . .. .deps .git .gitignore .libs secdev:/usr/src/rsyslog# ls .libs lmtcpclt.la lmtcpclt.lai lmtcpclt_la-tcpclt.o lmtcpclt.so lmtcpsrv.la lmtcpsrv.lai lmtcpsrv_la-tcpsrv.o lmtcpsrv_la-tcps_sess.o lmtcpsrv.so secdev:/usr/src/rsyslog# !git git checkout -f origin/v5-devel Checking out files: 100% (497/497), done. HEAD is now at 6322343... Merge branch 'master' into v5-devel secdev:/usr/src/rsyslog# ls runtime/rule* runtime/rule.c runtime/rule.h runtime/ruleset.c runtime/ruleset.h secdev:/usr/src/rsyslog# autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 secdev:/usr/src/rsyslog# autoreconf -V autoreconf (GNU Autoconf) 2.61 Copyright (C) 2006 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Written by David J. MacKenzie and Akim Demaille. --- Autoconf 2.50 chosen by Debian wrapper script. For information and tuning advice see autoconf(1). > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:02:23 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:02:23 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: never mind, pkg-config was not installed on this server (and I've got _no_ idea how I was sucessfully compiling the v4-stable branch on this box) David Lang On Wed, 1 Jul 2009, david at lang.hm wrote: > Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. > > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:22:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:22:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > never mind, pkg-config was not installed on this server (and I've got _no_ > idea how I was sucessfully compiling the v4-stable branch on this box) that wasn't it either, it turns out that the libtool package is needed as well. disabling the testbench doesn't seem to work for me (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' Making all in tests make[2]: Entering directory `/usr/src/rsyslog/tests' CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/sh: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 secdev:/usr/src/rsyslog# history |tail 1408 ./configure 1409 ./configure --help 1410 ./configure --enable-imfile 1411 make 1412 ./configure --help |grep test 1413 ./configure --enable-imfile --enable-testbench=no 1414 make 1415 make clean 1416 make 1417 history |tail I'm now installing java.. David Lang > On Wed, 1 Jul 2009, david at lang.hm wrote: > >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >>> runtime/apc.c) are missing. I have just checked, they are in the git branch. >>> Also I have no problems compiling that version, I tried it on Debian this >>> morning during the zlib fix. >> >> hmm, those files were definantly missing. >> >> I had done a git checkout origin/v5-devel, I then did a git checkout -f >> origin/v5-devel and the files are there. >> >> now it is dieing in the configure step :-( >> >> checking for FSSTND support... yes >> ./configure: line 10307: syntax error near unexpected token `GNUTLS,' >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= >> 1.4.0)' >> make: *** [config.status] Error 2 >> >> >> I tried doing the autoreconf command, and it isn't happy >> >> #autoreconf -fvi >> autoreconf2.50: Entering directory `.' >> autoreconf2.50: configure.ac: not using Gettext >> autoreconf2.50: running: aclocal --force -I m4 >> autoreconf2.50: configure.ac: tracing >> autoreconf2.50: configure.ac: not using Libtool >> autoreconf2.50: running: /usr/bin/autoconf --force >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC >> If this token and others are legitimate, please use >> m4_pattern_allow. >> See the Autoconf documentation. >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 >> >> my next step is going to be to nuke the entire directory and try a >> checkout again. >> >> David Lang >> >> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>> Sent: Wednesday, July 01, 2009 7:37 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> back on the debian 5 system I get the following error with v5-devel >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>>> >>>> >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>>> rsyslogd-omfwd.o >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>>> ../runtime/.libs/librsyslog.a -ldl -lrt >>>> rsyslogd-syslogd.o: In function `mainloop': >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>>> `execScheduled' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtExit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>>> to `rulesetClassExit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>>> to `ruleClassExit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtInit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>>> to `propClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>>> to `ruleClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>>> to `rulesetClassInit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>>> function `objClassInit': >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>>> `apcClassInit' >>>> collect2: ld returned 1 exit status >>>> make[2]: *** [rsyslogd] Error 1 >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>>> make[1]: *** [all-recursive] Error 1 >>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>> make: *** [all] Error 2 >>>> >>>> >>>> >>>> On Wed, 1 Jul 2009, Rainer >>>> Gerhards wrote: >>>> >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>>> From: Rainer Gerhards >>>>> Reply-To: rsyslog-users >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> Of course (and as always), the problem rooted in rsyslog. >>>> The commit says it >>>>> all: >>>>> >>>>> >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>>> 39b46630d95a8d8 >>>>> 308f383bec1b8be8 >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>>> >>>>>>>> Looks like a problem "with" Debian. I guess I defined >>>> some types that >>>>>> zlib >>>>>>>> also defines to the same names. I have to admit that >>>> Michael Biebl told >>>>>> me, >>>>>>>> but it somehow slipped my mind... Will test on Debian >>>> today and fix. >>>>>>> >>>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>>> development libraries installed (unable to find zlib.h), after I >>>>> installed >>>>>>> that it gave me the error I posted below. >>>>>> >>>>>> I have begun to look into it. Debian has a slightly newer >>>> version of zlib >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>>> header the problem >>>>>> occurs. I now need to look at the root cause, but it is >>>> well hidden ;) >>>>>> >>>>>>> >>>>>>> v4 didn't complain about the missing library to start with >>>>>>> >>>>>>> as I posted in another message, I was suprised the the >>>> problem didn't go >>>>>>> away when I configured with --disable-zlib >>>>>> >>>>>> That's a bug in omfile, it obviously does not yet properly >>>> reflect this >>>>>> switch. Will fix that one, too. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>>> >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>>> >>>>>>>>>> ubuntu 9.04 >>>>>>>>> >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>>> compiling >>>>>>>>> on) >>>>>>>>> >>>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>> >>>>>>>>>> zlib version >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>>> compression >>>>>> library >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>>> >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>>> compression library - development >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> error >>>>>>>>>> >>>>>>>>>> In file included from zlibw.h:28, >>>>>>>>>> from stream.h:72, >>>>>>>>>> from obj.h:50, >>>>>>>>>> from rsyslog.h:482, >>>>>>>>>> from stringbuf.c:39: >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>>> make: *** [all] Error 2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> David Lang >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Jul 2 07:57:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 07:57:04 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAB0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 11:23 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > never mind, pkg-config was not installed on this server > (and I've got _no_ > > idea how I was sucessfully compiling the v4-stable branch > on this box) > > that wasn't it either, it turns out that the libtool package > is needed as > well. > > disabling the testbench doesn't seem to work for me Did you use "--enable-testbench=no"? This definitely works for me... > > (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) > make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' > Making all in tests > make[2]: Entering directory `/usr/src/rsyslog/tests' > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/sh: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > secdev:/usr/src/rsyslog# history |tail > 1408 ./configure > 1409 ./configure --help > 1410 ./configure --enable-imfile > 1411 make > 1412 ./configure --help |grep test > 1413 ./configure --enable-imfile --enable-testbench=no > 1414 make > 1415 make clean > 1416 make > 1417 history |tail > > > I'm now installing java.. > > David Lang > > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > >> > >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> > >>> This sounds like a couple of files (runtime/rule.c, > runtime/ruleset.c, > >>> runtime/apc.c) are missing. I have just checked, they are > in the git branch. > >>> Also I have no problems compiling that version, I tried > it on Debian this > >>> morning during the zlib fix. > >> > >> hmm, those files were definantly missing. > >> > >> I had done a git checkout origin/v5-devel, I then did a > git checkout -f > >> origin/v5-devel and the files are there. > >> > >> now it is dieing in the configure step :-( > >> > >> checking for FSSTND support... yes > >> ./configure: line 10307: syntax error near unexpected > token `GNUTLS,' > >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > >> 1.4.0)' > >> make: *** [config.status] Error 2 > >> > >> > >> I tried doing the autoreconf command, and it isn't happy > >> > >> #autoreconf -fvi > >> autoreconf2.50: Entering directory `.' > >> autoreconf2.50: configure.ac: not using Gettext > >> autoreconf2.50: running: aclocal --force -I m4 > >> autoreconf2.50: configure.ac: tracing > >> autoreconf2.50: configure.ac: not using Libtool > >> autoreconf2.50: running: /usr/bin/autoconf --force > >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > >> If this token and others are legitimate, please use > >> m4_pattern_allow. > >> See the Autoconf documentation. > >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > >> > >> my next step is going to be to nuke the entire directory and try a > >> checkout again. > >> > >> David Lang > >> > >> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com > >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >>>> Sent: Wednesday, July 01, 2009 7:37 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>> > >>>> back on the debian 5 system I get the following error > with v5-devel > >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 > >>>> > >>>> > >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > >>>> rsyslogd-omfwd.o > >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > >>>> ../runtime/.libs/librsyslog.a -ldl -lrt > >>>> rsyslogd-syslogd.o: In function `mainloop': > >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > >>>> `execScheduled' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtExit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > >>>> to `rulesetClassExit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > >>>> to `ruleClassExit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtInit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > >>>> to `propClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > >>>> to `ruleClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > >>>> to `rulesetClassInit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > >>>> function `objClassInit': > >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > >>>> `apcClassInit' > >>>> collect2: ld returned 1 exit status > >>>> make[2]: *** [rsyslogd] Error 1 > >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' > >>>> make[1]: *** [all-recursive] Error 1 > >>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>> make: *** [all] Error 2 > >>>> > >>>> > >>>> > >>>> On Wed, 1 Jul 2009, Rainer > >>>> Gerhards wrote: > >>>> > >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 > >>>>> From: Rainer Gerhards > >>>>> Reply-To: rsyslog-users > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> Of course (and as always), the problem rooted in rsyslog. > >>>> The commit says it > >>>>> all: > >>>>> > >>>>> > >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > >>>> 39b46630d95a8d8 > >>>>> 308f383bec1b8be8 > >>>>> > >>>>> Rainer > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM > >>>>>> To: rsyslog-users > >>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>> > >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>>>>>> > >>>>>>>> Looks like a problem "with" Debian. I guess I defined > >>>> some types that > >>>>>> zlib > >>>>>>>> also defines to the same names. I have to admit that > >>>> Michael Biebl told > >>>>>> me, > >>>>>>>> but it somehow slipped my mind... Will test on Debian > >>>> today and fix. > >>>>>>> > >>>>>>> master and v5-devel also complained that I didn't > have the zlib > >>>>>>> development libraries installed (unable to find > zlib.h), after I > >>>>> installed > >>>>>>> that it gave me the error I posted below. > >>>>>> > >>>>>> I have begun to look into it. Debian has a slightly newer > >>>> version of zlib > >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > >>>> header the problem > >>>>>> occurs. I now need to look at the root cause, but it is > >>>> well hidden ;) > >>>>>> > >>>>>>> > >>>>>>> v4 didn't complain about the missing library to start with > >>>>>>> > >>>>>>> as I posted in another message, I was suprised the the > >>>> problem didn't go > >>>>>>> away when I configured with --disable-zlib > >>>>>> > >>>>>> That's a bug in omfile, it obviously does not yet properly > >>>> reflect this > >>>>>> switch. Will fix that one, too. > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>>> Rainer > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>>>> > >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>>>>>> > >>>>>>>>>> ubuntu 9.04 > >>>>>>>>> > >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which > machine I was > >>>>> compiling > >>>>>>>>> on) > >>>>>>>>> > >>>>>>>>> master also fails with the same error, v4-stable > has no problem. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> zlib version > >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose > >>>> compression > >>>>>> library > >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>>>>>> > >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>>>>>> compression library - development > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> error > >>>>>>>>>> > >>>>>>>>>> In file included from zlibw.h:28, > >>>>>>>>>> from stream.h:72, > >>>>>>>>>> from obj.h:50, > >>>>>>>>>> from rsyslog.h:482, > >>>>>>>>>> from stringbuf.c:39: > >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gzseek64' > >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gztell64' > >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>>>>>> make: *** [all] Error 2 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> David Lang > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>> http://www.rsyslog.com > >>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From dirk.schulz at kinzesberg.de Thu Jul 2 10:34:45 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 02 Jul 2009 10:34:45 +0200 Subject: [rsyslog] Rsyslog on NetBSD 5 Message-ID: <4A4C7125.6090109@kinzesberg.de> Hi folks, I am running Rsyslog on RedHat/CentOS and Debian/Ubuntu for quite a while now and am happy with it. Now I would like to test it on NetBSD 5, but I do not find anything on that out there on the web. NetBSD themselves do not offer it in package source, and there is no report on that on the web. Anyone on the list using rsyslog on NetBSD (5)? Any hints or help? Or is it just compile and run? Thanks in advance for any help or link to related information. Dirk From rgerhards at hq.adiscon.com Thu Jul 2 15:29:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 15:29:22 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Hi all, with the next released of rsyslog v4/v5, I will introduce so-called "rulesets". They are effectively a sequence of rules that can directly be bound to an input (as long as the input supports this feature, currently not all do). I hope that this mechanism will simplify configuration. It will also enhance performance in most cases (as less filter logic is involved). A howto with detail description is available here: http://www.rsyslog.com/doc-multi_ruleset.html Rainer From tbergfeld at hq.adiscon.com Thu Jul 2 16:11:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 2 Jul 2009 16:11:15 +0200 Subject: [rsyslog] rsyslog 3.22.1 (v3-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACB@GRFEXC.intern.adiscon.com> Hi all, rsyslog 3.22.1, a member of the v3-stable branch, has been released today. The release mainly consists of bugfixes like a fix for an invalid error message issued if $inlcudeConfig was on an empty set of files. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-163.phtml Changelog: http://www.rsyslog.com/Article381.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From aoz.syn at gmail.com Thu Jul 2 17:13:35 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 2 Jul 2009 09:13:35 -0600 Subject: [rsyslog] v4/v5: new mult-ruleset support In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > I hope that this mechanism will simplify configuration. It will also enhance > performance in most cases (as less filter logic is involved). A howto with > detail description is available here: Very pleasing; this is an excellent step toward some of the enhanced configuration semantics I miss from syslog-ng. Perhaps I'm alone here, but I feel that the current syntax is detrimental to complex configurations, largely due to [understandable] compatibility adherence. Oh, to be a college student again and able to take on a SoC project to write a secondary configuration parser. From rgerhards at hq.adiscon.com Thu Jul 2 17:17:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 17:17:19 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Thursday, July 02, 2009 5:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] v4/v5: new mult-ruleset support > > On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > > I hope that this mechanism will simplify configuration. It will also enhance > > performance in most cases (as less filter logic is involved). A howto with > > detail description is available here: > > Very pleasing; this is an excellent step toward some of the enhanced > configuration semantics I miss from syslog-ng. Yeah, I started with the goal of full compatibility, but some complex things really get hard in that mode. > Perhaps I'm alone > here, but I feel that the current syntax is detrimental to complex > configurations, me, too ;) >largely due to [understandable] compatibility > adherence. Oh, to be a college student again and able to take on a > SoC project to write a secondary configuration parser. The script engine should bring this ... when I am finally able to go to it (doesn't look like I am anytime soon...). Rainer From tbergfeld at hq.adiscon.com Fri Jul 3 11:38:04 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 3 Jul 2009 11:38:04 +0200 Subject: [rsyslog] rsyslog 4.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAE8@GRFEXC.intern.adiscon.com> Hi all, Today, we have released rsyslog 4.5.0, a member of the development branch. This version offers improved performance and reduced memory requirements of the msg object. It further provides you with better config error messages, added capability to fsync() queue disc files for enhanced reliability, new configuration commands and stricter parsing of the hostname in rfc3164 mode. There are also some bug fixes included and the max value for $DynaFileCacheSize has been reduced to a more suitable value. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-164.phtml Changelog: http://www.rsyslog.com/Article380.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Mon Jul 6 14:03:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Jul 2009 14:03:23 +0200 Subject: [rsyslog] git branch changes Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB08@GRFEXC.intern.adiscon.com> Hi all, I made a mistake last Friday when creating the new releases. I accidently merged the v5-devel branch into the master branch, which was v4-devel up to then. I now looked at the result and think it is best to drop the v5-devel and create a new v4-devel branch (I can identify the wrong merge and will base v4-devel on it). This hopefully prevents any other such mistake in the future... Somehow my mind is on "master is always the most current". So please use v4-devel for the v4 tree and master for the v5-tree. Once v5 has become / confirmed to be sufficiently stable, I'll stop development in v4 and we will go back to the stable/beta/devel versions with only one active development version. I am right now preparing the new branches, will send a short mail when I am done. I just thought I let you know asap that something went wrong. Sorry for the hassle, Rainer From tbergfeld at hq.adiscon.com Wed Jul 8 16:48:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 8 Jul 2009 16:48:09 +0200 Subject: [rsyslog] rsyslog 5.1.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB43@GRFEXC.intern.adiscon.com> Hi all, We have released rsyslog 5.1.2, a new version of the development branch. This version offers improved performance and includes important fixes like a fix for an abort condition when RecvFrom was not set and message reduction was on, this happened e.g. with imuxsock. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-166.phtml Changelog: http://www.rsyslog.com/Article386.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Thu Jul 9 17:56:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Jul 2009 17:56:36 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Sorry folks, have not worked on this topic since March - but I never forget ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, March 04, 2009 8:17 AM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Ok, here is a diff that works. > I have integrated David's UDP spoofing patch into v5-devel (master branch) as a separate output module (named omudpspoof). I have extracted the necessary functionality from omfwd and it now works quite well. However, the configuration is not yet as typically found in rsyslog (not that I find it very elegant, but consistency is a plus ;)). Also, doc is missing. My next steps will be to address these issues. However, for those interested, I just wanted to tell you that the git master branch now contains a version capable of spoofing (I have verified this with both an rsyslog receiver as well as Wireshark - hopefully I can also add an automated test for this functionality, but this is not trivial). > it cycles the source IP address from 32000-42000 (since we are just > sending, and not creating a normal socket this should not matter) > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > to use it create a template that puts the hostname-ip ahead of what you > want to send, similar to > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > %HOSTNAME% %syslogtag%%msg%\n" > > *.* @10.0.0.100;TraditionalFwdFormat > > the one problem right now is that any logs sent from the local box will go > out with a source IP of 127.0.0.1 > > I wasted a bit of time trying to setup filters to use a different template > if $myhostname == $fromhost, but apparently the filtering doesn't allow > comparing two properties I overlooked this in march. The script-based filter engine does not even know that what is being compared are properties, so there is no restriction in what can be compared. So this must either have been a config issue or a bug. I just wanted to tell you should you come into a similar situation again. >, and then I realized that you have a very > high-performance name cache now, so you could easily replace my trivial > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > line with a call to the name lookup and then the %fromhost-ip% could be > replaced by %fromhost% in the template and everything would work sanely > (assuming forward and reverse name resolution are sane ;-) And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > the IP layer options, but I don't know exactly what yet. Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand shows up. Even then, I consider it to be low priority except given very good reasoning ;) Rainer > > I wanted to float this first to see what you think before spending much > more time on it. > > David Lang From doherty at crystal.harvard.edu Thu Jul 9 23:05:30 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Thu, 9 Jul 2009 17:05:30 -0400 Subject: [rsyslog] High CPU utlization, and memory usage Message-ID: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Hi, I recently deployed rsyslog on a server. I'm using a central server that receives logs from currently just 4 other machines. It's using TLS. It's a pretty basic setup, I didn't do anything fancy. Twice this week the central server has had issues, and when I logged in to check, I saw that rsyslog was using all the swap (2GB) and using over 300MB of the 512MB RAM, plus using 70% or more CPU time. I'm running version 4.2.0 Can you give me some idea where to start looking for what's causing this? It seems to run fine for a day or so, and then over a very short amount of time, maybe 2-60 minutes, the memory and cpu usage spikes. Here's a snippet from the /etc/rsyslog.conf if that helps. Thanks. --Peter $ModLoad immark $ModLoad imklog $ModLoad imuxsock # local messages $ModLoad imtcp # TCP listener $MarkMessagePeriod 1200 # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerRun 10514 # start up listener at port 10514 From david at lang.hm Thu Jul 9 23:35:30 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 9 Jul 2009 14:35:30 -0700 (PDT) Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: On Thu, 9 Jul 2009, Peter Doherty wrote: > Hi, I recently deployed rsyslog on a server. I'm using a central > server that receives logs from currently just 4 other machines. It's > using TLS. It's a pretty basic setup, I didn't do anything fancy. > > Twice this week the central server has had issues, and when I logged > in to check, I saw that rsyslog was using all the swap (2GB) and using > over 300MB of the 512MB RAM, plus using 70% or more CPU time. one thing when examining the load is to keep in mind that rsyslog uses multiple threads, in linux hit 'H' in top to have it show the individual threads. it's very possible that the high cpu load is due to the encryption overhead one thing that I do when testing is that once I identify which thread is eating a significant amount of cpu I do 'strace -p ' for that thread for a few seconds, looking at that output I can ususally make a fair guess at what the thread is busy working on. using that much ram would leave me guessing that the ability to write the log file stopped, and the queue is filling up. David Lang > I'm running version 4.2.0 > Can you give me some idea where to start looking for what's causing > this? It seems to run fine for a day or so, and then over a very > short amount of time, maybe 2-60 minutes, the memory and cpu usage > spikes. > > Here's a snippet from the /etc/rsyslog.conf if that helps. > > Thanks. > --Peter > > > $ModLoad immark > $ModLoad imklog > $ModLoad imuxsock # local messages > $ModLoad imtcp # TCP listener > > $MarkMessagePeriod 1200 > > # make gtls driver the default > $DefaultNetstreamDriver gtls > > # certificate files > $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem > $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem > $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem > > > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > $InputTCPServerRun 10514 # start up listener at port 10514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Jul 10 14:07:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 10 Jul 2009 14:07:24 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB66@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have completed this module now. It is part of the regular master branch. Some basic doc is available here: http://www.rsyslog.com/doc-omudpspoof.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, July 09, 2009 5:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Sorry folks, have not worked on this topic since March - but I never forget > ;) > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, March 04, 2009 8:17 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] UDP source forging. > > > > Ok, here is a diff that works. > > > > I have integrated David's UDP spoofing patch into v5-devel (master branch) as > a separate output module (named omudpspoof). I have extracted the necessary > functionality from omfwd and it now works quite well. However, the > configuration is not yet as typically found in rsyslog (not that I find it > very elegant, but consistency is a plus ;)). Also, doc is missing. My next > steps will be to address these issues. However, for those interested, I just > wanted to tell you that the git master branch now contains a version capable > of spoofing (I have verified this with both an rsyslog receiver as well as > Wireshark - hopefully I can also add an automated test for this > functionality, but this is not trivial). > > > it cycles the source IP address from 32000-42000 (since we are just > > sending, and not creating a normal socket this should not matter) > > > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > > > to use it create a template that puts the hostname-ip ahead of what you > > want to send, similar to > > > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > > %HOSTNAME% %syslogtag%%msg%\n" > > > > *.* @10.0.0.100;TraditionalFwdFormat > > > > the one problem right now is that any logs sent from the local box will go > > out with a source IP of 127.0.0.1 > > > > I wasted a bit of time trying to setup filters to use a different template > > if $myhostname == $fromhost, but apparently the filtering doesn't allow > > comparing two properties > > I overlooked this in march. The script-based filter engine does not even know > that what is being compared are properties, so there is no restriction in > what can be compared. So this must either have been a config issue or a bug. > I just wanted to tell you should you come into a similar situation again. > > >, and then I realized that you have a very > > high-performance name cache now, so you could easily replace my trivial > > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > > line with a call to the name lookup and then the %fromhost-ip% could be > > replaced by %fromhost% in the template and everything would work sanely > > (assuming forward and reverse name resolution are sane ;-) > > And another point to stress: rsyslogd does *not* yet have a high-performance > cache. All it does is cache the last host (and only the last host). This > works exceptionally well if a large bunch of messages arrive from the same > host, but "cache" performance can easily be thrashed with multiple senders. > All in all, in practice, it works reasonably well, as on a busy system at > least a couple of messages are usually from the same sender. I plan to add a > real cache some time later, personally I would hope to see it this summer. > > > > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > > the IP layer options, but I don't know exactly what yet. > > Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand > shows up. Even then, I consider it to be low priority except given very good > reasoning ;) > > Rainer > > > > > I wanted to float this first to see what you think before spending much > > more time on it. > > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From doherty at crystal.harvard.edu Fri Jul 10 16:04:27 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Fri, 10 Jul 2009 10:04:27 -0400 Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: <47B915CC-ACD1-496C-B91F-597EC84831E8@crystal.harvard.edu> On Jul 9, 2009, at 5:35 PM, david at lang.hm wrote: > > one thing when examining the load is to keep in mind that rsyslog uses > multiple threads, in linux hit 'H' in top to have it show the > individual > threads. > > it's very possible that the high cpu load is due to the encryption > overhead > > one thing that I do when testing is that once I identify which > thread is > eating a significant amount of cpu I do 'strace -p ' for that > thread > for a few seconds, looking at that output I can ususally make a fair > guess > at what the thread is busy working on. > > using that much ram would leave me guessing that the ability to > write the > log file stopped, and the queue is filling up. > > David Lang I'll have to do a little further testing, but I've got a hunch what caused this state. One of the computers sometimes gets into a state where it starts flooding its syslog with errors from a program. On the order of hundreds of messages a minute. I can correct this one case, but I can't guarantee some other system won't ever have some error that causes it to start flooding it's logs. Are there settings in rsyslog that can control this? Essentially I need something that will prevent a DoS style attack. Something that drops messages from the queue if they come in too fast from a certain host? Or I often see messages from syslogd systems which just say "Last message repeated n times" Can I enable that functionality in rsyslog? --Peter From ko at ibl.fr Fri Jul 10 17:04:25 2009 From: ko at ibl.fr (ko) Date: Fri, 10 Jul 2009 17:04:25 +0200 Subject: [rsyslog] One file per host logging Message-ID: <4A575879.3030308@ibl.fr> Hi, I'm a new-by in rsyslog. I want to do a centralized syslog server, and I want to know if it's possible to do a per host logging, I mean : all log comming from 192.168.1.1 going to /var/log/server1.log all log comming from 192.168.1.2 going to /var/log/server2.log ... and so on... Thanks for your help K. From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 10 17:17:53 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 10 Jul 2009 17:17:53 +0200 Subject: [rsyslog] One file per host logging In-Reply-To: <4A575879.3030308@ibl.fr> References: <4A575879.3030308@ibl.fr> Message-ID: <200907101717.56354.Luis.Fernando.Munoz.Mejias@cern.ch> Ko, > > I'm a new-by in rsyslog. I want to do a centralized syslog server, and > I want to know if it's possible to do a per host logging, I mean : Indeed you can. Just specify the file name as a template. For instance: $template FileNamePerHost,"/var/log/%HOSTNAME%.log" And then, store everything on the files specified by the template: *.* FileNamePerHost You can even have folders per day, just review the documentation on templates and properties. Cheers -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Mon Jul 13 14:17:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 14:17:04 +0200 Subject: [rsyslog] rsyslog - what's next? Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Hi all, I think we made some really good progress with rsyslog's capabilities (feature set & performance) in the past couple of month. I also think I have reached a milestone. As such, I've taken a bit time off to think about where to head to next. And as I'd like to have this available as a handy reference, I've blogged about. So if you are interested in what I intend to focus on the next time (maybe two to three month), please have a look at my blog post: http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html Of course, urgent things will have priority over the general goal, but I am pretty sure the planned effort will have a very positive effect on rsyslog's capabilities and qualities even in the medium term. As always, comments and questions are welcome. Thanks, Rainer From david at lang.hm Mon Jul 13 19:09:32 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 10:09:32 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 13 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I think we made some really good progress with rsyslog's capabilities > (feature set & performance) in the past couple of month. I also think I have > reached a milestone. As such, I've taken a bit time off to think about where > to head to next. And as I'd like to have this available as a handy reference, > I've blogged about. > > So if you are interested in what I intend to focus on the next time (maybe > two to three month), please have a look at my blog post: > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > Of course, urgent things will have priority over the general goal, but I am > pretty sure the planned effort will have a very positive effect on rsyslog's > capabilities and qualities even in the medium term. > > As always, comments and questions are welcome. I don't think that you need to do much for UDP recieve performance. as long as it doesn't need to do DNS lookups it can recieve and insert into a memory queue at full gig-E speeds. the only think you may want to do here is to extend the DNS cache so that instead of only caching the last think you looked up, you cache everything until a restart or HUP (ideally the HUP should be a configurable option) the machines I had on order with the 10G interfaces were misconfigured by the vendor, so I don't have the 10G cards yet :-( when I get them I'll do more testing and see how much faster I can push rsyslog. David Lang From aoz.syn at gmail.com Mon Jul 13 19:38:17 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 11:38:17 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> On Mon, Jul 13, 2009 at 11:09, wrote: > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) I haven't looked at the DNS cache code, but unless you're also caching and handling the records' TTL, blindly caching DNS records within the app is a dark road to go down. Some apps (namely browsers) do it anyway, but get away with it by setting their internal TTL significantly lower than that of a typical record. What kind of performance hit are you actually seeing with DNS lookups? From david at lang.hm Mon Jul 13 20:11:02 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 11:11:02 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 11:09, wrote: >> is to extend the DNS cache so that instead of only caching the last think >> you looked up, you cache everything until a restart or HUP (ideally the >> HUP should be a configurable option) > > I haven't looked at the DNS cache code, but unless you're also caching > and handling the records' TTL, blindly caching DNS records within the > app is a dark road to go down. Some apps (namely browsers) do it > anyway, but get away with it by setting their internal TTL > significantly lower than that of a typical record. for Internet access I completely agree with you, but for a log server you don't have IPs changing names very frequently. As a result it becomes practical to cache the lookup unconditionally until a restart/HUP. even in a 'highly dynamic' environment you are usually adding servers instead of changing the names of IP addresses. note that caching the lookup at all can be disabled via a config option > What kind of performance hit are you actually seeing with DNS lookups? 6 months ago it was a factor of 10x to 100x (it's probably more now since rsyslog has gotten faster) the problem is that to do a name lookup requires a LOT of steps first the name resolver library looks several places on the filesystem for various config files then it reads /etc/hosts, parses it, and looks for the name/IP then it makes a network connection to the nameserver, sends a message, waits for the reciever to parse the message and look it up in it's datastore, then send the message back to the requester who needs to parse the message and if you network packet gets dropped due to congestion on the network, you wait 30 seconds and retry. doing all of this for every UDP syslog packet that you recieve results in a _lot_ of system calls, and a lot of delays. even if you have the name in the /etc/hosts file, the overhead of looking in the filesystem and parsing the file is significant. without doing DNS lookups, rsyslog is able to recieve UDP logs at almost 400,000 per second (effectivly Gig-E wire speed with 256 byte log messages), _very_ few DNS servers can handle requests at that speed. in fact, at that request rate, you use about half of a Gig-E connection just for the DNS requests (more if you request more data, like what the TTL would be, or don't give it the best name the first time and need to make multiple requests with different domains attached or something like that) David Lang From aoz.syn at gmail.com Mon Jul 13 21:43:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 13:43:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> On Mon, Jul 13, 2009 at 12:11, wrote: > for Internet access I completely agree with you, but for a log server you > don't have IPs changing names very frequently. As a result it becomes > practical to cache the lookup unconditionally until a restart/HUP. even in > a 'highly dynamic' environment you are usually adding servers instead of > changing the names of IP addresses. The problem with that approach is that it assumes it is acceptable to interrupt service for any trivial DNS change. How do you propose to deal withDNS-level availability tools like round-robin or Cisco's GSS? There are many approaches that use DNS as both a scalability and a redundancy tool that TTL-agnostic caching will not interact well with. > the problem is that to do a name lookup requires a LOT of steps I'm well aware of the series of steps, but am unconvinced caching belongs in the application. Have you tried using local caching (nscd) or segregating the DNS traffic from the production syslog traffic? Most enterprise-grade configurations (for whatever app) I've seen use a separate interface for administration and metadata like reverse lookups anyway. > even if you have the name in the /etc/hosts file, the overhead of looking > in the filesystem and parsing the file is significant. How significant? Unless the host is poorly configured, /etc/hosts is going to be in the filesystem cache and presented at near-memory speeds until it gets invalidated or evicted. > without doing DNS lookups, rsyslog is able to recieve UDP logs at almost > 400,000 per second (effectivly Gig-E wire speed with 256 byte log > messages), _very_ few DNS servers can handle requests at that speed. in To clarify since I'm not looking at the code right now: are you saying rsyslog performs a blocking lookup per-packet? If so that's certainly a sub-optimal approach, but there are often better ways (namely delayed resolution) to fix that than breaking RFC compatibility. > fact, at that request rate, you use about half of a Gig-E connection just > for the DNS requests (more if you request more data, like what the TTL > would be, or don't give it the best name the first time and need to make > multiple requests with different domains attached or something like that) Your example invalidates itself - the TTL comes with the query whether or not you use it, and any caching mechanism (TTL-sensitive or not) will shortcut such sub-optimal queries after the first one. Caching is a good thing as long as it's done in a compliant manner or documented in a fashion that clearly identifies it as a broken (if performant) approach. From ktm at rice.edu Mon Jul 13 21:37:57 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 13 Jul 2009 14:37:57 -0500 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <20090713193757.GB4988@it.is.rice.edu> On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I think we made some really good progress with rsyslog's capabilities > > (feature set & performance) in the past couple of month. I also think I have > > reached a milestone. As such, I've taken a bit time off to think about where > > to head to next. And as I'd like to have this available as a handy reference, > > I've blogged about. > > > > So if you are interested in what I intend to focus on the next time (maybe > > two to three month), please have a look at my blog post: > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > Of course, urgent things will have priority over the general goal, but I am > > pretty sure the planned effort will have a very positive effect on rsyslog's > > capabilities and qualities even in the medium term. > > > > As always, comments and questions are welcome. > > I don't think that you need to do much for UDP recieve performance. as > long as it doesn't need to do DNS lookups it can recieve and insert into a > memory queue at full gig-E speeds. the only think you may want to do here > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) > > the machines I had on order with the 10G interfaces were misconfigured by > the vendor, so I don't have the 10G cards yet :-( when I get them I'll do > more testing and see how much faster I can push rsyslog. > > David Lang I agree with David. DNS lookups can really sap both bandwidth and CPU. I am not sure that caching everything since a restart is a good idea though. Even if you could do a lookup, cache the TTL and IP and then relookup after the TTL that would be great. Even having it re-query the DNS in a worker thread to sweep through the cached values periodically would really help performance. Regards, Ken From rgerhards at hq.adiscon.com Mon Jul 13 22:25:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 22:25:07 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <20090713193757.GB4988@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB80@GRFEXC.intern.adiscon.com> One important point to stress is that rsyslog does NOT yet have a real DNS cache! Let me quote my mail from July, 9th[1]: == And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. == Adding a the cache is not really a focus activity - I think it could be done in a couple of days. I also personally do not see an issue with obying TTL - or discarding entries every five minutes or so. In my POV, it doesn't really matter if they are quickly discarded. If there is low traffic, it doesn't make a difference at all. If we handle hundereds of thousands message per second, one lockup every few billion of messages doesn't really matter either. The compare operation is relatively fast and will not add significantly to the runtime (at least I'd guess that). In regard to imudp, I think there is a couple of more optimizations possible. I'd expect that the biggest hit (after DNS lookup) is switching from select to epoll. Re-using property copying is another one (reduces expensive memory writes). A batching interface seems also possible. Enhanced pipelining is another possibility. I also think that the current solution will gain very different results based on the communication patterns. So rather than (educated) guessing, I would prefer to have some good instrumentation to get a grip on all this variants. Also, imudp is just a sample, there are more areas that could benefit from a focussed analytic view. Plus that performance tools can be used to stress-testing and thus finding otherwise hard to find bugs. Thus I think it is worth spending some time on an improved toolset (and will pay off in the long term). Rainer [1] http://lists.adiscon.net/pipermail/rsyslog/2009-July/002432.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Monday, July 13, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > > > Hi all, > > > > > > I think we made some really good progress with rsyslog's > capabilities > > > (feature set & performance) in the past couple of month. > I also think I have > > > reached a milestone. As such, I've taken a bit time off > to think about where > > > to head to next. And as I'd like to have this available > as a handy reference, > > > I've blogged about. > > > > > > So if you are interested in what I intend to focus on the > next time (maybe > > > two to three month), please have a look at my blog post: > > > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > > > Of course, urgent things will have priority over the > general goal, but I am > > > pretty sure the planned effort will have a very positive > effect on rsyslog's > > > capabilities and qualities even in the medium term. > > > > > > As always, comments and questions are welcome. > > > > I don't think that you need to do much for UDP recieve > performance. as > > long as it doesn't need to do DNS lookups it can recieve > and insert into a > > memory queue at full gig-E speeds. the only think you may > want to do here > > is to extend the DNS cache so that instead of only caching > the last think > > you looked up, you cache everything until a restart or HUP > (ideally the > > HUP should be a configurable option) > > > > the machines I had on order with the 10G interfaces were > misconfigured by > > the vendor, so I don't have the 10G cards yet :-( when I > get them I'll do > > more testing and see how much faster I can push rsyslog. > > > > David Lang > > I agree with David. DNS lookups can really sap both bandwidth and CPU. > I am not sure that caching everything since a restart is a good idea > though. Even if you could do a lookup, cache the TTL and IP and then > relookup after the TTL that would be great. Even having it re-query > the DNS in a worker thread to sweep through the cached values > periodically > would really help performance. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Jul 13 22:38:52 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 13:38:52 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 12:11, wrote: >> for Internet access I completely agree with you, but for a log server you >> don't have IPs changing names very frequently. As a result it becomes >> practical to cache the lookup unconditionally until a restart/HUP. even in >> a 'highly dynamic' environment you are usually adding servers instead of >> changing the names of IP addresses. > > The problem with that approach is that it assumes it is acceptable to > interrupt service for any trivial DNS change. How do you propose to > deal withDNS-level availability tools like round-robin or Cisco's GSS? > There are many approaches that use DNS as both a scalability and a > redundancy tool that TTL-agnostic caching will not interact well with. HUP does not interrupt services. >> the problem is that to do a name lookup requires a LOT of steps > > I'm well aware of the series of steps, but am unconvinced caching > belongs in the application. Have you tried using local caching (nscd) > or segregating the DNS traffic from the production syslog traffic? > Most enterprise-grade configurations (for whatever app) I've seen use > a separate interface for administration and metadata like reverse > lookups anyway. yes, doing a local DNS cacheing server does not avoid all the local filesystem lookups and host file parsing. it does speed up the network connection. >> even if you have the name in the /etc/hosts file, the overhead of looking >> in the filesystem and parsing the file is significant. > > How significant? Unless the host is poorly configured, /etc/hosts is > going to be in the filesystem cache and presented at near-memory > speeds until it gets invalidated or evicted. the system calls to access it (and to first lookup if it should be accessed at all) take enough time to be significant at these speeds. >> without doing DNS lookups, rsyslog is able to recieve UDP logs at almost >> 400,000 per second (effectivly Gig-E wire speed with 256 byte log >> messages), _very_ few DNS servers can handle requests at that speed. in > > To clarify since I'm not looking at the code right now: are you saying > rsyslog performs a blocking lookup per-packet? If so that's certainly > a sub-optimal approach, but there are often better ways (namely > delayed resolution) to fix that than breaking RFC compatibility. correct, it does all input processing in a single thread. so if that thread is stalled doing a lookup it's not processing other packets. when the data is first inserted into the message queue it is complete with all lookups done. >> fact, at that request rate, you use about half of a Gig-E connection just >> for the DNS requests (more if you request more data, like what the TTL >> would be, or don't give it the best name the first time and need to make >> multiple requests with different domains attached or something like that) > > Your example invalidates itself - the TTL comes with the query whether > or not you use it, and any caching mechanism (TTL-sensitive or not) > will shortcut such sub-optimal queries after the first one. > > Caching is a good thing as long as it's done in a compliant manner or > documented in a fashion that clearly identifies it as a broken (if > performant) approach. you want the ability to cache the name lookup no matter what method is used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include a TTL. I agree that the details of how the cache works should be documented. David Lang From aoz.syn at gmail.com Mon Jul 13 23:24:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 15:24:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> On Mon, Jul 13, 2009 at 14:38, wrote: > HUP does not interrupt services. I'm sorry; last October you noted that for certain configurations SIGHUP will drop memory-queued messages. In addition to the other notes in the thread (tearing down TCP connections, etc.), it sounds an awful lot like a service interruption. Has that since changed for 3.x or 4.2? > the system calls to access it (and to first lookup if it should be > accessed at all) take enough time to be significant at these speeds. May I gently prod you to substantiate this statement? I don't doubt that it makes calls to external libraries, and that those calls are likely more expensive than internal resolution, but "significant" is not significant without numbers to back it up. Even a simple speed comparison for a few million packets between 'no resolution' and 'in /etc/hosts with a cold cache' would be useful. > you want the ability to cache the name lookup no matter what method is > used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > a TTL. I have no problem expanding the scope of discussion to resolution methods beyond DNS (which was the original premise), but if a name service does not provide an explicit TTL, it must be assumed as an implicit '0', which blind caching will break. That said, each of those methods still provides some [internal] method of caching and invalidation that, to be audit-grade correct, rsyslog will either have to replicate or trust. I still can't see a problem with having something to the effect of a "$BreakNSCache" configuration option, but by default a syslog daemon should play as strictly by the rules as possible. Then again, I'm RB, not RG. ;) From david at lang.hm Mon Jul 13 23:58:33 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 14:58:33 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 14:38, wrote: >> HUP does not interrupt services. > > I'm sorry; last October you noted that for certain configurations > SIGHUP will drop memory-queued messages. In addition to the other > notes in the thread (tearing down TCP connections, etc.), it sounds an > awful lot like a service interruption. Has that since changed for 3.x > or 4.2? yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not do a complete teardown, halt, and restart, but instead just closes and re-opens files and connections so that no long messages are lost in a restart. this matches the traditional use of HUP to get syslogd to release the files it's writing to so that they can be rotated away. it doesn't re-read the config file due to the fact that the rsyslog config file is so complex and can significantly alter the software by loading modules. >> the system calls to access it (and to first lookup if it should be >> accessed at all) take enough time to be significant at these speeds. > > May I gently prod you to substantiate this statement? I don't doubt > that it makes calls to external libraries, and that those calls are > likely more expensive than internal resolution, but "significant" is > not significant without numbers to back it up. Even a simple speed > comparison for a few million packets between 'no resolution' and 'in > /etc/hosts with a cold cache' would be useful. I would have to do a new set of tests to get you concrete numbers, but back in roughly the october timeframe last year I was doing extensive tests on this and with hosts files vs. no resolution I was seeing 4x or more differences (with a tiny, 5-line hosts file to parse) at the time I think I discussed it on a long performance thread on the website. at the time I did quite a number of strace dumps to show how much time was being eaten up in the system calls. RG has done a LOT of cleanup since then (to the point that the failsafe audit mode is in the ballpark of being as fast as the in-memory mode was back then) >> you want the ability to cache the name lookup no matter what method is >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include >> a TTL. > > I have no problem expanding the scope of discussion to resolution > methods beyond DNS (which was the original premise), but if a name > service does not provide an explicit TTL, it must be assumed as an > implicit '0', which blind caching will break. That said, each of > those methods still provides some [internal] method of caching and > invalidation that, to be audit-grade correct, rsyslog will either have > to replicate or trust. I still can't see a problem with having > something to the effect of a "$BreakNSCache" configuration option, but > by default a syslog daemon should play as strictly by the rules as > possible. well, if you are really wanting audit-grade logging, will you use anything other than the IP address (or what is already in the message)? any lookup that you do is a potential for corruption. I'm fine with the default being to do a lookup for each item. I just want a way to avoid it, and if I can do that with a cache instead of having to disable DNS, I will do so. it's very common for servers to need to disable DNS lookups for their logs. do a quick search for apache dns performance and you will find that it's a very common problem people have there if they enable lookups on the sender's IP address. > Then again, I'm RB, not RG. ;) more people speaking up is good. I have strong opinions, and real problems to address, but RG needs to hear from people other than me. David Lang From aoz.syn at gmail.com Tue Jul 14 05:12:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 21:12:57 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> On Mon, Jul 13, 2009 at 15:58, wrote: > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. Hopefully some of the tooling Rainer is considering will better enable performance monitoring and we can test the effect of some of these suggested changes. I don't completely discount strace as a performance testing tool, but it's inherently intrusive, making Heisenberg quite happy. > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address Indeed; I've generally eschewed DNS for such critical pieces of infrastructure, instead opting for resolution during analysis and correlation with other logs. That largely stems from network security work and the associated disdain for using DNS entries for rules. Even so, I'm curious whether a properly tuned library-integrated cache like nscd (as opposed to a local caching resolver) would provide a sufficient performance increase in the interim. I spent a couple of hours this evening digging through the related code with the intent of starting work on delayed resolution, moving the lookup to runtime/msg.c:getHOSTNAME. It might not be the right place, but would have the advantage of only resolving hosts you explicitly request it for. Unfortunately, the AllowedSender pragma is a sticking point, forcing per-packet resolution; otherwise, it seemed a pretty trivial conversion. I don't know how much (if any) performance increase that approach would grant, but it would at least move a blocking call out of that single thread. I may experiment with adding a "NoAllowedSenderDNS" later, but it's beyond my time flexibility right now. From david at lang.hm Tue Jul 14 06:54:37 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 21:54:37 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 15:58, wrote: >> at the time I did quite a number of strace dumps to show how much time was >> being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. absolutly, especially when you enable timing of the calls !!! (a gettimeofday call before and after evey syscall) but if the program hasn't been examined to try and reduce the number of syscalls it makes, strace is a good place to start. once you get down to a minimum of nessasary syscalls, strace stops being useful. >> well, if you are really wanting audit-grade logging, will you use anything >> other than the IP address (or what is already in the message)? any lookup >> that you do is a potential for corruption. >> >> I'm fine with the default being to do a lookup for each item. I just want >> a way to avoid it, and if I can do that with a cache instead of having to >> disable DNS, I will do so. >> >> it's very common for servers to need to disable DNS lookups for their >> logs. do a quick search for apache dns performance and you will find that >> it's a very common problem people have there if they enable lookups on the >> sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. I'll try and get a chance to look into that > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. interesting thought. David Lang From rgerhards at hq.adiscon.com Tue Jul 14 07:17:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 07:17:24 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB82@GRFEXC.intern.adiscon.com> I am digesting this thread (just got up ;)), but let me spell out something explicitely: the traditional perception of what is fast and what is slow is no longer valid if you aim at a rate of several hunderd-thousand messages per second. For example, it then makes a difference if you (memory) write a single 32 byte buffer per record or not. It may be in the region of 100 to 500 records per second, but these differences sum up. So trying to avoid such writes, even a the costs of (currently much cheaper) reads is something to aim at. I'll see that I get together a more elaborate description inside my blog, probably it is useful. I learned a lot of lessons the past nine month ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, July 14, 2009 6:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 15:58, wrote: > >> at the time I did quite a number of strace dumps to show > how much time was > >> being eaten up in the system calls. > > > > Hopefully some of the tooling Rainer is considering will > better enable > > performance monitoring and we can test the effect of some of these > > suggested changes. I don't completely discount strace as a > > performance testing tool, but it's inherently intrusive, making > > Heisenberg quite happy. > > absolutly, especially when you enable timing of the calls !!! (a > gettimeofday call before and after evey syscall) > > but if the program hasn't been examined to try and reduce the > number of > syscalls it makes, strace is a good place to start. > > once you get down to a minimum of nessasary syscalls, strace > stops being > useful. > > >> well, if you are really wanting audit-grade logging, will > you use anything > >> other than the IP address (or what is already in the > message)? any lookup > >> that you do is a potential for corruption. > >> > >> I'm fine with the default being to do a lookup for each > item. I just want > >> a way to avoid it, and if I can do that with a cache > instead of having to > >> disable DNS, I will do so. > >> > >> it's very common for servers to need to disable DNS > lookups for their > >> logs. do a quick search for apache dns performance and you > will find that > >> it's a very common problem people have there if they > enable lookups on the > >> sender's IP address > > > > Indeed; I've generally eschewed DNS for such critical pieces of > > infrastructure, instead opting for resolution during analysis and > > correlation with other logs. That largely stems from > network security > > work and the associated disdain for using DNS entries for > rules. Even > > so, I'm curious whether a properly tuned library-integrated > cache like > > nscd (as opposed to a local caching resolver) would provide a > > sufficient performance increase in the interim. > > I'll try and get a chance to look into that > > > I spent a couple of hours this evening digging through the related > > code with the intent of starting work on delayed resolution, moving > > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > > place, but would have the advantage of only resolving hosts you > > explicitly request it for. Unfortunately, the > AllowedSender pragma is > > a sticking point, forcing per-packet resolution; otherwise, > it seemed > > a pretty trivial conversion. I don't know how much (if any) > > performance increase that approach would grant, but it > would at least > > move a blocking call out of that single thread. I may > experiment with > > adding a "NoAllowedSenderDNS" later, but it's beyond my time > > flexibility right now. > > interesting thought. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 10:37:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 10:37:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, July 13, 2009 11:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 14:38, wrote: > >> HUP does not interrupt services. > > > > I'm sorry; last October you noted that for certain configurations > > SIGHUP will drop memory-queued messages. In addition to the other > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > awful lot like a service interruption. Has that since changed for 3.x > > or 4.2? > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not > do a complete teardown, halt, and restart, but instead just closes and > re-opens files and connections so that no long messages are lost in a > restart. And that is the default setting! > > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. For v5, I actually think of adding a "rsyslogd restart deamon" for those that insist on traditional HUP semantics. That deamon would lie dormant in memory until it receives HUP, in which case it does a shutdown and restart of the real rsyslogd. With that, I could cleanup a lot of complexity. One major issue is that under some circumstances I need to cancel output threads when they do not finish their processing within the allotted timeouts. Canceling threads is always a bit dangerous, but there currently is no always-reliable way around this. To make it happen, a lot of enable/disable cancel calls, including cancel cleanup handlers are made throughout the code. If we would not have restart-type HUPs, I could simply cancel those threads and not care about resources not being cleaned up (the process cleanup will take care of that). So I could also remove all the enable/disable cancel logic and its helpers. Of course, this is not something done as quickly as I write those lines and it must be made sure that any inconsistence does not affect the rest of the shutdown. But without those annoying restart-type HUPs, it is much simpler to do... > > >> the system calls to access it (and to first lookup if it should be > >> accessed at all) take enough time to be significant at these speeds. > > > > May I gently prod you to substantiate this statement? I don't doubt > > that it makes calls to external libraries, and that those calls are > > likely more expensive than internal resolution, but "significant" is > > not significant without numbers to back it up. Even a simple speed > > comparison for a few million packets between 'no resolution' and 'in > > /etc/hosts with a cold cache' would be useful. > > I would have to do a new set of tests to get you concrete numbers, but > back in roughly the october timeframe last year I was doing extensive > tests on this and with hosts files vs. no resolution I was seeing 4x or > more differences (with a tiny, 5-line hosts file to parse) > > at the time I think I discussed it on a long performance thread on the > website. > > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. RG has done a LOT of cleanup since > then (to the point that the failsafe audit mode is in the ballpark of > being as fast as the in-memory mode was back then) > > >> you want the ability to cache the name lookup no matter what method is > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > >> a TTL. > > > > I have no problem expanding the scope of discussion to resolution > > methods beyond DNS (which was the original premise), but if a name > > service does not provide an explicit TTL, it must be assumed as an > > implicit '0', which blind caching will break. That said, each of > > those methods still provides some [internal] method of caching and > > invalidation that, to be audit-grade correct, rsyslog will either have > > to replicate or trust. I still can't see a problem with having > > something to the effect of a "$BreakNSCache" configuration option, but > > by default a syslog daemon should play as strictly by the rules as > > possible. > > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address. > > > Then again, I'm RB, not RG. ;) > > more people speaking up is good. I have strong opinions, and real problems > to address, but RG needs to hear from people other than me. Definitely! I often simply do not realize where a need is, and I always often make mistakes or have wrong perceptions. If nobody objects, I am lost and so is the project ;) Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 11:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 11:02:40 +0200 Subject: [rsyslog] DNS performance - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Tuesday, July 14, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 15:58, wrote: > > at the time I did quite a number of strace dumps to show how much time was > > being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. That's exactly part of the "new tools" idea, but obviously this instrumentation will also affect the overall picture. But I think I can do the base counters with a few atomic increments, so the effect should be not that bad. The real work shall then be done in the front-end. > > > well, if you are really wanting audit-grade logging, will you use anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on the > > sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. > > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. That's indeed an interesting thought. The allowedSenders is not really that much of a problem. First of all, not all inputs support them (one may argue they should, but one may also argue that a firewall is a much better place to block traffic...). Secondly, $AllowedSender is not used in most configurations. Rsyslog can detect that the sender name is not needed and in this case can actually defer the name lookup. Of course, that doesn't help in situations where the lookup is required, but it helps in many configurations. This is also the spirit of some of my recent performance enhancements: not all work equally well under all circumstances. But I have tried to make only those suffer that actually need the feature that bears the cost. I hope to extend that idea to the queue, where an ultra-fast mode may automatically be selected if the parameters indicate that is possible. But, again, some tooling to test this and its affects would be quite useful. Rainer From iamsayan at gmail.com Tue Jul 14 17:52:53 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:52:53 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Sorry if this is out of context. > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. Is there any other way to make the rsyslog process reread the conf file other than a restart? This is the only reason I use the traditional SIGHUP processing, I send HUP after I alter the rsyslog.conf file. On Tue, Jul 14, 2009 at 4:37 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Monday, July 13, 2009 11:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog - what's next? > > > > On Mon, 13 Jul 2009, RB wrote: > > > > > On Mon, Jul 13, 2009 at 14:38, wrote: > > >> HUP does not interrupt services. > > > > > > I'm sorry; last October you noted that for certain configurations > > > SIGHUP will drop memory-queued messages. In addition to the other > > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > > awful lot like a service interruption. Has that since changed for 3.x > > > or 4.2? > > > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP > not > > do a complete teardown, halt, and restart, but instead just closes and > > re-opens files and connections so that no long messages are lost in a > > restart. > > And that is the default setting! > > > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. > However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > For v5, I actually think of adding a "rsyslogd restart deamon" for those > that > insist on traditional HUP semantics. That deamon would lie dormant in > memory > until it receives HUP, in which case it does a shutdown and restart of the > real rsyslogd. With that, I could cleanup a lot of complexity. > > One major issue is that under some circumstances I need to cancel output > threads when they do not finish their processing within the allotted > timeouts. Canceling threads is always a bit dangerous, but there currently > is > no always-reliable way around this. To make it happen, a lot of > enable/disable cancel calls, including cancel cleanup handlers are made > throughout the code. If we would not have restart-type HUPs, I could simply > cancel those threads and not care about resources not being cleaned up (the > process cleanup will take care of that). So I could also remove all the > enable/disable cancel logic and its helpers. Of course, this is not > something > done as quickly as I write those lines and it must be made sure that any > inconsistence does not affect the rest of the shutdown. But without those > annoying restart-type HUPs, it is much simpler to do... > > > > >> the system calls to access it (and to first lookup if it should be > > >> accessed at all) take enough time to be significant at these speeds. > > > > > > May I gently prod you to substantiate this statement? I don't doubt > > > that it makes calls to external libraries, and that those calls are > > > likely more expensive than internal resolution, but "significant" is > > > not significant without numbers to back it up. Even a simple speed > > > comparison for a few million packets between 'no resolution' and 'in > > > /etc/hosts with a cold cache' would be useful. > > > > I would have to do a new set of tests to get you concrete numbers, but > > back in roughly the october timeframe last year I was doing extensive > > tests on this and with hosts files vs. no resolution I was seeing 4x or > > more differences (with a tiny, 5-line hosts file to parse) > > > > at the time I think I discussed it on a long performance thread on the > > website. > > > > at the time I did quite a number of strace dumps to show how much time > was > > being eaten up in the system calls. RG has done a LOT of cleanup since > > then (to the point that the failsafe audit mode is in the ballpark of > > being as fast as the in-memory mode was back then) > > > > >> you want the ability to cache the name lookup no matter what method is > > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not > include > > >> a TTL. > > > > > > I have no problem expanding the scope of discussion to resolution > > > methods beyond DNS (which was the original premise), but if a name > > > service does not provide an explicit TTL, it must be assumed as an > > > implicit '0', which blind caching will break. That said, each of > > > those methods still provides some [internal] method of caching and > > > invalidation that, to be audit-grade correct, rsyslog will either have > > > to replicate or trust. I still can't see a problem with having > > > something to the effect of a "$BreakNSCache" configuration option, but > > > by default a syslog daemon should play as strictly by the rules as > > > possible. > > > > well, if you are really wanting audit-grade logging, will you use > anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on > the > > sender's IP address. > > > > > Then again, I'm RB, not RG. ;) > > > > more people speaking up is good. I have strong opinions, and real > problems > > to address, but RG needs to hear from people other than me. > > Definitely! I often simply do not realize where a need is, and I always > often > make mistakes or have wrong perceptions. If nobody objects, I am lost and > so > is the project ;) > > Rainer > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 17:55:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 17:55:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Sorry if this is out of context. > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > > Is there any other way to make the rsyslog process reread the conf file > other than a restart? > This is the only reason I use the traditional SIGHUP processing, I send HUP > after I alter the rsyslog.conf file. No, but I always fail to see the big difference between typing $ kill -HUP ?cat /var/run/rsyslog.log? And $ /etc/rc.d/rsyslogd restart ;) Use the later one, and you do not need to use sighup. Even better, you can now use HUP for a lightweight "close the files only" log rotation thing ;) Rainer From iamsayan at gmail.com Tue Jul 14 17:58:58 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:58:58 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Yes exactly, I have changed the script to do just that recently , I was just wondering whether this is the reason why you get so much resistance when you want to deprecate the traditional HUP processing... On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > next? > > > > Sorry if this is out of context. > > > > > this matches the traditional use of HUP to get syslogd to release the > > > files it's writing to so that they can be rotated away. > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > > file is so complex and can significantly alter the software by loading > > > modules. > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > the > > root cause for a lot of complexities (read performance bottlenecks) > inside > > the engine. And all this for something that is not strictly needed. > However, > > there are always many opponents when I intend to remove the traditional > HUP > > behavior. > > > > > > Is there any other way to make the rsyslog process reread the conf file > > other than a restart? > > This is the only reason I use the traditional SIGHUP processing, I send > HUP > > after I alter the rsyslog.conf file. > > > No, but I always fail to see the big difference between typing > > $ kill -HUP ?cat /var/run/rsyslog.log? > > And > > $ /etc/rc.d/rsyslogd restart > > ;) > > Use the later one, and you do not need to use sighup. Even better, you can > now use HUP for a lightweight "close the files only" log rotation thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 18:06:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:06:38 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Yes exactly, I have changed the script to do just that recently , I was just > wondering whether this is the reason why you get so much resistance when you > want to deprecate the traditional HUP processing... I think it is maily a "tradition" argument along the lines "but I am used to...". Still hard to beat if everybody complains ;) I don't see any technicaly reasons, especially as a restart-type restart IS a restart (nothing less), just carried out in the most complex way ;) Rainer > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > wrote: > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > next? > > > > > > Sorry if this is out of context. > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > files it's writing to so that they can be rotated away. > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > config > > > > file is so complex and can significantly alter the software by loading > > > > modules. > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > > the > > > root cause for a lot of complexities (read performance bottlenecks) > > inside > > > the engine. And all this for something that is not strictly needed. > > However, > > > there are always many opponents when I intend to remove the traditional > > HUP > > > behavior. > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > other than a restart? > > > This is the only reason I use the traditional SIGHUP processing, I send > > HUP > > > after I alter the rsyslog.conf file. > > > > > > No, but I always fail to see the big difference between typing > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > And > > > > $ /etc/rc.d/rsyslogd restart > > > > ;) > > > > Use the later one, and you do not need to use sighup. Even better, you can > > now use HUP for a lightweight "close the files only" log rotation thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 18:17:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:17:59 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com><87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB99@GRFEXC.intern.adiscon.com> Oh, and I forgot to mention that recent builds loudly complain about config errors on stderr when you do a real restart bit silently need to dump these error messages to their respective bins (where experience tells nobody sees them...) when doing restart-type HUP. Maybe the result of this discussion is that I should really drop that misfeature in v5. Any violent opponents? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, July 14, 2009 6:07 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > > Yes exactly, I have changed the script to do just that recently , I was > just > > wondering whether this is the reason why you get so much resistance when > you > > want to deprecate the traditional HUP processing... > > I think it is maily a "tradition" argument along the lines "but I am used > to...". Still hard to beat if everybody complains ;) I don't see any > technicaly reasons, especially as a restart-type restart IS a restart > (nothing less), just carried out in the most complex way ;) > > Rainer > > > > > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > > wrote: > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > > next? > > > > > > > > Sorry if this is out of context. > > > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > > files it's writing to so that they can be rotated away. > > > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > > config > > > > > file is so complex and can significantly alter the software by > loading > > > > > modules. > > > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it > is > > > > the > > > > root cause for a lot of complexities (read performance bottlenecks) > > > inside > > > > the engine. And all this for something that is not strictly needed. > > > However, > > > > there are always many opponents when I intend to remove the traditional > > > HUP > > > > behavior. > > > > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > > other than a restart? > > > > This is the only reason I use the traditional SIGHUP processing, I send > > > HUP > > > > after I alter the rsyslog.conf file. > > > > > > > > > No, but I always fail to see the big difference between typing > > > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > > > And > > > > > > $ /etc/rc.d/rsyslogd restart > > > > > > ;) > > > > > > Use the later one, and you do not need to use sighup. Even better, you > can > > > now use HUP for a lightweight "close the files only" log rotation thing > ;) > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 15 12:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Jul 2009 12:06:53 +0200 Subject: [rsyslog] HUP Processing - restart type will go away Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBAA@GRFEXC.intern.adiscon.com> Hi all, based on yesterday's discussion, and review of the (partly new) arguments, I have decided to go through the hassle of finally deprecating the restart-type HUP. I have changed the v3 doc so that it no longer explicitly recommends using HUP to apply config changes. I have added a compatibility doc to v4 which explains why HUP will go away and when. I invite you to read this document: http://www.rsyslog.com/doc-v4compatibility.html With the recent v4-devel (4.5.1, just released), I have also changed the default of $HUPisRestart to "off". In v5, I have added another compatibility document. Finally, I will remove the restart-type HUP in v5 soon, starting with the directive go away and later actually removing the (lots of) code that support it (and finally getting the benefits). Rainer From tbergfeld at hq.adiscon.com Wed Jul 15 14:30:23 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 15 Jul 2009 14:30:23 +0200 Subject: [rsyslog] rsyslog 4.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBB1@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 4.5.1, a member of the development branch. This version offers improved performannce. There were also some bugfixes like a fix for a potential segfault when zip-compressed syslog records were received. It further provides the ability for the TCP output action to "rebind" its send socket after sending n messages (actually, it re-opens the connection, the name is used because this is a concept very similiar to $ActionUDPRebindInterval). This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml Changelog: http://www.rsyslog.com/Article388.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From alex at segv.de Wed Jul 15 11:20:30 2009 From: alex at segv.de (Alexander Elbs) Date: Wed, 15 Jul 2009 11:20:30 +0200 Subject: [rsyslog] Use precision timestamps Message-ID: <20090715092030.GC19320@segv.de> Hi, when using syslog(3) an application can send log messages via /dev/log to rsyslog and then to e.g. a file. If I enable high precision timestamps in rsyslog the log messages have a more precise timestamp. However there is some delay between the application generating a log message and rsyslog adding the timestamp. So why settle for less? :) (Well it is a distributed application, i.e. several processes and computers. So to debug interactions between the parts the correct ordering and timing is very important to me.) I wrote some code that opens /dev/log itself and sends the new format directly. This works very nice and I get the timestamps I want. Example code: -------------- #!/usr/bin/python import socket log = socket.socket( socket.AF_UNIX, socket.SOCK_DGRAM ) log.connect( "/dev/log" ) # VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG # PRI: 23 (==local7) * 8 + 4 (==warning) = 188 log.send( "<188>1 2009-07-15T09:45:12.463435Z mycomputer TEST_CLIENT 12345 SOME_PACKAGE This is a test message" ) log.close() -------------- However I have a few questions: - Is there some library code I could use that accepts high precision timestamps? Some kind of successor to syslog(3). - Is there a recommended way to detect if the syslog daemon will accept the new format? Currently this could mean checking if rsyslogd is listening on /dev/log or someone else. Otherwise the logging code needs to fall back to the old format that is understood by any syslog daemon (and use only second resolution). Mfg Alexander Elbs -- Alexander Elbs *** eMail alex at segv.de From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 14:40:42 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 14:40:42 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <200907161440.43049.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Jul 16 15:31:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Jul 2009 15:31:17 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Just a quick note: there is no such thing as a whitespace inside a syslog tag (read rfc3164, 5424). The message is severely malformed and i do not see any way how rsyslog could guess the intention of the sender correctly. Have a look at the doc set, there is a full doc page with elaborate info on such cases. (i dont have it at hand right now) rainer ----- Urspr?ngliche Nachricht ----- Von: "Luis Fernando Mu?oz Mej?as" An: "rsyslog at lists.adiscon.com" Gesendet: 16.07.09 14:41 Betreff: [rsyslog] Syslogtags with whitespaces misparsed? Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 15:48:31 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 15:48:31 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Message-ID: <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > Just a quick note: there is no such thing as a whitespace inside a syslog > tag (read rfc3164, 5424). Thanks a lot, I'll tell people around here. Now I know who I must piss off. :) > The message is severely malformed and i do not see any way how rsyslog > could guess the intention of the sender correctly Indeed, rsyslog is not responsible for misbehaving senders, I just needed to clarify the source of my problem. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 10:13:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 10:13:02 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Just let me say that I try really hard to make rsyslog guess right for even the most malformed mesage format. The real issue is that there have been no standards for many years, and so there is a lot of incompatible formats out there. In this case, however, I do really not see how I could handle that intelligently within the parser (except if we create per-host custom parsers, something on my list but not even begun to work on). I guess that the gconf folks will not want to change their format, because that, too could potentially also break a lot of things (log parsers!). A solution within rsyslog configuration could be to check the hostname and apply some special template (which then utilizes the property replacer to extract the "right" information) when the hostname is "gconf". That, of course, means that no host ever must be named "gconf", but I think you can enforce that ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Luis > Fernando Mu?oz Mej?as > Sent: Thursday, July 16, 2009 3:49 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > > Just a quick note: there is no such thing as a whitespace > inside a syslog > > tag (read rfc3164, 5424). > > Thanks a lot, I'll tell people around here. Now I know who I must piss > off. :) > > > The message is severely malformed and i do not see any way > how rsyslog > > could guess the intention of the sender correctly > > Indeed, rsyslog is not responsible for misbehaving senders, I just > needed to clarify the source of my problem. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 17 15:05:38 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 17 Jul 2009 15:05:38 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Message-ID: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > In this case, however, I do really not see how I could handle that > intelligently within the parser My guesswork, as you call it on the document you reference is something like "the first word after the timestamp must be the host name, and from there to the first colon it's all syslogtag; if there's no colon I'll do whatever I want but crashing". But it's guessing, against RFCs, and I don't really think a syslog parser should play fortune-telling. > I guess that the gconf folks will not want to change their format, because > that, too could potentially also break a lot of things (log parsers!). I'll try to follow this up to gconf guys, so that they know that any log parsing of their messages is necessarily lacking *crucial* information. Anyways, gconf is the least important application to me, and I see some services around here showing the same symptoms. > A solution within rsyslog configuration In my scenario, the message has gone through several syslog relays and the correct host information is lost before it comes to my service, so there is no way to configure rsyslog to solve it. Another funny example is syslog's habit of saying "last message repeated N times". These messages don't have a colon or anything useful to delimit the application name. In this case, I receive *lots* of messages from a host called "last". Thanks for the clarifications. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 15:42:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 15:42:10 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com><200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBCC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Friday, July 17, 2009 3:06 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > > In this case, however, I do really not see how I could handle that > > intelligently within the parser > > My guesswork, as you call it on the document you reference is something > like "the first word after the timestamp must be the host name, Just to clarify so that everyone sees where the issues are ;) First off, how do you know if there is a hostname? If the message is lacking a hostname, the TAG will become it. Rsyslog assumes that a hostname is present, but knows it is a TAG instead if a character that is not valid inside the hostname. In this example "gconf" there is no way to know this is not a valid hostname. > and from > there to the first colon it's all syslogtag; Next actually not-sosubtle issue: But what do you do if there is no colon at the end of the syslog TAG? The rfcs demand none (because the header filed is SP-terminated) and also in practice there is not always a colon in it. So following this rule would break RFC compatibility and also probably break a lot more real-world cases than it fixes. > if there's no colon I'll do > whatever I want but crashing". But it's guessing, against RFCs, and I > don't really think a syslog parser should play fortune-telling. > > > I guess that the gconf folks will not want to change their format, because > > that, too could potentially also break a lot of things (log parsers!). > > I'll try to follow this up to gconf guys, so that they know that any log > parsing of their messages is necessarily lacking *crucial* > information. Anyways, gconf is the least important application to me, > and I see some services around here showing the same symptoms. > > > A solution within rsyslog configuration > > In my scenario, the message has gone through several syslog relays and > the correct host information is lost before it comes to my service, so > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". Brings up an interesting thought (for another time) - it may make sense to de-multiplex these "last message repeated N times", but that's quite some effort... Rainer > > Thanks for the clarifications. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Fri Jul 17 15:54:40 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 17 Jul 2009 07:54:40 -0600 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <4255c2570907170654i7f46cfb8la4440d4bf0838bd1@mail.gmail.com> On Fri, Jul 17, 2009 at 07:05, Luis Fernando Mu?oz Mej?as wrote: > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". It was these precise messages from *BSD that drove me to using the fromhost-ip property as opposed to whatever the sender "says" they are. It breaks down in the face of relay hosts, but when I get to the point of needing those I plan on relying on secondary parsing. From rgerhards at hq.adiscon.com Mon Jul 20 16:02:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Jul 2009 16:02:55 +0200 Subject: [rsyslog] rsyslog status Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBE8@GRFEXC.intern.adiscon.com> Hi all, contrary to what I originally thought, I have worked "a bit" on the queue engine again. I couldn't resist to unleash at least some of the potential that the now gone-away restart-type HUP offered. I am probably through with a round of changes, and it looks rather well. The code has been greatly simplified, what I hope will also enhance its stability and make finding threading bugs much easier. All in all, the new engine looks much cleaner than the previous one. I also gained a bit more performance, but that is not so much of a difference. At the high end, you may say an increase in steady message rate by maybe one or two percent. The overhead of running multiple action queues has been reduced. The new code will probably be released tomorrow (will do some final checks), but it is available in the git master branch right now. I will now see if I begin to look into the set of performance monitoring tools or postpone that after my vacation and look at threaded inputs (at least for an initial pilot). I just thought I let you know. Rainer From tbergfeld at hq.adiscon.com Wed Jul 29 08:08:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 29 Jul 2009 08:08:15 +0200 Subject: [rsyslog] rsyslog 5.1.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC41@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 5.1.3, a member of the v5-development branch. This version offers a change in the architecture, queue now always has at least one worker thread if not running in direct mode. Previous versions could run without any active workers. This simplifies the code at a very small expense. See v5 compatibility note document for more in-depth discussion. There are also some bug fixes like a fix for a minor static memory leak while reading configuration. Furthermore there is the ability added to terminate input modules not via pthread_cancel but an alternate approach via pthread_kill. This is somewhat safer as we do not need to think about the cancel-safeness of all libraries we use. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-168.phtml Changelog: http://www.rsyslog.com/Article390.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Jul 30 03:24:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 29 Jul 2009 18:24:12 -0700 (PDT) Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf Message-ID: I have the following in my config file :programname, isequal, "iaalog" ~ :programname, isequal, "plug-gw" ~ *.* @192.168.210.246:514;TraditionalFormat the first time a log entry comes along that has a program name of iaalog rsyslog crashes 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 21:09:05|history|1428800' 6183.244047218:40f00950: Message has legacy syslog format. 6183.244051641:40f00950: Called action, logging to builtin-file 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx 6183.244062030:40f00950: entering actionCalldoAction(), state: itx 6183.244066591:40f00950: file to log to: /var/log/messages 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy 6183.244086454:40f00950: action call returned 0 6183.244091070:40f00950: Filter: check for property 'programname' (value 'iaalog') isequal 'iaalog': TRUE 6183.244096297:40f00950: Called action, logging to builtin-discard 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx 6183.244106591:40f00950: entering actionCalldoAction(), state: itx 6183.244109759:40f00950: 6183.244113010:40f00950: action call returned -2002 David Lang From toppiprc at gmail.com Thu Jul 30 04:02:37 2009 From: toppiprc at gmail.com (=?UTF-8?B?6JSh6LaF?=) Date: Thu, 30 Jul 2009 10:02:37 +0800 Subject: [rsyslog] how log non-ascii characters into mysql Message-ID: Hi, I want to record some non-ascii (e.g. UTF8) text using rsyslog. When the message is written to file, it is recorded as I logged, so it can be decoded and read well. But if the message is written to mysql, I can't decode it, even through I have changed charset configurations of the mysql database, tables and columns. What has happened before the messages were sent to mysql? Another problem, suppose I know the encodings of messages from all devices, and I want to transform them into some uniform one before saving. How can I achive it? Best regards Cai Chao From rgerhards at hq.adiscon.com Thu Jul 30 11:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 11:14:57 +0200 Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC52@GRFEXC.intern.adiscon.com> David, thanks for bringing this up. Actually, there were two problems in the code: number one was the segfault but number two was that no message was ever discarded. I have fixed both, but the second fix might cause some regression. The testbench (now extended to do a basic test of the discard functionality) runs through without problems, but obviously it does not yet cover all potential cases [and it will take more time until it really does...]. I will now leave for my vacation very shortly, so I cannot do any more in-depth testing and need to hope that this does not case too bad regressions. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, July 30, 2009 3:24 AM > To: rsyslog-users > Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf > > I have the following in my config file > > :programname, isequal, "iaalog" ~ > :programname, isequal, "plug-gw" ~ > *.* @192.168.210.246:514;TraditionalFormat > > the first time a log entry comes along that has a program name of iaalog > rsyslog crashes > > 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg > '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 > 21:09:05|history|1428800' > 6183.244047218:40f00950: Message has legacy syslog format. > 6183.244051641:40f00950: Called action, logging to builtin-file > 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx > 6183.244062030:40f00950: entering actionCalldoAction(), state: itx > 6183.244066591:40f00950: file to log to: /var/log/messages > 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 > 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 > 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes > 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy > 6183.244086454:40f00950: action call returned 0 > 6183.244091070:40f00950: Filter: check for property 'programname' (value > 'iaalog') isequal 'iaalog': TRUE > 6183.244096297:40f00950: Called action, logging to builtin-discard > 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx > 6183.244106591:40f00950: entering actionCalldoAction(), state: itx > 6183.244109759:40f00950: > 6183.244113010:40f00950: action call returned -2002 > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 12:03:33 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 12:03:33 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> Message-ID: <4A716FF5.8060608@redhat.com> Hi, just wanted to let you know that rsyslog-4.2.0 is now in rawhide and will be included in Fedora 12. Tomas On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > Hi Peter, > > that sounds quite interesting. I'll see that I get the v4-stable out > tomorrow. > > Thanks for following up... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >> Sent: Monday, June 22, 2009 3:53 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi folks, >> >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) >> >> At first we need v4 stable. If we have stable v4 we will push it into > rawhide >> and F11. This must be done ASAP, I see deadline in mid-July. >> >> Peter. >> >> >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>> Hi all, >>> >>> I guess RHEL6 will be based on Fedora 11 & 12. >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>> It's sure that RHEL6 includes v3 at least. >>> >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>> http://fedoraproject.org/wiki/Schedule >>> If v4 is stable and requested to be included in F12, v4 will be >>> shipped in F12. >>> >>> Best Rio. >>> >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>> That a good question and I, too, would like to know the answer. In >>>> any case, >>>> I hope it will not be v2. Giving that RHEL is a conservative >>>> distribution >>>> (and it has to be for the target base), I would assume that if they >>>> go for a >>>> newer version, it is v3. >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>> To: rsyslog at lists.adiscon.com >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>> >>>>> Hello, >>>>> is there any info about what rsyslog version will be available in >>>>> RHEL >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>> the >>>>> time "RHEL6 package freeze" will be in stable status? >>>>> >>>>> Thank you, >>>>> Jiri H. >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Jul 30 12:56:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 12:56:26 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Hi, thanks for sharing this excellent news! If I may try to ponder you for some more information... Does this increase the chance that a recent version of rsyslog will be part of the next version of RHEL? ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Thursday, July 30, 2009 12:04 PM > To: rsyslog-users > Subject: Re: [rsyslog] RSyslog version in RHEL 6 > > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas > > On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > > Hi Peter, > > > > that sounds quite interesting. I'll see that I get the v4-stable out > > tomorrow. > > > > Thanks for following up... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec > >> Sent: Monday, June 22, 2009 3:53 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 > >> > >> Hi folks, > >> > >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) > >> > >> At first we need v4 stable. If we have stable v4 we will push it into > > rawhide > >> and F11. This must be done ASAP, I see deadline in mid-July. > >> > >> Peter. > >> > >> > >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: > >>> Hi all, > >>> > >>> I guess RHEL6 will be based on Fedora 11 & 12. > >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. > >>> It's sure that RHEL6 includes v3 at least. > >>> > >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. > >>> http://fedoraproject.org/wiki/Schedule > >>> If v4 is stable and requested to be included in F12, v4 will be > >>> shipped in F12. > >>> > >>> Best Rio. > >>> > >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: > >>>> That a good question and I, too, would like to know the answer. In > >>>> any case, > >>>> I hope it will not be v2. Giving that RHEL is a conservative > >>>> distribution > >>>> (and it has to be for the target base), I would assume that if they > >>>> go for a > >>>> newer version, it is v3. > >>>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka > >>>>> Sent: Saturday, June 20, 2009 9:48 PM > >>>>> To: rsyslog at lists.adiscon.com > >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 > >>>>> > >>>>> Hello, > >>>>> is there any info about what rsyslog version will be available in > >>>>> RHEL > >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in > >>>>> the > >>>>> time "RHEL6 package freeze" will be in stable status? > >>>>> > >>>>> Thank you, > >>>>> Jiri H. > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 13:22:08 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 13:22:08 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Message-ID: <4A718260.8090806@redhat.com> As far as I know, this version will be the one to go into RHEL 6.0. Tomas On 07/30/2009 12:56 PM, Rainer Gerhards wrote: > Hi, > > thanks for sharing this excellent news! If I may try to ponder you for some > more information... Does this increase the chance that a recent version of > rsyslog will be part of the next version of RHEL? ;) > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >> Sent: Thursday, July 30, 2009 12:04 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi, >> >> just wanted to let you know that rsyslog-4.2.0 is now in rawhide and >> will be included in Fedora 12. >> >> Tomas >> >> On 06/22/2009 02:24 PM, Rainer Gerhards wrote: >>> Hi Peter, >>> >>> that sounds quite interesting. I'll see that I get the v4-stable out >>> tomorrow. >>> >>> Thanks for following up... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >>>> Sent: Monday, June 22, 2009 3:53 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >>>> >>>> Hi folks, >>>> >>>> rsyslog maintainer is on vacation at the moment so I'm answering this Q. > :) >>>> At first we need v4 stable. If we have stable v4 we will push it into >>> rawhide >>>> and F11. This must be done ASAP, I see deadline in mid-July. >>>> >>>> Peter. >>>> >>>> >>>> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>>>> Hi all, >>>>> >>>>> I guess RHEL6 will be based on Fedora 11 & 12. >>>>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>>>> It's sure that RHEL6 includes v3 at least. >>>>> >>>>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>>>> http://fedoraproject.org/wiki/Schedule >>>>> If v4 is stable and requested to be included in F12, v4 will be >>>>> shipped in F12. >>>>> >>>>> Best Rio. >>>>> >>>>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>>>> That a good question and I, too, would like to know the answer. In >>>>>> any case, >>>>>> I hope it will not be v2. Giving that RHEL is a conservative >>>>>> distribution >>>>>> (and it has to be for the target base), I would assume that if they >>>>>> go for a >>>>>> newer version, it is v3. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>>>> To: rsyslog at lists.adiscon.com >>>>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>>>> >>>>>>> Hello, >>>>>>> is there any info about what rsyslog version will be available in >>>>>>> RHEL >>>>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>>>> the >>>>>>> time "RHEL6 package freeze" will be in stable status? >>>>>>> >>>>>>> Thank you, >>>>>>> Jiri H. >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Jul 30 13:29:11 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 13:29:11 +0200 Subject: [rsyslog] merge failures Message-ID: Hi, just noticed this in v4-stable grep "<<<<" * -R tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am Seems like a merge error. Haven't checked the other branches. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Jul 30 14:23:23 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 14:23:23 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <4A716FF5.8060608@redhat.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com> <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: 2009/7/30 Tomas Heinrich : > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas Hi Tomas, that's good news. FWIW the current version in Debian unstable/testing is also 4.2.0, so squeeze will also ship with at least 4.2.x and Ubuntu also made the switch to rsyslog. Their next release 9.10 will have rsyslog 4.2.0 as default syslog daemon. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Jul 30 15:43:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 15:43:21 +0200 Subject: [rsyslog] merge failures References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC5E@GRFEXC.intern.adiscon.com> Thanks Fixed in v4-stable now, will propagate to the other branches over time... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, July 30, 2009 1:29 PM > To: rsyslog-users > Subject: [rsyslog] merge failures > > Hi, > just noticed this in v4-stable > > grep "<<<<" * -R > tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am > > Seems like a merge error. > > Haven't checked the other branches. > > Cheers, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 00:42:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:42:14 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > ubuntu 9.04 correction, debian 5.0, not ubuntu (forgot which machine I was compiling on) master also fails with the same error, v4-stable has no problem. David Lang > zlib version > /* zlib.h -- interface of the 'zlib' general purpose compression library > version 1.2.3.3, October 2nd, 2006 > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > compression library - development > > > error > > In file included from zlibw.h:28, > from stream.h:72, > from obj.h:50, > from rsyslog.h:482, > from stringbuf.c:39: > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gzseek64' > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gztell64' > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > before 'off64_t' > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > before 'off64_t' > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 00:50:13 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:50:13 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > On Tue, 30 Jun 2009, david at lang.hm wrote: > >> ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. and the error remains the same with configure --disable-zlib David Lang > David Lang > > >> zlib version >> /* zlib.h -- interface of the 'zlib' general purpose compression library >> version 1.2.3.3, October 2nd, 2006 >> >> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >> compression library - development >> >> >> error >> >> In file included from zlibw.h:28, >> from stream.h:72, >> from obj.h:50, >> from rsyslog.h:482, >> from stringbuf.c:39: >> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gzseek64' >> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gztell64' >> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >> before 'off64_t' >> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >> before 'off64_t' >> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 09:13:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 09:13:29 +0200 Subject: [rsyslog] v5-devel won't compile References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Looks like a problem "with" Debian. I guess I defined some types that zlib also defines to the same names. I have to admit that Michael Biebl told me, but it somehow slipped my mind... Will test on Debian today and fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 12:42 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Tue, 30 Jun 2009, david at lang.hm wrote: > > > ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. > > David Lang > > > > zlib version > > /* zlib.h -- interface of the 'zlib' general purpose compression library > > version 1.2.3.3, October 2nd, 2006 > > > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > compression library - development > > > > > > error > > > > In file included from zlibw.h:28, > > from stream.h:72, > > from obj.h:50, > > from rsyslog.h:482, > > from stringbuf.c:39: > > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gzseek64' > > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gztell64' > > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > > before 'off64_t' > > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > > before 'off64_t' > > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory `/usr/src/rsyslog' > > make: *** [all] Error 2 > > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:10:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:10:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Looks like a problem "with" Debian. I guess I defined some types that zlib > also defines to the same names. I have to admit that Michael Biebl told me, > but it somehow slipped my mind... Will test on Debian today and fix. master and v5-devel also complained that I didn't have the zlib development libraries installed (unable to find zlib.h), after I installed that it gave me the error I posted below. v4 didn't complain about the missing library to start with as I posted in another message, I was suprised the the problem didn't go away when I configured with --disable-zlib David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 12:42 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> On Tue, 30 Jun 2009, david at lang.hm wrote: >> >>> ubuntu 9.04 >> >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling >> on) >> >> master also fails with the same error, v4-stable has no problem. >> >> David Lang >> >> >>> zlib version >>> /* zlib.h -- interface of the 'zlib' general purpose compression library >>> version 1.2.3.3, October 2nd, 2006 >>> >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>> compression library - development >>> >>> >>> error >>> >>> In file included from zlibw.h:28, >>> from stream.h:72, >>> from obj.h:50, >>> from rsyslog.h:482, >>> from stringbuf.c:39: >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gzseek64' >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gztell64' >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:14:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:14:37 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:11 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Looks like a problem "with" Debian. I guess I defined some types that zlib > > also defines to the same names. I have to admit that Michael Biebl told me, > > but it somehow slipped my mind... Will test on Debian today and fix. > > master and v5-devel also complained that I didn't have the zlib > development libraries installed (unable to find zlib.h), after I installed > that it gave me the error I posted below. I have begun to look into it. Debian has a slightly newer version of zlib than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem occurs. I now need to look at the root cause, but it is well hidden ;) > > v4 didn't complain about the missing library to start with > > as I posted in another message, I was suprised the the problem didn't go > away when I configured with --disable-zlib That's a bug in omfile, it obviously does not yet properly reflect this switch. Will fix that one, too. Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Wednesday, July 01, 2009 12:42 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > >> > >>> ubuntu 9.04 > >> > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > >> on) > >> > >> master also fails with the same error, v4-stable has no problem. > >> > >> David Lang > >> > >> > >>> zlib version > >>> /* zlib.h -- interface of the 'zlib' general purpose compression library > >>> version 1.2.3.3, October 2nd, 2006 > >>> > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>> compression library - development > >>> > >>> > >>> error > >>> > >>> In file included from zlibw.h:28, > >>> from stream.h:72, > >>> from obj.h:50, > >>> from rsyslog.h:482, > >>> from stringbuf.c:39: > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gzseek64' > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gztell64' > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>> make[1]: *** [all-recursive] Error 1 > >>> make[1]: Leaving directory `/usr/src/rsyslog' > >>> make: *** [all] Error 2 > >>> > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 10:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:35:56 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Of course (and as always), the problem rooted in rsyslog. The commit says it all: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 308f383bec1b8be8 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, July 01, 2009 10:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, July 01, 2009 10:11 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > > > Looks like a problem "with" Debian. I guess I defined some types that > zlib > > > also defines to the same names. I have to admit that Michael Biebl told > me, > > > but it somehow slipped my mind... Will test on Debian today and fix. > > > > master and v5-devel also complained that I didn't have the zlib > > development libraries installed (unable to find zlib.h), after I installed > > that it gave me the error I posted below. > > I have begun to look into it. Debian has a slightly newer version of zlib > than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > occurs. I now need to look at the root cause, but it is well hidden ;) > > > > > v4 didn't complain about the missing library to start with > > > > as I posted in another message, I was suprised the the problem didn't go > > away when I configured with --disable-zlib > > That's a bug in omfile, it obviously does not yet properly reflect this > switch. Will fix that one, too. > > Rainer > > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > >> Sent: Wednesday, July 01, 2009 12:42 AM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] v5-devel won't compile > > >> > > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > > >> > > >>> ubuntu 9.04 > > >> > > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > > >> on) > > >> > > >> master also fails with the same error, v4-stable has no problem. > > >> > > >> David Lang > > >> > > >> > > >>> zlib version > > >>> /* zlib.h -- interface of the 'zlib' general purpose compression > library > > >>> version 1.2.3.3, October 2nd, 2006 > > >>> > > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > >>> compression library - development > > >>> > > >>> > > >>> error > > >>> > > >>> In file included from zlibw.h:28, > > >>> from stream.h:72, > > >>> from obj.h:50, > > >>> from rsyslog.h:482, > > >>> from stringbuf.c:39: > > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gzseek64' > > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gztell64' > > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > >>> make[1]: *** [all-recursive] Error 1 > > >>> make[1]: Leaving directory `/usr/src/rsyslog' > > >>> make: *** [all] Error 2 > > >>> > > >>> > > >>> David Lang > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:53:15 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:53:15 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 I'll test it when I get into the office. when I tried to do it on my laptop I ran into the testbench failure again, even with --disable-testbench CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/bash: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 config.log: $ ./configure --disable-testbench David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:56:08 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAA0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Of course (and as always), the problem rooted in rsyslog. The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > > 308f383bec1b8be8 > > I'll test it when I get into the office. when I tried to do it on my > laptop I ran into the testbench failure again, even with > --disable-testbench > > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/bash: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > Mmhhh... I did my testing on a debian 5.0 machine without javadev and with --enable-testbench=no. Seems to make a difference (I am still not deep inside autoconf...). > > config.log: $ ./configure --disable-testbench > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined some types that > >> zlib > >>>> also defines to the same names. I have to admit that Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > >> occurs. I now need to look at the root cause, but it is well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 15:18:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 15:18:35 +0200 Subject: [rsyslog] performance optimization (milestone) done Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have reached a milestone with my optimizations. My goal of reducing malloc/free calls has now been reached. I need to revisit priorities, but I will probably move away from performance optimization for a while now (maybe some smaller things that I stumble across, but not a real effort). As I have written recently, there is still ample potential for optimization. I will "just" probably not explore it at this point in time. However, I would like to ask one question that could lead to a new effort (in v5). Looking at the queue engine, I see that managing the queue, especially in ultra-reliable mode, requires a lot of effort, what also means a lot of CPU time. If we relax reliability conditions, we could definitely save some time here (probably a real lot!). So my question is what would be the least reliability that you need for a non-audit, high-performance scenario. I can envision that it is sufficient to have this: All messages handed over to the queue will be processed if no system failure occurs and if there is space available inside the queue. All messages are queued in main memory. If messages are tried to be enqueued when the queue is full, these messages will be lost. Message loss victims will be strictly based on order of enqueue (queue full condition) and not severity or any other property (evaluating that would take time, again). Also, it is acceptable to lose unprocessed messages during rsyslogd shutdown, if they cannot be processed within the (configurable) shutdown intervals. Would this make sense for sufficiently important use cases? If so, I could see that (over time!) I implement a very fast queue driver based on that relaxed criteria. With it, I think I could also create a lock-free version (but not immediately), which would definitely be a big performance gain. Feedback is appreciated. The optimized builds are currently in the git master and v5-devel branches, new releases are due soon (but will see that I do at least a bit more doc work before releasing them). Rainer From david at lang.hm Wed Jul 1 18:58:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 09:58:55 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I just wanted to let you know that I have reached a milestone with my > optimizations. My goal of reducing malloc/free calls has now been reached. I > need to revisit priorities, but I will probably move away from performance > optimization for a while now (maybe some smaller things that I stumble > across, but not a real effort). > > As I have written recently, there is still ample potential for optimization. > I will "just" probably not explore it at this point in time. > > However, I would like to ask one question that could lead to a new effort (in > v5). Looking at the queue engine, I see that managing the queue, especially > in ultra-reliable mode, requires a lot of effort, what also means a lot of > CPU time. > > If we relax reliability conditions, we could definitely save some time here > (probably a real lot!). So my question is what would be the least reliability > that you need for a non-audit, high-performance scenario. I can envision that > it is sufficient to have this: > > All messages handed over to the queue will be processed if no system failure > occurs and if there is space available inside the queue. All messages are > queued in main memory. If messages are tried to be enqueued when the queue is > full, these messages will be lost. Message loss victims will be strictly > based on order of enqueue (queue full condition) and not severity or any > other property (evaluating that would take time, again). Also, it is > acceptable to lose unprocessed messages during rsyslogd shutdown, if they > cannot be processed within the (configurable) shutdown intervals. > > Would this make sense for sufficiently important use cases? If so, I could > see that (over time!) I implement a very fast queue driver based on that > relaxed criteria. With it, I think I could also create a lock-free version > (but not immediately), which would definitely be a big performance gain. how is this different than what happens today? David Lang > Feedback is appreciated. > > The optimized builds are currently in the git master and v5-devel branches, > new releases are due soon (but will see that I do at least a bit more doc > work before releasing them). > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 19:37:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 10:37:27 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: back on the debian 5 system I get the following error with v5-devel commit 6322343eca1084b509386a94c1bcf2ca819f1220 gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o rsyslogd-omfwd.o rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread ../runtime/.libs/librsyslog.a -ldl -lrt rsyslogd-syslogd.o: In function `mainloop': /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to `execScheduled' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtExit': /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference to `rulesetClassExit' /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference to `ruleClassExit' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtInit': /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference to `propClassInit' /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference to `ruleClassInit' /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference to `rulesetClassInit' ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In function `objClassInit': /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to `apcClassInit' collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/rsyslog/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Date: Wed, 1 Jul 2009 10:35:56 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:24:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:24:17 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, runtime/apc.c) are missing. I have just checked, they are in the git branch. Also I have no problems compiling that version, I tried it on Debian this morning during the zlib fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 7:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > back on the debian 5 system I get the following error with v5-devel > commit 6322343eca1084b509386a94c1bcf2ca819f1220 > > > gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > rsyslogd-omfwd.o > rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > ../runtime/.libs/librsyslog.a -ldl -lrt > rsyslogd-syslogd.o: In function `mainloop': > /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > `execScheduled' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtExit': > /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > to `rulesetClassExit' > /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > to `ruleClassExit' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtInit': > /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > to `propClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > to `ruleClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > to `rulesetClassInit' > ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > function `objClassInit': > /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > `apcClassInit' > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > > On Wed, 1 Jul 2009, Rainer > Gerhards wrote: > > > Date: Wed, 1 Jul 2009 10:35:56 +0200 > > From: Rainer Gerhards > > Reply-To: rsyslog-users > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > Of course (and as always), the problem rooted in rsyslog. > The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > 39b46630d95a8d8 > > 308f383bec1b8be8 > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined > some types that > >> zlib > >>>> also defines to the same names. I have to admit that > Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian > today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer > version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > header the problem > >> occurs. I now need to look at the root cause, but it is > well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the > problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly > reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose > compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:29:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:29:08 +0200 Subject: [rsyslog] performance optimization (milestone) done References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 6:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] performance optimization (milestone) done > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I just wanted to let you know that I have reached a > milestone with my > > optimizations. My goal of reducing malloc/free calls has > now been reached. I > > need to revisit priorities, but I will probably move away > from performance > > optimization for a while now (maybe some smaller things > that I stumble > > across, but not a real effort). > > > > As I have written recently, there is still ample potential > for optimization. > > I will "just" probably not explore it at this point in time. > > > > However, I would like to ask one question that could lead > to a new effort (in > > v5). Looking at the queue engine, I see that managing the > queue, especially > > in ultra-reliable mode, requires a lot of effort, what also > means a lot of > > CPU time. > > > > If we relax reliability conditions, we could definitely > save some time here > > (probably a real lot!). So my question is what would be the > least reliability > > that you need for a non-audit, high-performance scenario. I > can envision that > > it is sufficient to have this: > > > > All messages handed over to the queue will be processed if > no system failure > > occurs and if there is space available inside the queue. > All messages are > > queued in main memory. If messages are tried to be enqueued > when the queue is > > full, these messages will be lost. Message loss victims > will be strictly > > based on order of enqueue (queue full condition) and not > severity or any > > other property (evaluating that would take time, again). Also, it is > > acceptable to lose unprocessed messages during rsyslogd > shutdown, if they > > cannot be processed within the (configurable) shutdown intervals. > > > > Would this make sense for sufficiently important use cases? > If so, I could > > see that (over time!) I implement a very fast queue driver > based on that > > relaxed criteria. With it, I think I could also create a > lock-free version > > (but not immediately), which would definitely be a big > performance gain. > > how is this different than what happens today? Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, persistence, reliability, discard messages based on priority, slow down sender, rate limiting ... stuff. All nice and useful, but takes time... What I described above would be the sole capability, (almost) none of these bells and whistles (except those that are handled in the action engine, I do not know out of my head which these are, but a low subset of what I said). Rainer > > David Lang > > > Feedback is appreciated. > > > > The optimized builds are currently in the git master and > v5-devel branches, > > new releases are due soon (but will see that I do at least > a bit more doc > > work before releasing them). > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:51:36 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, > runtime/apc.c) are missing. I have just checked, they are in the git branch. > Also I have no problems compiling that version, I tried it on Debian this > morning during the zlib fix. hmm, those files were definantly missing. I had done a git checkout origin/v5-devel, I then did a git checkout -f origin/v5-devel and the files are there. now it is dieing in the configure step :-( checking for FSSTND support... yes ./configure: line 10307: syntax error near unexpected token `GNUTLS,' ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= 1.4.0)' make: *** [config.status] Error 2 I tried doing the autoreconf command, and it isn't happy #autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 my next step is going to be to nuke the entire directory and try a checkout again. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 7:37 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> back on the debian 5 system I get the following error with v5-devel >> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >> >> >> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >> rsyslogd-omfwd.o >> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >> ../runtime/.libs/librsyslog.a -ldl -lrt >> rsyslogd-syslogd.o: In function `mainloop': >> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >> `execScheduled' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtExit': >> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >> to `rulesetClassExit' >> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >> to `ruleClassExit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtInit': >> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >> to `propClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >> to `ruleClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >> to `rulesetClassInit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >> function `objClassInit': >> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >> `apcClassInit' >> collect2: ld returned 1 exit status >> make[2]: *** [rsyslogd] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/tools' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> >> On Wed, 1 Jul 2009, Rainer >> Gerhards wrote: >> >>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> Of course (and as always), the problem rooted in rsyslog. >> The commit says it >>> all: >>> >>> >> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >> 39b46630d95a8d8 >>> 308f383bec1b8be8 >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>> >>>>>> Looks like a problem "with" Debian. I guess I defined >> some types that >>>> zlib >>>>>> also defines to the same names. I have to admit that >> Michael Biebl told >>>> me, >>>>>> but it somehow slipped my mind... Will test on Debian >> today and fix. >>>>> >>>>> master and v5-devel also complained that I didn't have the zlib >>>>> development libraries installed (unable to find zlib.h), after I >>> installed >>>>> that it gave me the error I posted below. >>>> >>>> I have begun to look into it. Debian has a slightly newer >> version of zlib >>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >> header the problem >>>> occurs. I now need to look at the root cause, but it is >> well hidden ;) >>>> >>>>> >>>>> v4 didn't complain about the missing library to start with >>>>> >>>>> as I posted in another message, I was suprised the the >> problem didn't go >>>>> away when I configured with --disable-zlib >>>> >>>> That's a bug in omfile, it obviously does not yet properly >> reflect this >>>> switch. Will fix that one, too. >>>> >>>> Rainer >>>> >>>>> >>>>> David Lang >>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>> >>>>>>>> ubuntu 9.04 >>>>>>> >>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>> compiling >>>>>>> on) >>>>>>> >>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>>> zlib version >>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >> compression >>>> library >>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>> >>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>> compression library - development >>>>>>>> >>>>>>>> >>>>>>>> error >>>>>>>> >>>>>>>> In file included from zlibw.h:28, >>>>>>>> from stream.h:72, >>>>>>>> from obj.h:50, >>>>>>>> from rsyslog.h:482, >>>>>>>> from stringbuf.c:39: >>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gztell64' >>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>> make: *** [all] Error 2 >>>>>>>> >>>>>>>> >>>>>>>> David Lang >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:52:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:52:57 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 6:59 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] performance optimization (milestone) done >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> Hi all, >>> >>> I just wanted to let you know that I have reached a >> milestone with my >>> optimizations. My goal of reducing malloc/free calls has >> now been reached. I >>> need to revisit priorities, but I will probably move away >> from performance >>> optimization for a while now (maybe some smaller things >> that I stumble >>> across, but not a real effort). >>> >>> As I have written recently, there is still ample potential >> for optimization. >>> I will "just" probably not explore it at this point in time. >>> >>> However, I would like to ask one question that could lead >> to a new effort (in >>> v5). Looking at the queue engine, I see that managing the >> queue, especially >>> in ultra-reliable mode, requires a lot of effort, what also >> means a lot of >>> CPU time. >>> >>> If we relax reliability conditions, we could definitely >> save some time here >>> (probably a real lot!). So my question is what would be the >> least reliability >>> that you need for a non-audit, high-performance scenario. I >> can envision that >>> it is sufficient to have this: >>> >>> All messages handed over to the queue will be processed if >> no system failure >>> occurs and if there is space available inside the queue. >> All messages are >>> queued in main memory. If messages are tried to be enqueued >> when the queue is >>> full, these messages will be lost. Message loss victims >> will be strictly >>> based on order of enqueue (queue full condition) and not >> severity or any >>> other property (evaluating that would take time, again). Also, it is >>> acceptable to lose unprocessed messages during rsyslogd >> shutdown, if they >>> cannot be processed within the (configurable) shutdown intervals. >>> >>> Would this make sense for sufficiently important use cases? >> If so, I could >>> see that (over time!) I implement a very fast queue driver >> based on that >>> relaxed criteria. With it, I think I could also create a >> lock-free version >>> (but not immediately), which would definitely be a big >> performance gain. >> >> how is this different than what happens today? > > Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, > persistence, reliability, discard messages based on priority, slow down > sender, rate limiting ... stuff. All nice and useful, but takes time... > > What I described above would be the sole capability, (almost) none of these > bells and whistles (except those that are handled in the action engine, I do > not know out of my head which these are, but a low subset of what I said). what you described as the new, limited capabilities sounds like what I would expect from a syslog daemon. David Lang From david at lang.hm Wed Jul 1 22:58:59 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:58:59 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. still no luck, secdev:/usr/src/rsyslog# rm -r * secdev:/usr/src/rsyslog# ls -a . .. .deps .git .gitignore .libs secdev:/usr/src/rsyslog# ls .libs lmtcpclt.la lmtcpclt.lai lmtcpclt_la-tcpclt.o lmtcpclt.so lmtcpsrv.la lmtcpsrv.lai lmtcpsrv_la-tcpsrv.o lmtcpsrv_la-tcps_sess.o lmtcpsrv.so secdev:/usr/src/rsyslog# !git git checkout -f origin/v5-devel Checking out files: 100% (497/497), done. HEAD is now at 6322343... Merge branch 'master' into v5-devel secdev:/usr/src/rsyslog# ls runtime/rule* runtime/rule.c runtime/rule.h runtime/ruleset.c runtime/ruleset.h secdev:/usr/src/rsyslog# autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 secdev:/usr/src/rsyslog# autoreconf -V autoreconf (GNU Autoconf) 2.61 Copyright (C) 2006 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Written by David J. MacKenzie and Akim Demaille. --- Autoconf 2.50 chosen by Debian wrapper script. For information and tuning advice see autoconf(1). > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:02:23 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:02:23 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: never mind, pkg-config was not installed on this server (and I've got _no_ idea how I was sucessfully compiling the v4-stable branch on this box) David Lang On Wed, 1 Jul 2009, david at lang.hm wrote: > Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. > > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:22:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:22:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > never mind, pkg-config was not installed on this server (and I've got _no_ > idea how I was sucessfully compiling the v4-stable branch on this box) that wasn't it either, it turns out that the libtool package is needed as well. disabling the testbench doesn't seem to work for me (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' Making all in tests make[2]: Entering directory `/usr/src/rsyslog/tests' CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/sh: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 secdev:/usr/src/rsyslog# history |tail 1408 ./configure 1409 ./configure --help 1410 ./configure --enable-imfile 1411 make 1412 ./configure --help |grep test 1413 ./configure --enable-imfile --enable-testbench=no 1414 make 1415 make clean 1416 make 1417 history |tail I'm now installing java.. David Lang > On Wed, 1 Jul 2009, david at lang.hm wrote: > >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >>> runtime/apc.c) are missing. I have just checked, they are in the git branch. >>> Also I have no problems compiling that version, I tried it on Debian this >>> morning during the zlib fix. >> >> hmm, those files were definantly missing. >> >> I had done a git checkout origin/v5-devel, I then did a git checkout -f >> origin/v5-devel and the files are there. >> >> now it is dieing in the configure step :-( >> >> checking for FSSTND support... yes >> ./configure: line 10307: syntax error near unexpected token `GNUTLS,' >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= >> 1.4.0)' >> make: *** [config.status] Error 2 >> >> >> I tried doing the autoreconf command, and it isn't happy >> >> #autoreconf -fvi >> autoreconf2.50: Entering directory `.' >> autoreconf2.50: configure.ac: not using Gettext >> autoreconf2.50: running: aclocal --force -I m4 >> autoreconf2.50: configure.ac: tracing >> autoreconf2.50: configure.ac: not using Libtool >> autoreconf2.50: running: /usr/bin/autoconf --force >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC >> If this token and others are legitimate, please use >> m4_pattern_allow. >> See the Autoconf documentation. >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 >> >> my next step is going to be to nuke the entire directory and try a >> checkout again. >> >> David Lang >> >> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>> Sent: Wednesday, July 01, 2009 7:37 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> back on the debian 5 system I get the following error with v5-devel >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>>> >>>> >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>>> rsyslogd-omfwd.o >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>>> ../runtime/.libs/librsyslog.a -ldl -lrt >>>> rsyslogd-syslogd.o: In function `mainloop': >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>>> `execScheduled' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtExit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>>> to `rulesetClassExit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>>> to `ruleClassExit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtInit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>>> to `propClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>>> to `ruleClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>>> to `rulesetClassInit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>>> function `objClassInit': >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>>> `apcClassInit' >>>> collect2: ld returned 1 exit status >>>> make[2]: *** [rsyslogd] Error 1 >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>>> make[1]: *** [all-recursive] Error 1 >>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>> make: *** [all] Error 2 >>>> >>>> >>>> >>>> On Wed, 1 Jul 2009, Rainer >>>> Gerhards wrote: >>>> >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>>> From: Rainer Gerhards >>>>> Reply-To: rsyslog-users >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> Of course (and as always), the problem rooted in rsyslog. >>>> The commit says it >>>>> all: >>>>> >>>>> >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>>> 39b46630d95a8d8 >>>>> 308f383bec1b8be8 >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>>> >>>>>>>> Looks like a problem "with" Debian. I guess I defined >>>> some types that >>>>>> zlib >>>>>>>> also defines to the same names. I have to admit that >>>> Michael Biebl told >>>>>> me, >>>>>>>> but it somehow slipped my mind... Will test on Debian >>>> today and fix. >>>>>>> >>>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>>> development libraries installed (unable to find zlib.h), after I >>>>> installed >>>>>>> that it gave me the error I posted below. >>>>>> >>>>>> I have begun to look into it. Debian has a slightly newer >>>> version of zlib >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>>> header the problem >>>>>> occurs. I now need to look at the root cause, but it is >>>> well hidden ;) >>>>>> >>>>>>> >>>>>>> v4 didn't complain about the missing library to start with >>>>>>> >>>>>>> as I posted in another message, I was suprised the the >>>> problem didn't go >>>>>>> away when I configured with --disable-zlib >>>>>> >>>>>> That's a bug in omfile, it obviously does not yet properly >>>> reflect this >>>>>> switch. Will fix that one, too. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>>> >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>>> >>>>>>>>>> ubuntu 9.04 >>>>>>>>> >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>>> compiling >>>>>>>>> on) >>>>>>>>> >>>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>> >>>>>>>>>> zlib version >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>>> compression >>>>>> library >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>>> >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>>> compression library - development >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> error >>>>>>>>>> >>>>>>>>>> In file included from zlibw.h:28, >>>>>>>>>> from stream.h:72, >>>>>>>>>> from obj.h:50, >>>>>>>>>> from rsyslog.h:482, >>>>>>>>>> from stringbuf.c:39: >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>>> make: *** [all] Error 2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> David Lang >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Jul 2 07:57:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 07:57:04 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAB0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 11:23 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > never mind, pkg-config was not installed on this server > (and I've got _no_ > > idea how I was sucessfully compiling the v4-stable branch > on this box) > > that wasn't it either, it turns out that the libtool package > is needed as > well. > > disabling the testbench doesn't seem to work for me Did you use "--enable-testbench=no"? This definitely works for me... > > (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) > make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' > Making all in tests > make[2]: Entering directory `/usr/src/rsyslog/tests' > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/sh: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > secdev:/usr/src/rsyslog# history |tail > 1408 ./configure > 1409 ./configure --help > 1410 ./configure --enable-imfile > 1411 make > 1412 ./configure --help |grep test > 1413 ./configure --enable-imfile --enable-testbench=no > 1414 make > 1415 make clean > 1416 make > 1417 history |tail > > > I'm now installing java.. > > David Lang > > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > >> > >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> > >>> This sounds like a couple of files (runtime/rule.c, > runtime/ruleset.c, > >>> runtime/apc.c) are missing. I have just checked, they are > in the git branch. > >>> Also I have no problems compiling that version, I tried > it on Debian this > >>> morning during the zlib fix. > >> > >> hmm, those files were definantly missing. > >> > >> I had done a git checkout origin/v5-devel, I then did a > git checkout -f > >> origin/v5-devel and the files are there. > >> > >> now it is dieing in the configure step :-( > >> > >> checking for FSSTND support... yes > >> ./configure: line 10307: syntax error near unexpected > token `GNUTLS,' > >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > >> 1.4.0)' > >> make: *** [config.status] Error 2 > >> > >> > >> I tried doing the autoreconf command, and it isn't happy > >> > >> #autoreconf -fvi > >> autoreconf2.50: Entering directory `.' > >> autoreconf2.50: configure.ac: not using Gettext > >> autoreconf2.50: running: aclocal --force -I m4 > >> autoreconf2.50: configure.ac: tracing > >> autoreconf2.50: configure.ac: not using Libtool > >> autoreconf2.50: running: /usr/bin/autoconf --force > >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > >> If this token and others are legitimate, please use > >> m4_pattern_allow. > >> See the Autoconf documentation. > >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > >> > >> my next step is going to be to nuke the entire directory and try a > >> checkout again. > >> > >> David Lang > >> > >> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com > >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >>>> Sent: Wednesday, July 01, 2009 7:37 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>> > >>>> back on the debian 5 system I get the following error > with v5-devel > >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 > >>>> > >>>> > >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > >>>> rsyslogd-omfwd.o > >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > >>>> ../runtime/.libs/librsyslog.a -ldl -lrt > >>>> rsyslogd-syslogd.o: In function `mainloop': > >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > >>>> `execScheduled' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtExit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > >>>> to `rulesetClassExit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > >>>> to `ruleClassExit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtInit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > >>>> to `propClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > >>>> to `ruleClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > >>>> to `rulesetClassInit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > >>>> function `objClassInit': > >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > >>>> `apcClassInit' > >>>> collect2: ld returned 1 exit status > >>>> make[2]: *** [rsyslogd] Error 1 > >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' > >>>> make[1]: *** [all-recursive] Error 1 > >>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>> make: *** [all] Error 2 > >>>> > >>>> > >>>> > >>>> On Wed, 1 Jul 2009, Rainer > >>>> Gerhards wrote: > >>>> > >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 > >>>>> From: Rainer Gerhards > >>>>> Reply-To: rsyslog-users > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> Of course (and as always), the problem rooted in rsyslog. > >>>> The commit says it > >>>>> all: > >>>>> > >>>>> > >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > >>>> 39b46630d95a8d8 > >>>>> 308f383bec1b8be8 > >>>>> > >>>>> Rainer > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM > >>>>>> To: rsyslog-users > >>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>> > >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>>>>>> > >>>>>>>> Looks like a problem "with" Debian. I guess I defined > >>>> some types that > >>>>>> zlib > >>>>>>>> also defines to the same names. I have to admit that > >>>> Michael Biebl told > >>>>>> me, > >>>>>>>> but it somehow slipped my mind... Will test on Debian > >>>> today and fix. > >>>>>>> > >>>>>>> master and v5-devel also complained that I didn't > have the zlib > >>>>>>> development libraries installed (unable to find > zlib.h), after I > >>>>> installed > >>>>>>> that it gave me the error I posted below. > >>>>>> > >>>>>> I have begun to look into it. Debian has a slightly newer > >>>> version of zlib > >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > >>>> header the problem > >>>>>> occurs. I now need to look at the root cause, but it is > >>>> well hidden ;) > >>>>>> > >>>>>>> > >>>>>>> v4 didn't complain about the missing library to start with > >>>>>>> > >>>>>>> as I posted in another message, I was suprised the the > >>>> problem didn't go > >>>>>>> away when I configured with --disable-zlib > >>>>>> > >>>>>> That's a bug in omfile, it obviously does not yet properly > >>>> reflect this > >>>>>> switch. Will fix that one, too. > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>>> Rainer > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>>>> > >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>>>>>> > >>>>>>>>>> ubuntu 9.04 > >>>>>>>>> > >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which > machine I was > >>>>> compiling > >>>>>>>>> on) > >>>>>>>>> > >>>>>>>>> master also fails with the same error, v4-stable > has no problem. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> zlib version > >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose > >>>> compression > >>>>>> library > >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>>>>>> > >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>>>>>> compression library - development > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> error > >>>>>>>>>> > >>>>>>>>>> In file included from zlibw.h:28, > >>>>>>>>>> from stream.h:72, > >>>>>>>>>> from obj.h:50, > >>>>>>>>>> from rsyslog.h:482, > >>>>>>>>>> from stringbuf.c:39: > >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gzseek64' > >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gztell64' > >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>>>>>> make: *** [all] Error 2 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> David Lang > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>> http://www.rsyslog.com > >>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From dirk.schulz at kinzesberg.de Thu Jul 2 10:34:45 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 02 Jul 2009 10:34:45 +0200 Subject: [rsyslog] Rsyslog on NetBSD 5 Message-ID: <4A4C7125.6090109@kinzesberg.de> Hi folks, I am running Rsyslog on RedHat/CentOS and Debian/Ubuntu for quite a while now and am happy with it. Now I would like to test it on NetBSD 5, but I do not find anything on that out there on the web. NetBSD themselves do not offer it in package source, and there is no report on that on the web. Anyone on the list using rsyslog on NetBSD (5)? Any hints or help? Or is it just compile and run? Thanks in advance for any help or link to related information. Dirk From rgerhards at hq.adiscon.com Thu Jul 2 15:29:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 15:29:22 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Hi all, with the next released of rsyslog v4/v5, I will introduce so-called "rulesets". They are effectively a sequence of rules that can directly be bound to an input (as long as the input supports this feature, currently not all do). I hope that this mechanism will simplify configuration. It will also enhance performance in most cases (as less filter logic is involved). A howto with detail description is available here: http://www.rsyslog.com/doc-multi_ruleset.html Rainer From tbergfeld at hq.adiscon.com Thu Jul 2 16:11:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 2 Jul 2009 16:11:15 +0200 Subject: [rsyslog] rsyslog 3.22.1 (v3-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACB@GRFEXC.intern.adiscon.com> Hi all, rsyslog 3.22.1, a member of the v3-stable branch, has been released today. The release mainly consists of bugfixes like a fix for an invalid error message issued if $inlcudeConfig was on an empty set of files. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-163.phtml Changelog: http://www.rsyslog.com/Article381.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From aoz.syn at gmail.com Thu Jul 2 17:13:35 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 2 Jul 2009 09:13:35 -0600 Subject: [rsyslog] v4/v5: new mult-ruleset support In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > I hope that this mechanism will simplify configuration. It will also enhance > performance in most cases (as less filter logic is involved). A howto with > detail description is available here: Very pleasing; this is an excellent step toward some of the enhanced configuration semantics I miss from syslog-ng. Perhaps I'm alone here, but I feel that the current syntax is detrimental to complex configurations, largely due to [understandable] compatibility adherence. Oh, to be a college student again and able to take on a SoC project to write a secondary configuration parser. From rgerhards at hq.adiscon.com Thu Jul 2 17:17:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 17:17:19 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Thursday, July 02, 2009 5:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] v4/v5: new mult-ruleset support > > On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > > I hope that this mechanism will simplify configuration. It will also enhance > > performance in most cases (as less filter logic is involved). A howto with > > detail description is available here: > > Very pleasing; this is an excellent step toward some of the enhanced > configuration semantics I miss from syslog-ng. Yeah, I started with the goal of full compatibility, but some complex things really get hard in that mode. > Perhaps I'm alone > here, but I feel that the current syntax is detrimental to complex > configurations, me, too ;) >largely due to [understandable] compatibility > adherence. Oh, to be a college student again and able to take on a > SoC project to write a secondary configuration parser. The script engine should bring this ... when I am finally able to go to it (doesn't look like I am anytime soon...). Rainer From tbergfeld at hq.adiscon.com Fri Jul 3 11:38:04 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 3 Jul 2009 11:38:04 +0200 Subject: [rsyslog] rsyslog 4.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAE8@GRFEXC.intern.adiscon.com> Hi all, Today, we have released rsyslog 4.5.0, a member of the development branch. This version offers improved performance and reduced memory requirements of the msg object. It further provides you with better config error messages, added capability to fsync() queue disc files for enhanced reliability, new configuration commands and stricter parsing of the hostname in rfc3164 mode. There are also some bug fixes included and the max value for $DynaFileCacheSize has been reduced to a more suitable value. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-164.phtml Changelog: http://www.rsyslog.com/Article380.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Mon Jul 6 14:03:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Jul 2009 14:03:23 +0200 Subject: [rsyslog] git branch changes Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB08@GRFEXC.intern.adiscon.com> Hi all, I made a mistake last Friday when creating the new releases. I accidently merged the v5-devel branch into the master branch, which was v4-devel up to then. I now looked at the result and think it is best to drop the v5-devel and create a new v4-devel branch (I can identify the wrong merge and will base v4-devel on it). This hopefully prevents any other such mistake in the future... Somehow my mind is on "master is always the most current". So please use v4-devel for the v4 tree and master for the v5-tree. Once v5 has become / confirmed to be sufficiently stable, I'll stop development in v4 and we will go back to the stable/beta/devel versions with only one active development version. I am right now preparing the new branches, will send a short mail when I am done. I just thought I let you know asap that something went wrong. Sorry for the hassle, Rainer From tbergfeld at hq.adiscon.com Wed Jul 8 16:48:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 8 Jul 2009 16:48:09 +0200 Subject: [rsyslog] rsyslog 5.1.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB43@GRFEXC.intern.adiscon.com> Hi all, We have released rsyslog 5.1.2, a new version of the development branch. This version offers improved performance and includes important fixes like a fix for an abort condition when RecvFrom was not set and message reduction was on, this happened e.g. with imuxsock. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-166.phtml Changelog: http://www.rsyslog.com/Article386.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Thu Jul 9 17:56:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Jul 2009 17:56:36 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Sorry folks, have not worked on this topic since March - but I never forget ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, March 04, 2009 8:17 AM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Ok, here is a diff that works. > I have integrated David's UDP spoofing patch into v5-devel (master branch) as a separate output module (named omudpspoof). I have extracted the necessary functionality from omfwd and it now works quite well. However, the configuration is not yet as typically found in rsyslog (not that I find it very elegant, but consistency is a plus ;)). Also, doc is missing. My next steps will be to address these issues. However, for those interested, I just wanted to tell you that the git master branch now contains a version capable of spoofing (I have verified this with both an rsyslog receiver as well as Wireshark - hopefully I can also add an automated test for this functionality, but this is not trivial). > it cycles the source IP address from 32000-42000 (since we are just > sending, and not creating a normal socket this should not matter) > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > to use it create a template that puts the hostname-ip ahead of what you > want to send, similar to > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > %HOSTNAME% %syslogtag%%msg%\n" > > *.* @10.0.0.100;TraditionalFwdFormat > > the one problem right now is that any logs sent from the local box will go > out with a source IP of 127.0.0.1 > > I wasted a bit of time trying to setup filters to use a different template > if $myhostname == $fromhost, but apparently the filtering doesn't allow > comparing two properties I overlooked this in march. The script-based filter engine does not even know that what is being compared are properties, so there is no restriction in what can be compared. So this must either have been a config issue or a bug. I just wanted to tell you should you come into a similar situation again. >, and then I realized that you have a very > high-performance name cache now, so you could easily replace my trivial > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > line with a call to the name lookup and then the %fromhost-ip% could be > replaced by %fromhost% in the template and everything would work sanely > (assuming forward and reverse name resolution are sane ;-) And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > the IP layer options, but I don't know exactly what yet. Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand shows up. Even then, I consider it to be low priority except given very good reasoning ;) Rainer > > I wanted to float this first to see what you think before spending much > more time on it. > > David Lang From doherty at crystal.harvard.edu Thu Jul 9 23:05:30 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Thu, 9 Jul 2009 17:05:30 -0400 Subject: [rsyslog] High CPU utlization, and memory usage Message-ID: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Hi, I recently deployed rsyslog on a server. I'm using a central server that receives logs from currently just 4 other machines. It's using TLS. It's a pretty basic setup, I didn't do anything fancy. Twice this week the central server has had issues, and when I logged in to check, I saw that rsyslog was using all the swap (2GB) and using over 300MB of the 512MB RAM, plus using 70% or more CPU time. I'm running version 4.2.0 Can you give me some idea where to start looking for what's causing this? It seems to run fine for a day or so, and then over a very short amount of time, maybe 2-60 minutes, the memory and cpu usage spikes. Here's a snippet from the /etc/rsyslog.conf if that helps. Thanks. --Peter $ModLoad immark $ModLoad imklog $ModLoad imuxsock # local messages $ModLoad imtcp # TCP listener $MarkMessagePeriod 1200 # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerRun 10514 # start up listener at port 10514 From david at lang.hm Thu Jul 9 23:35:30 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 9 Jul 2009 14:35:30 -0700 (PDT) Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: On Thu, 9 Jul 2009, Peter Doherty wrote: > Hi, I recently deployed rsyslog on a server. I'm using a central > server that receives logs from currently just 4 other machines. It's > using TLS. It's a pretty basic setup, I didn't do anything fancy. > > Twice this week the central server has had issues, and when I logged > in to check, I saw that rsyslog was using all the swap (2GB) and using > over 300MB of the 512MB RAM, plus using 70% or more CPU time. one thing when examining the load is to keep in mind that rsyslog uses multiple threads, in linux hit 'H' in top to have it show the individual threads. it's very possible that the high cpu load is due to the encryption overhead one thing that I do when testing is that once I identify which thread is eating a significant amount of cpu I do 'strace -p ' for that thread for a few seconds, looking at that output I can ususally make a fair guess at what the thread is busy working on. using that much ram would leave me guessing that the ability to write the log file stopped, and the queue is filling up. David Lang > I'm running version 4.2.0 > Can you give me some idea where to start looking for what's causing > this? It seems to run fine for a day or so, and then over a very > short amount of time, maybe 2-60 minutes, the memory and cpu usage > spikes. > > Here's a snippet from the /etc/rsyslog.conf if that helps. > > Thanks. > --Peter > > > $ModLoad immark > $ModLoad imklog > $ModLoad imuxsock # local messages > $ModLoad imtcp # TCP listener > > $MarkMessagePeriod 1200 > > # make gtls driver the default > $DefaultNetstreamDriver gtls > > # certificate files > $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem > $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem > $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem > > > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > $InputTCPServerRun 10514 # start up listener at port 10514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Jul 10 14:07:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 10 Jul 2009 14:07:24 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB66@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have completed this module now. It is part of the regular master branch. Some basic doc is available here: http://www.rsyslog.com/doc-omudpspoof.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, July 09, 2009 5:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Sorry folks, have not worked on this topic since March - but I never forget > ;) > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, March 04, 2009 8:17 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] UDP source forging. > > > > Ok, here is a diff that works. > > > > I have integrated David's UDP spoofing patch into v5-devel (master branch) as > a separate output module (named omudpspoof). I have extracted the necessary > functionality from omfwd and it now works quite well. However, the > configuration is not yet as typically found in rsyslog (not that I find it > very elegant, but consistency is a plus ;)). Also, doc is missing. My next > steps will be to address these issues. However, for those interested, I just > wanted to tell you that the git master branch now contains a version capable > of spoofing (I have verified this with both an rsyslog receiver as well as > Wireshark - hopefully I can also add an automated test for this > functionality, but this is not trivial). > > > it cycles the source IP address from 32000-42000 (since we are just > > sending, and not creating a normal socket this should not matter) > > > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > > > to use it create a template that puts the hostname-ip ahead of what you > > want to send, similar to > > > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > > %HOSTNAME% %syslogtag%%msg%\n" > > > > *.* @10.0.0.100;TraditionalFwdFormat > > > > the one problem right now is that any logs sent from the local box will go > > out with a source IP of 127.0.0.1 > > > > I wasted a bit of time trying to setup filters to use a different template > > if $myhostname == $fromhost, but apparently the filtering doesn't allow > > comparing two properties > > I overlooked this in march. The script-based filter engine does not even know > that what is being compared are properties, so there is no restriction in > what can be compared. So this must either have been a config issue or a bug. > I just wanted to tell you should you come into a similar situation again. > > >, and then I realized that you have a very > > high-performance name cache now, so you could easily replace my trivial > > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > > line with a call to the name lookup and then the %fromhost-ip% could be > > replaced by %fromhost% in the template and everything would work sanely > > (assuming forward and reverse name resolution are sane ;-) > > And another point to stress: rsyslogd does *not* yet have a high-performance > cache. All it does is cache the last host (and only the last host). This > works exceptionally well if a large bunch of messages arrive from the same > host, but "cache" performance can easily be thrashed with multiple senders. > All in all, in practice, it works reasonably well, as on a busy system at > least a couple of messages are usually from the same sender. I plan to add a > real cache some time later, personally I would hope to see it this summer. > > > > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > > the IP layer options, but I don't know exactly what yet. > > Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand > shows up. Even then, I consider it to be low priority except given very good > reasoning ;) > > Rainer > > > > > I wanted to float this first to see what you think before spending much > > more time on it. > > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From doherty at crystal.harvard.edu Fri Jul 10 16:04:27 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Fri, 10 Jul 2009 10:04:27 -0400 Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: <47B915CC-ACD1-496C-B91F-597EC84831E8@crystal.harvard.edu> On Jul 9, 2009, at 5:35 PM, david at lang.hm wrote: > > one thing when examining the load is to keep in mind that rsyslog uses > multiple threads, in linux hit 'H' in top to have it show the > individual > threads. > > it's very possible that the high cpu load is due to the encryption > overhead > > one thing that I do when testing is that once I identify which > thread is > eating a significant amount of cpu I do 'strace -p ' for that > thread > for a few seconds, looking at that output I can ususally make a fair > guess > at what the thread is busy working on. > > using that much ram would leave me guessing that the ability to > write the > log file stopped, and the queue is filling up. > > David Lang I'll have to do a little further testing, but I've got a hunch what caused this state. One of the computers sometimes gets into a state where it starts flooding its syslog with errors from a program. On the order of hundreds of messages a minute. I can correct this one case, but I can't guarantee some other system won't ever have some error that causes it to start flooding it's logs. Are there settings in rsyslog that can control this? Essentially I need something that will prevent a DoS style attack. Something that drops messages from the queue if they come in too fast from a certain host? Or I often see messages from syslogd systems which just say "Last message repeated n times" Can I enable that functionality in rsyslog? --Peter From ko at ibl.fr Fri Jul 10 17:04:25 2009 From: ko at ibl.fr (ko) Date: Fri, 10 Jul 2009 17:04:25 +0200 Subject: [rsyslog] One file per host logging Message-ID: <4A575879.3030308@ibl.fr> Hi, I'm a new-by in rsyslog. I want to do a centralized syslog server, and I want to know if it's possible to do a per host logging, I mean : all log comming from 192.168.1.1 going to /var/log/server1.log all log comming from 192.168.1.2 going to /var/log/server2.log ... and so on... Thanks for your help K. From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 10 17:17:53 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 10 Jul 2009 17:17:53 +0200 Subject: [rsyslog] One file per host logging In-Reply-To: <4A575879.3030308@ibl.fr> References: <4A575879.3030308@ibl.fr> Message-ID: <200907101717.56354.Luis.Fernando.Munoz.Mejias@cern.ch> Ko, > > I'm a new-by in rsyslog. I want to do a centralized syslog server, and > I want to know if it's possible to do a per host logging, I mean : Indeed you can. Just specify the file name as a template. For instance: $template FileNamePerHost,"/var/log/%HOSTNAME%.log" And then, store everything on the files specified by the template: *.* FileNamePerHost You can even have folders per day, just review the documentation on templates and properties. Cheers -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Mon Jul 13 14:17:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 14:17:04 +0200 Subject: [rsyslog] rsyslog - what's next? Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Hi all, I think we made some really good progress with rsyslog's capabilities (feature set & performance) in the past couple of month. I also think I have reached a milestone. As such, I've taken a bit time off to think about where to head to next. And as I'd like to have this available as a handy reference, I've blogged about. So if you are interested in what I intend to focus on the next time (maybe two to three month), please have a look at my blog post: http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html Of course, urgent things will have priority over the general goal, but I am pretty sure the planned effort will have a very positive effect on rsyslog's capabilities and qualities even in the medium term. As always, comments and questions are welcome. Thanks, Rainer From david at lang.hm Mon Jul 13 19:09:32 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 10:09:32 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 13 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I think we made some really good progress with rsyslog's capabilities > (feature set & performance) in the past couple of month. I also think I have > reached a milestone. As such, I've taken a bit time off to think about where > to head to next. And as I'd like to have this available as a handy reference, > I've blogged about. > > So if you are interested in what I intend to focus on the next time (maybe > two to three month), please have a look at my blog post: > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > Of course, urgent things will have priority over the general goal, but I am > pretty sure the planned effort will have a very positive effect on rsyslog's > capabilities and qualities even in the medium term. > > As always, comments and questions are welcome. I don't think that you need to do much for UDP recieve performance. as long as it doesn't need to do DNS lookups it can recieve and insert into a memory queue at full gig-E speeds. the only think you may want to do here is to extend the DNS cache so that instead of only caching the last think you looked up, you cache everything until a restart or HUP (ideally the HUP should be a configurable option) the machines I had on order with the 10G interfaces were misconfigured by the vendor, so I don't have the 10G cards yet :-( when I get them I'll do more testing and see how much faster I can push rsyslog. David Lang From aoz.syn at gmail.com Mon Jul 13 19:38:17 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 11:38:17 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> On Mon, Jul 13, 2009 at 11:09, wrote: > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) I haven't looked at the DNS cache code, but unless you're also caching and handling the records' TTL, blindly caching DNS records within the app is a dark road to go down. Some apps (namely browsers) do it anyway, but get away with it by setting their internal TTL significantly lower than that of a typical record. What kind of performance hit are you actually seeing with DNS lookups? From david at lang.hm Mon Jul 13 20:11:02 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 11:11:02 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 11:09, wrote: >> is to extend the DNS cache so that instead of only caching the last think >> you looked up, you cache everything until a restart or HUP (ideally the >> HUP should be a configurable option) > > I haven't looked at the DNS cache code, but unless you're also caching > and handling the records' TTL, blindly caching DNS records within the > app is a dark road to go down. Some apps (namely browsers) do it > anyway, but get away with it by setting their internal TTL > significantly lower than that of a typical record. for Internet access I completely agree with you, but for a log server you don't have IPs changing names very frequently. As a result it becomes practical to cache the lookup unconditionally until a restart/HUP. even in a 'highly dynamic' environment you are usually adding servers instead of changing the names of IP addresses. note that caching the lookup at all can be disabled via a config option > What kind of performance hit are you actually seeing with DNS lookups? 6 months ago it was a factor of 10x to 100x (it's probably more now since rsyslog has gotten faster) the problem is that to do a name lookup requires a LOT of steps first the name resolver library looks several places on the filesystem for various config files then it reads /etc/hosts, parses it, and looks for the name/IP then it makes a network connection to the nameserver, sends a message, waits for the reciever to parse the message and look it up in it's datastore, then send the message back to the requester who needs to parse the message and if you network packet gets dropped due to congestion on the network, you wait 30 seconds and retry. doing all of this for every UDP syslog packet that you recieve results in a _lot_ of system calls, and a lot of delays. even if you have the name in the /etc/hosts file, the overhead of looking in the filesystem and parsing the file is significant. without doing DNS lookups, rsyslog is able to recieve UDP logs at almost 400,000 per second (effectivly Gig-E wire speed with 256 byte log messages), _very_ few DNS servers can handle requests at that speed. in fact, at that request rate, you use about half of a Gig-E connection just for the DNS requests (more if you request more data, like what the TTL would be, or don't give it the best name the first time and need to make multiple requests with different domains attached or something like that) David Lang From aoz.syn at gmail.com Mon Jul 13 21:43:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 13:43:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> On Mon, Jul 13, 2009 at 12:11, wrote: > for Internet access I completely agree with you, but for a log server you > don't have IPs changing names very frequently. As a result it becomes > practical to cache the lookup unconditionally until a restart/HUP. even in > a 'highly dynamic' environment you are usually adding servers instead of > changing the names of IP addresses. The problem with that approach is that it assumes it is acceptable to interrupt service for any trivial DNS change. How do you propose to deal withDNS-level availability tools like round-robin or Cisco's GSS? There are many approaches that use DNS as both a scalability and a redundancy tool that TTL-agnostic caching will not interact well with. > the problem is that to do a name lookup requires a LOT of steps I'm well aware of the series of steps, but am unconvinced caching belongs in the application. Have you tried using local caching (nscd) or segregating the DNS traffic from the production syslog traffic? Most enterprise-grade configurations (for whatever app) I've seen use a separate interface for administration and metadata like reverse lookups anyway. > even if you have the name in the /etc/hosts file, the overhead of looking > in the filesystem and parsing the file is significant. How significant? Unless the host is poorly configured, /etc/hosts is going to be in the filesystem cache and presented at near-memory speeds until it gets invalidated or evicted. > without doing DNS lookups, rsyslog is able to recieve UDP logs at almost > 400,000 per second (effectivly Gig-E wire speed with 256 byte log > messages), _very_ few DNS servers can handle requests at that speed. in To clarify since I'm not looking at the code right now: are you saying rsyslog performs a blocking lookup per-packet? If so that's certainly a sub-optimal approach, but there are often better ways (namely delayed resolution) to fix that than breaking RFC compatibility. > fact, at that request rate, you use about half of a Gig-E connection just > for the DNS requests (more if you request more data, like what the TTL > would be, or don't give it the best name the first time and need to make > multiple requests with different domains attached or something like that) Your example invalidates itself - the TTL comes with the query whether or not you use it, and any caching mechanism (TTL-sensitive or not) will shortcut such sub-optimal queries after the first one. Caching is a good thing as long as it's done in a compliant manner or documented in a fashion that clearly identifies it as a broken (if performant) approach. From ktm at rice.edu Mon Jul 13 21:37:57 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 13 Jul 2009 14:37:57 -0500 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <20090713193757.GB4988@it.is.rice.edu> On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I think we made some really good progress with rsyslog's capabilities > > (feature set & performance) in the past couple of month. I also think I have > > reached a milestone. As such, I've taken a bit time off to think about where > > to head to next. And as I'd like to have this available as a handy reference, > > I've blogged about. > > > > So if you are interested in what I intend to focus on the next time (maybe > > two to three month), please have a look at my blog post: > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > Of course, urgent things will have priority over the general goal, but I am > > pretty sure the planned effort will have a very positive effect on rsyslog's > > capabilities and qualities even in the medium term. > > > > As always, comments and questions are welcome. > > I don't think that you need to do much for UDP recieve performance. as > long as it doesn't need to do DNS lookups it can recieve and insert into a > memory queue at full gig-E speeds. the only think you may want to do here > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) > > the machines I had on order with the 10G interfaces were misconfigured by > the vendor, so I don't have the 10G cards yet :-( when I get them I'll do > more testing and see how much faster I can push rsyslog. > > David Lang I agree with David. DNS lookups can really sap both bandwidth and CPU. I am not sure that caching everything since a restart is a good idea though. Even if you could do a lookup, cache the TTL and IP and then relookup after the TTL that would be great. Even having it re-query the DNS in a worker thread to sweep through the cached values periodically would really help performance. Regards, Ken From rgerhards at hq.adiscon.com Mon Jul 13 22:25:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 22:25:07 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <20090713193757.GB4988@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB80@GRFEXC.intern.adiscon.com> One important point to stress is that rsyslog does NOT yet have a real DNS cache! Let me quote my mail from July, 9th[1]: == And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. == Adding a the cache is not really a focus activity - I think it could be done in a couple of days. I also personally do not see an issue with obying TTL - or discarding entries every five minutes or so. In my POV, it doesn't really matter if they are quickly discarded. If there is low traffic, it doesn't make a difference at all. If we handle hundereds of thousands message per second, one lockup every few billion of messages doesn't really matter either. The compare operation is relatively fast and will not add significantly to the runtime (at least I'd guess that). In regard to imudp, I think there is a couple of more optimizations possible. I'd expect that the biggest hit (after DNS lookup) is switching from select to epoll. Re-using property copying is another one (reduces expensive memory writes). A batching interface seems also possible. Enhanced pipelining is another possibility. I also think that the current solution will gain very different results based on the communication patterns. So rather than (educated) guessing, I would prefer to have some good instrumentation to get a grip on all this variants. Also, imudp is just a sample, there are more areas that could benefit from a focussed analytic view. Plus that performance tools can be used to stress-testing and thus finding otherwise hard to find bugs. Thus I think it is worth spending some time on an improved toolset (and will pay off in the long term). Rainer [1] http://lists.adiscon.net/pipermail/rsyslog/2009-July/002432.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Monday, July 13, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > > > Hi all, > > > > > > I think we made some really good progress with rsyslog's > capabilities > > > (feature set & performance) in the past couple of month. > I also think I have > > > reached a milestone. As such, I've taken a bit time off > to think about where > > > to head to next. And as I'd like to have this available > as a handy reference, > > > I've blogged about. > > > > > > So if you are interested in what I intend to focus on the > next time (maybe > > > two to three month), please have a look at my blog post: > > > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > > > Of course, urgent things will have priority over the > general goal, but I am > > > pretty sure the planned effort will have a very positive > effect on rsyslog's > > > capabilities and qualities even in the medium term. > > > > > > As always, comments and questions are welcome. > > > > I don't think that you need to do much for UDP recieve > performance. as > > long as it doesn't need to do DNS lookups it can recieve > and insert into a > > memory queue at full gig-E speeds. the only think you may > want to do here > > is to extend the DNS cache so that instead of only caching > the last think > > you looked up, you cache everything until a restart or HUP > (ideally the > > HUP should be a configurable option) > > > > the machines I had on order with the 10G interfaces were > misconfigured by > > the vendor, so I don't have the 10G cards yet :-( when I > get them I'll do > > more testing and see how much faster I can push rsyslog. > > > > David Lang > > I agree with David. DNS lookups can really sap both bandwidth and CPU. > I am not sure that caching everything since a restart is a good idea > though. Even if you could do a lookup, cache the TTL and IP and then > relookup after the TTL that would be great. Even having it re-query > the DNS in a worker thread to sweep through the cached values > periodically > would really help performance. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Jul 13 22:38:52 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 13:38:52 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 12:11, wrote: >> for Internet access I completely agree with you, but for a log server you >> don't have IPs changing names very frequently. As a result it becomes >> practical to cache the lookup unconditionally until a restart/HUP. even in >> a 'highly dynamic' environment you are usually adding servers instead of >> changing the names of IP addresses. > > The problem with that approach is that it assumes it is acceptable to > interrupt service for any trivial DNS change. How do you propose to > deal withDNS-level availability tools like round-robin or Cisco's GSS? > There are many approaches that use DNS as both a scalability and a > redundancy tool that TTL-agnostic caching will not interact well with. HUP does not interrupt services. >> the problem is that to do a name lookup requires a LOT of steps > > I'm well aware of the series of steps, but am unconvinced caching > belongs in the application. Have you tried using local caching (nscd) > or segregating the DNS traffic from the production syslog traffic? > Most enterprise-grade configurations (for whatever app) I've seen use > a separate interface for administration and metadata like reverse > lookups anyway. yes, doing a local DNS cacheing server does not avoid all the local filesystem lookups and host file parsing. it does speed up the network connection. >> even if you have the name in the /etc/hosts file, the overhead of looking >> in the filesystem and parsing the file is significant. > > How significant? Unless the host is poorly configured, /etc/hosts is > going to be in the filesystem cache and presented at near-memory > speeds until it gets invalidated or evicted. the system calls to access it (and to first lookup if it should be accessed at all) take enough time to be significant at these speeds. >> without doing DNS lookups, rsyslog is able to recieve UDP logs at almost >> 400,000 per second (effectivly Gig-E wire speed with 256 byte log >> messages), _very_ few DNS servers can handle requests at that speed. in > > To clarify since I'm not looking at the code right now: are you saying > rsyslog performs a blocking lookup per-packet? If so that's certainly > a sub-optimal approach, but there are often better ways (namely > delayed resolution) to fix that than breaking RFC compatibility. correct, it does all input processing in a single thread. so if that thread is stalled doing a lookup it's not processing other packets. when the data is first inserted into the message queue it is complete with all lookups done. >> fact, at that request rate, you use about half of a Gig-E connection just >> for the DNS requests (more if you request more data, like what the TTL >> would be, or don't give it the best name the first time and need to make >> multiple requests with different domains attached or something like that) > > Your example invalidates itself - the TTL comes with the query whether > or not you use it, and any caching mechanism (TTL-sensitive or not) > will shortcut such sub-optimal queries after the first one. > > Caching is a good thing as long as it's done in a compliant manner or > documented in a fashion that clearly identifies it as a broken (if > performant) approach. you want the ability to cache the name lookup no matter what method is used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include a TTL. I agree that the details of how the cache works should be documented. David Lang From aoz.syn at gmail.com Mon Jul 13 23:24:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 15:24:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> On Mon, Jul 13, 2009 at 14:38, wrote: > HUP does not interrupt services. I'm sorry; last October you noted that for certain configurations SIGHUP will drop memory-queued messages. In addition to the other notes in the thread (tearing down TCP connections, etc.), it sounds an awful lot like a service interruption. Has that since changed for 3.x or 4.2? > the system calls to access it (and to first lookup if it should be > accessed at all) take enough time to be significant at these speeds. May I gently prod you to substantiate this statement? I don't doubt that it makes calls to external libraries, and that those calls are likely more expensive than internal resolution, but "significant" is not significant without numbers to back it up. Even a simple speed comparison for a few million packets between 'no resolution' and 'in /etc/hosts with a cold cache' would be useful. > you want the ability to cache the name lookup no matter what method is > used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > a TTL. I have no problem expanding the scope of discussion to resolution methods beyond DNS (which was the original premise), but if a name service does not provide an explicit TTL, it must be assumed as an implicit '0', which blind caching will break. That said, each of those methods still provides some [internal] method of caching and invalidation that, to be audit-grade correct, rsyslog will either have to replicate or trust. I still can't see a problem with having something to the effect of a "$BreakNSCache" configuration option, but by default a syslog daemon should play as strictly by the rules as possible. Then again, I'm RB, not RG. ;) From david at lang.hm Mon Jul 13 23:58:33 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 14:58:33 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 14:38, wrote: >> HUP does not interrupt services. > > I'm sorry; last October you noted that for certain configurations > SIGHUP will drop memory-queued messages. In addition to the other > notes in the thread (tearing down TCP connections, etc.), it sounds an > awful lot like a service interruption. Has that since changed for 3.x > or 4.2? yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not do a complete teardown, halt, and restart, but instead just closes and re-opens files and connections so that no long messages are lost in a restart. this matches the traditional use of HUP to get syslogd to release the files it's writing to so that they can be rotated away. it doesn't re-read the config file due to the fact that the rsyslog config file is so complex and can significantly alter the software by loading modules. >> the system calls to access it (and to first lookup if it should be >> accessed at all) take enough time to be significant at these speeds. > > May I gently prod you to substantiate this statement? I don't doubt > that it makes calls to external libraries, and that those calls are > likely more expensive than internal resolution, but "significant" is > not significant without numbers to back it up. Even a simple speed > comparison for a few million packets between 'no resolution' and 'in > /etc/hosts with a cold cache' would be useful. I would have to do a new set of tests to get you concrete numbers, but back in roughly the october timeframe last year I was doing extensive tests on this and with hosts files vs. no resolution I was seeing 4x or more differences (with a tiny, 5-line hosts file to parse) at the time I think I discussed it on a long performance thread on the website. at the time I did quite a number of strace dumps to show how much time was being eaten up in the system calls. RG has done a LOT of cleanup since then (to the point that the failsafe audit mode is in the ballpark of being as fast as the in-memory mode was back then) >> you want the ability to cache the name lookup no matter what method is >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include >> a TTL. > > I have no problem expanding the scope of discussion to resolution > methods beyond DNS (which was the original premise), but if a name > service does not provide an explicit TTL, it must be assumed as an > implicit '0', which blind caching will break. That said, each of > those methods still provides some [internal] method of caching and > invalidation that, to be audit-grade correct, rsyslog will either have > to replicate or trust. I still can't see a problem with having > something to the effect of a "$BreakNSCache" configuration option, but > by default a syslog daemon should play as strictly by the rules as > possible. well, if you are really wanting audit-grade logging, will you use anything other than the IP address (or what is already in the message)? any lookup that you do is a potential for corruption. I'm fine with the default being to do a lookup for each item. I just want a way to avoid it, and if I can do that with a cache instead of having to disable DNS, I will do so. it's very common for servers to need to disable DNS lookups for their logs. do a quick search for apache dns performance and you will find that it's a very common problem people have there if they enable lookups on the sender's IP address. > Then again, I'm RB, not RG. ;) more people speaking up is good. I have strong opinions, and real problems to address, but RG needs to hear from people other than me. David Lang From aoz.syn at gmail.com Tue Jul 14 05:12:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 21:12:57 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> On Mon, Jul 13, 2009 at 15:58, wrote: > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. Hopefully some of the tooling Rainer is considering will better enable performance monitoring and we can test the effect of some of these suggested changes. I don't completely discount strace as a performance testing tool, but it's inherently intrusive, making Heisenberg quite happy. > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address Indeed; I've generally eschewed DNS for such critical pieces of infrastructure, instead opting for resolution during analysis and correlation with other logs. That largely stems from network security work and the associated disdain for using DNS entries for rules. Even so, I'm curious whether a properly tuned library-integrated cache like nscd (as opposed to a local caching resolver) would provide a sufficient performance increase in the interim. I spent a couple of hours this evening digging through the related code with the intent of starting work on delayed resolution, moving the lookup to runtime/msg.c:getHOSTNAME. It might not be the right place, but would have the advantage of only resolving hosts you explicitly request it for. Unfortunately, the AllowedSender pragma is a sticking point, forcing per-packet resolution; otherwise, it seemed a pretty trivial conversion. I don't know how much (if any) performance increase that approach would grant, but it would at least move a blocking call out of that single thread. I may experiment with adding a "NoAllowedSenderDNS" later, but it's beyond my time flexibility right now. From david at lang.hm Tue Jul 14 06:54:37 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 21:54:37 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 15:58, wrote: >> at the time I did quite a number of strace dumps to show how much time was >> being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. absolutly, especially when you enable timing of the calls !!! (a gettimeofday call before and after evey syscall) but if the program hasn't been examined to try and reduce the number of syscalls it makes, strace is a good place to start. once you get down to a minimum of nessasary syscalls, strace stops being useful. >> well, if you are really wanting audit-grade logging, will you use anything >> other than the IP address (or what is already in the message)? any lookup >> that you do is a potential for corruption. >> >> I'm fine with the default being to do a lookup for each item. I just want >> a way to avoid it, and if I can do that with a cache instead of having to >> disable DNS, I will do so. >> >> it's very common for servers to need to disable DNS lookups for their >> logs. do a quick search for apache dns performance and you will find that >> it's a very common problem people have there if they enable lookups on the >> sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. I'll try and get a chance to look into that > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. interesting thought. David Lang From rgerhards at hq.adiscon.com Tue Jul 14 07:17:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 07:17:24 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB82@GRFEXC.intern.adiscon.com> I am digesting this thread (just got up ;)), but let me spell out something explicitely: the traditional perception of what is fast and what is slow is no longer valid if you aim at a rate of several hunderd-thousand messages per second. For example, it then makes a difference if you (memory) write a single 32 byte buffer per record or not. It may be in the region of 100 to 500 records per second, but these differences sum up. So trying to avoid such writes, even a the costs of (currently much cheaper) reads is something to aim at. I'll see that I get together a more elaborate description inside my blog, probably it is useful. I learned a lot of lessons the past nine month ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, July 14, 2009 6:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 15:58, wrote: > >> at the time I did quite a number of strace dumps to show > how much time was > >> being eaten up in the system calls. > > > > Hopefully some of the tooling Rainer is considering will > better enable > > performance monitoring and we can test the effect of some of these > > suggested changes. I don't completely discount strace as a > > performance testing tool, but it's inherently intrusive, making > > Heisenberg quite happy. > > absolutly, especially when you enable timing of the calls !!! (a > gettimeofday call before and after evey syscall) > > but if the program hasn't been examined to try and reduce the > number of > syscalls it makes, strace is a good place to start. > > once you get down to a minimum of nessasary syscalls, strace > stops being > useful. > > >> well, if you are really wanting audit-grade logging, will > you use anything > >> other than the IP address (or what is already in the > message)? any lookup > >> that you do is a potential for corruption. > >> > >> I'm fine with the default being to do a lookup for each > item. I just want > >> a way to avoid it, and if I can do that with a cache > instead of having to > >> disable DNS, I will do so. > >> > >> it's very common for servers to need to disable DNS > lookups for their > >> logs. do a quick search for apache dns performance and you > will find that > >> it's a very common problem people have there if they > enable lookups on the > >> sender's IP address > > > > Indeed; I've generally eschewed DNS for such critical pieces of > > infrastructure, instead opting for resolution during analysis and > > correlation with other logs. That largely stems from > network security > > work and the associated disdain for using DNS entries for > rules. Even > > so, I'm curious whether a properly tuned library-integrated > cache like > > nscd (as opposed to a local caching resolver) would provide a > > sufficient performance increase in the interim. > > I'll try and get a chance to look into that > > > I spent a couple of hours this evening digging through the related > > code with the intent of starting work on delayed resolution, moving > > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > > place, but would have the advantage of only resolving hosts you > > explicitly request it for. Unfortunately, the > AllowedSender pragma is > > a sticking point, forcing per-packet resolution; otherwise, > it seemed > > a pretty trivial conversion. I don't know how much (if any) > > performance increase that approach would grant, but it > would at least > > move a blocking call out of that single thread. I may > experiment with > > adding a "NoAllowedSenderDNS" later, but it's beyond my time > > flexibility right now. > > interesting thought. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 10:37:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 10:37:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, July 13, 2009 11:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 14:38, wrote: > >> HUP does not interrupt services. > > > > I'm sorry; last October you noted that for certain configurations > > SIGHUP will drop memory-queued messages. In addition to the other > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > awful lot like a service interruption. Has that since changed for 3.x > > or 4.2? > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not > do a complete teardown, halt, and restart, but instead just closes and > re-opens files and connections so that no long messages are lost in a > restart. And that is the default setting! > > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. For v5, I actually think of adding a "rsyslogd restart deamon" for those that insist on traditional HUP semantics. That deamon would lie dormant in memory until it receives HUP, in which case it does a shutdown and restart of the real rsyslogd. With that, I could cleanup a lot of complexity. One major issue is that under some circumstances I need to cancel output threads when they do not finish their processing within the allotted timeouts. Canceling threads is always a bit dangerous, but there currently is no always-reliable way around this. To make it happen, a lot of enable/disable cancel calls, including cancel cleanup handlers are made throughout the code. If we would not have restart-type HUPs, I could simply cancel those threads and not care about resources not being cleaned up (the process cleanup will take care of that). So I could also remove all the enable/disable cancel logic and its helpers. Of course, this is not something done as quickly as I write those lines and it must be made sure that any inconsistence does not affect the rest of the shutdown. But without those annoying restart-type HUPs, it is much simpler to do... > > >> the system calls to access it (and to first lookup if it should be > >> accessed at all) take enough time to be significant at these speeds. > > > > May I gently prod you to substantiate this statement? I don't doubt > > that it makes calls to external libraries, and that those calls are > > likely more expensive than internal resolution, but "significant" is > > not significant without numbers to back it up. Even a simple speed > > comparison for a few million packets between 'no resolution' and 'in > > /etc/hosts with a cold cache' would be useful. > > I would have to do a new set of tests to get you concrete numbers, but > back in roughly the october timeframe last year I was doing extensive > tests on this and with hosts files vs. no resolution I was seeing 4x or > more differences (with a tiny, 5-line hosts file to parse) > > at the time I think I discussed it on a long performance thread on the > website. > > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. RG has done a LOT of cleanup since > then (to the point that the failsafe audit mode is in the ballpark of > being as fast as the in-memory mode was back then) > > >> you want the ability to cache the name lookup no matter what method is > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > >> a TTL. > > > > I have no problem expanding the scope of discussion to resolution > > methods beyond DNS (which was the original premise), but if a name > > service does not provide an explicit TTL, it must be assumed as an > > implicit '0', which blind caching will break. That said, each of > > those methods still provides some [internal] method of caching and > > invalidation that, to be audit-grade correct, rsyslog will either have > > to replicate or trust. I still can't see a problem with having > > something to the effect of a "$BreakNSCache" configuration option, but > > by default a syslog daemon should play as strictly by the rules as > > possible. > > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address. > > > Then again, I'm RB, not RG. ;) > > more people speaking up is good. I have strong opinions, and real problems > to address, but RG needs to hear from people other than me. Definitely! I often simply do not realize where a need is, and I always often make mistakes or have wrong perceptions. If nobody objects, I am lost and so is the project ;) Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 11:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 11:02:40 +0200 Subject: [rsyslog] DNS performance - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Tuesday, July 14, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 15:58, wrote: > > at the time I did quite a number of strace dumps to show how much time was > > being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. That's exactly part of the "new tools" idea, but obviously this instrumentation will also affect the overall picture. But I think I can do the base counters with a few atomic increments, so the effect should be not that bad. The real work shall then be done in the front-end. > > > well, if you are really wanting audit-grade logging, will you use anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on the > > sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. > > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. That's indeed an interesting thought. The allowedSenders is not really that much of a problem. First of all, not all inputs support them (one may argue they should, but one may also argue that a firewall is a much better place to block traffic...). Secondly, $AllowedSender is not used in most configurations. Rsyslog can detect that the sender name is not needed and in this case can actually defer the name lookup. Of course, that doesn't help in situations where the lookup is required, but it helps in many configurations. This is also the spirit of some of my recent performance enhancements: not all work equally well under all circumstances. But I have tried to make only those suffer that actually need the feature that bears the cost. I hope to extend that idea to the queue, where an ultra-fast mode may automatically be selected if the parameters indicate that is possible. But, again, some tooling to test this and its affects would be quite useful. Rainer From iamsayan at gmail.com Tue Jul 14 17:52:53 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:52:53 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Sorry if this is out of context. > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. Is there any other way to make the rsyslog process reread the conf file other than a restart? This is the only reason I use the traditional SIGHUP processing, I send HUP after I alter the rsyslog.conf file. On Tue, Jul 14, 2009 at 4:37 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Monday, July 13, 2009 11:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog - what's next? > > > > On Mon, 13 Jul 2009, RB wrote: > > > > > On Mon, Jul 13, 2009 at 14:38, wrote: > > >> HUP does not interrupt services. > > > > > > I'm sorry; last October you noted that for certain configurations > > > SIGHUP will drop memory-queued messages. In addition to the other > > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > > awful lot like a service interruption. Has that since changed for 3.x > > > or 4.2? > > > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP > not > > do a complete teardown, halt, and restart, but instead just closes and > > re-opens files and connections so that no long messages are lost in a > > restart. > > And that is the default setting! > > > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. > However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > For v5, I actually think of adding a "rsyslogd restart deamon" for those > that > insist on traditional HUP semantics. That deamon would lie dormant in > memory > until it receives HUP, in which case it does a shutdown and restart of the > real rsyslogd. With that, I could cleanup a lot of complexity. > > One major issue is that under some circumstances I need to cancel output > threads when they do not finish their processing within the allotted > timeouts. Canceling threads is always a bit dangerous, but there currently > is > no always-reliable way around this. To make it happen, a lot of > enable/disable cancel calls, including cancel cleanup handlers are made > throughout the code. If we would not have restart-type HUPs, I could simply > cancel those threads and not care about resources not being cleaned up (the > process cleanup will take care of that). So I could also remove all the > enable/disable cancel logic and its helpers. Of course, this is not > something > done as quickly as I write those lines and it must be made sure that any > inconsistence does not affect the rest of the shutdown. But without those > annoying restart-type HUPs, it is much simpler to do... > > > > >> the system calls to access it (and to first lookup if it should be > > >> accessed at all) take enough time to be significant at these speeds. > > > > > > May I gently prod you to substantiate this statement? I don't doubt > > > that it makes calls to external libraries, and that those calls are > > > likely more expensive than internal resolution, but "significant" is > > > not significant without numbers to back it up. Even a simple speed > > > comparison for a few million packets between 'no resolution' and 'in > > > /etc/hosts with a cold cache' would be useful. > > > > I would have to do a new set of tests to get you concrete numbers, but > > back in roughly the october timeframe last year I was doing extensive > > tests on this and with hosts files vs. no resolution I was seeing 4x or > > more differences (with a tiny, 5-line hosts file to parse) > > > > at the time I think I discussed it on a long performance thread on the > > website. > > > > at the time I did quite a number of strace dumps to show how much time > was > > being eaten up in the system calls. RG has done a LOT of cleanup since > > then (to the point that the failsafe audit mode is in the ballpark of > > being as fast as the in-memory mode was back then) > > > > >> you want the ability to cache the name lookup no matter what method is > > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not > include > > >> a TTL. > > > > > > I have no problem expanding the scope of discussion to resolution > > > methods beyond DNS (which was the original premise), but if a name > > > service does not provide an explicit TTL, it must be assumed as an > > > implicit '0', which blind caching will break. That said, each of > > > those methods still provides some [internal] method of caching and > > > invalidation that, to be audit-grade correct, rsyslog will either have > > > to replicate or trust. I still can't see a problem with having > > > something to the effect of a "$BreakNSCache" configuration option, but > > > by default a syslog daemon should play as strictly by the rules as > > > possible. > > > > well, if you are really wanting audit-grade logging, will you use > anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on > the > > sender's IP address. > > > > > Then again, I'm RB, not RG. ;) > > > > more people speaking up is good. I have strong opinions, and real > problems > > to address, but RG needs to hear from people other than me. > > Definitely! I often simply do not realize where a need is, and I always > often > make mistakes or have wrong perceptions. If nobody objects, I am lost and > so > is the project ;) > > Rainer > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 17:55:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 17:55:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Sorry if this is out of context. > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > > Is there any other way to make the rsyslog process reread the conf file > other than a restart? > This is the only reason I use the traditional SIGHUP processing, I send HUP > after I alter the rsyslog.conf file. No, but I always fail to see the big difference between typing $ kill -HUP ?cat /var/run/rsyslog.log? And $ /etc/rc.d/rsyslogd restart ;) Use the later one, and you do not need to use sighup. Even better, you can now use HUP for a lightweight "close the files only" log rotation thing ;) Rainer From iamsayan at gmail.com Tue Jul 14 17:58:58 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:58:58 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Yes exactly, I have changed the script to do just that recently , I was just wondering whether this is the reason why you get so much resistance when you want to deprecate the traditional HUP processing... On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > next? > > > > Sorry if this is out of context. > > > > > this matches the traditional use of HUP to get syslogd to release the > > > files it's writing to so that they can be rotated away. > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > > file is so complex and can significantly alter the software by loading > > > modules. > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > the > > root cause for a lot of complexities (read performance bottlenecks) > inside > > the engine. And all this for something that is not strictly needed. > However, > > there are always many opponents when I intend to remove the traditional > HUP > > behavior. > > > > > > Is there any other way to make the rsyslog process reread the conf file > > other than a restart? > > This is the only reason I use the traditional SIGHUP processing, I send > HUP > > after I alter the rsyslog.conf file. > > > No, but I always fail to see the big difference between typing > > $ kill -HUP ?cat /var/run/rsyslog.log? > > And > > $ /etc/rc.d/rsyslogd restart > > ;) > > Use the later one, and you do not need to use sighup. Even better, you can > now use HUP for a lightweight "close the files only" log rotation thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 18:06:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:06:38 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Yes exactly, I have changed the script to do just that recently , I was just > wondering whether this is the reason why you get so much resistance when you > want to deprecate the traditional HUP processing... I think it is maily a "tradition" argument along the lines "but I am used to...". Still hard to beat if everybody complains ;) I don't see any technicaly reasons, especially as a restart-type restart IS a restart (nothing less), just carried out in the most complex way ;) Rainer > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > wrote: > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > next? > > > > > > Sorry if this is out of context. > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > files it's writing to so that they can be rotated away. > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > config > > > > file is so complex and can significantly alter the software by loading > > > > modules. > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > > the > > > root cause for a lot of complexities (read performance bottlenecks) > > inside > > > the engine. And all this for something that is not strictly needed. > > However, > > > there are always many opponents when I intend to remove the traditional > > HUP > > > behavior. > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > other than a restart? > > > This is the only reason I use the traditional SIGHUP processing, I send > > HUP > > > after I alter the rsyslog.conf file. > > > > > > No, but I always fail to see the big difference between typing > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > And > > > > $ /etc/rc.d/rsyslogd restart > > > > ;) > > > > Use the later one, and you do not need to use sighup. Even better, you can > > now use HUP for a lightweight "close the files only" log rotation thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 18:17:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:17:59 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com><87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB99@GRFEXC.intern.adiscon.com> Oh, and I forgot to mention that recent builds loudly complain about config errors on stderr when you do a real restart bit silently need to dump these error messages to their respective bins (where experience tells nobody sees them...) when doing restart-type HUP. Maybe the result of this discussion is that I should really drop that misfeature in v5. Any violent opponents? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, July 14, 2009 6:07 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > > Yes exactly, I have changed the script to do just that recently , I was > just > > wondering whether this is the reason why you get so much resistance when > you > > want to deprecate the traditional HUP processing... > > I think it is maily a "tradition" argument along the lines "but I am used > to...". Still hard to beat if everybody complains ;) I don't see any > technicaly reasons, especially as a restart-type restart IS a restart > (nothing less), just carried out in the most complex way ;) > > Rainer > > > > > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > > wrote: > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > > next? > > > > > > > > Sorry if this is out of context. > > > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > > files it's writing to so that they can be rotated away. > > > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > > config > > > > > file is so complex and can significantly alter the software by > loading > > > > > modules. > > > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it > is > > > > the > > > > root cause for a lot of complexities (read performance bottlenecks) > > > inside > > > > the engine. And all this for something that is not strictly needed. > > > However, > > > > there are always many opponents when I intend to remove the traditional > > > HUP > > > > behavior. > > > > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > > other than a restart? > > > > This is the only reason I use the traditional SIGHUP processing, I send > > > HUP > > > > after I alter the rsyslog.conf file. > > > > > > > > > No, but I always fail to see the big difference between typing > > > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > > > And > > > > > > $ /etc/rc.d/rsyslogd restart > > > > > > ;) > > > > > > Use the later one, and you do not need to use sighup. Even better, you > can > > > now use HUP for a lightweight "close the files only" log rotation thing > ;) > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 15 12:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Jul 2009 12:06:53 +0200 Subject: [rsyslog] HUP Processing - restart type will go away Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBAA@GRFEXC.intern.adiscon.com> Hi all, based on yesterday's discussion, and review of the (partly new) arguments, I have decided to go through the hassle of finally deprecating the restart-type HUP. I have changed the v3 doc so that it no longer explicitly recommends using HUP to apply config changes. I have added a compatibility doc to v4 which explains why HUP will go away and when. I invite you to read this document: http://www.rsyslog.com/doc-v4compatibility.html With the recent v4-devel (4.5.1, just released), I have also changed the default of $HUPisRestart to "off". In v5, I have added another compatibility document. Finally, I will remove the restart-type HUP in v5 soon, starting with the directive go away and later actually removing the (lots of) code that support it (and finally getting the benefits). Rainer From tbergfeld at hq.adiscon.com Wed Jul 15 14:30:23 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 15 Jul 2009 14:30:23 +0200 Subject: [rsyslog] rsyslog 4.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBB1@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 4.5.1, a member of the development branch. This version offers improved performannce. There were also some bugfixes like a fix for a potential segfault when zip-compressed syslog records were received. It further provides the ability for the TCP output action to "rebind" its send socket after sending n messages (actually, it re-opens the connection, the name is used because this is a concept very similiar to $ActionUDPRebindInterval). This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml Changelog: http://www.rsyslog.com/Article388.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From alex at segv.de Wed Jul 15 11:20:30 2009 From: alex at segv.de (Alexander Elbs) Date: Wed, 15 Jul 2009 11:20:30 +0200 Subject: [rsyslog] Use precision timestamps Message-ID: <20090715092030.GC19320@segv.de> Hi, when using syslog(3) an application can send log messages via /dev/log to rsyslog and then to e.g. a file. If I enable high precision timestamps in rsyslog the log messages have a more precise timestamp. However there is some delay between the application generating a log message and rsyslog adding the timestamp. So why settle for less? :) (Well it is a distributed application, i.e. several processes and computers. So to debug interactions between the parts the correct ordering and timing is very important to me.) I wrote some code that opens /dev/log itself and sends the new format directly. This works very nice and I get the timestamps I want. Example code: -------------- #!/usr/bin/python import socket log = socket.socket( socket.AF_UNIX, socket.SOCK_DGRAM ) log.connect( "/dev/log" ) # VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG # PRI: 23 (==local7) * 8 + 4 (==warning) = 188 log.send( "<188>1 2009-07-15T09:45:12.463435Z mycomputer TEST_CLIENT 12345 SOME_PACKAGE This is a test message" ) log.close() -------------- However I have a few questions: - Is there some library code I could use that accepts high precision timestamps? Some kind of successor to syslog(3). - Is there a recommended way to detect if the syslog daemon will accept the new format? Currently this could mean checking if rsyslogd is listening on /dev/log or someone else. Otherwise the logging code needs to fall back to the old format that is understood by any syslog daemon (and use only second resolution). Mfg Alexander Elbs -- Alexander Elbs *** eMail alex at segv.de From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 14:40:42 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 14:40:42 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <200907161440.43049.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Jul 16 15:31:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Jul 2009 15:31:17 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Just a quick note: there is no such thing as a whitespace inside a syslog tag (read rfc3164, 5424). The message is severely malformed and i do not see any way how rsyslog could guess the intention of the sender correctly. Have a look at the doc set, there is a full doc page with elaborate info on such cases. (i dont have it at hand right now) rainer ----- Urspr?ngliche Nachricht ----- Von: "Luis Fernando Mu?oz Mej?as" An: "rsyslog at lists.adiscon.com" Gesendet: 16.07.09 14:41 Betreff: [rsyslog] Syslogtags with whitespaces misparsed? Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 15:48:31 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 15:48:31 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Message-ID: <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > Just a quick note: there is no such thing as a whitespace inside a syslog > tag (read rfc3164, 5424). Thanks a lot, I'll tell people around here. Now I know who I must piss off. :) > The message is severely malformed and i do not see any way how rsyslog > could guess the intention of the sender correctly Indeed, rsyslog is not responsible for misbehaving senders, I just needed to clarify the source of my problem. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 10:13:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 10:13:02 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Just let me say that I try really hard to make rsyslog guess right for even the most malformed mesage format. The real issue is that there have been no standards for many years, and so there is a lot of incompatible formats out there. In this case, however, I do really not see how I could handle that intelligently within the parser (except if we create per-host custom parsers, something on my list but not even begun to work on). I guess that the gconf folks will not want to change their format, because that, too could potentially also break a lot of things (log parsers!). A solution within rsyslog configuration could be to check the hostname and apply some special template (which then utilizes the property replacer to extract the "right" information) when the hostname is "gconf". That, of course, means that no host ever must be named "gconf", but I think you can enforce that ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Luis > Fernando Mu?oz Mej?as > Sent: Thursday, July 16, 2009 3:49 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > > Just a quick note: there is no such thing as a whitespace > inside a syslog > > tag (read rfc3164, 5424). > > Thanks a lot, I'll tell people around here. Now I know who I must piss > off. :) > > > The message is severely malformed and i do not see any way > how rsyslog > > could guess the intention of the sender correctly > > Indeed, rsyslog is not responsible for misbehaving senders, I just > needed to clarify the source of my problem. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 17 15:05:38 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 17 Jul 2009 15:05:38 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Message-ID: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > In this case, however, I do really not see how I could handle that > intelligently within the parser My guesswork, as you call it on the document you reference is something like "the first word after the timestamp must be the host name, and from there to the first colon it's all syslogtag; if there's no colon I'll do whatever I want but crashing". But it's guessing, against RFCs, and I don't really think a syslog parser should play fortune-telling. > I guess that the gconf folks will not want to change their format, because > that, too could potentially also break a lot of things (log parsers!). I'll try to follow this up to gconf guys, so that they know that any log parsing of their messages is necessarily lacking *crucial* information. Anyways, gconf is the least important application to me, and I see some services around here showing the same symptoms. > A solution within rsyslog configuration In my scenario, the message has gone through several syslog relays and the correct host information is lost before it comes to my service, so there is no way to configure rsyslog to solve it. Another funny example is syslog's habit of saying "last message repeated N times". These messages don't have a colon or anything useful to delimit the application name. In this case, I receive *lots* of messages from a host called "last". Thanks for the clarifications. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 15:42:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 15:42:10 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com><200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBCC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Friday, July 17, 2009 3:06 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > > In this case, however, I do really not see how I could handle that > > intelligently within the parser > > My guesswork, as you call it on the document you reference is something > like "the first word after the timestamp must be the host name, Just to clarify so that everyone sees where the issues are ;) First off, how do you know if there is a hostname? If the message is lacking a hostname, the TAG will become it. Rsyslog assumes that a hostname is present, but knows it is a TAG instead if a character that is not valid inside the hostname. In this example "gconf" there is no way to know this is not a valid hostname. > and from > there to the first colon it's all syslogtag; Next actually not-sosubtle issue: But what do you do if there is no colon at the end of the syslog TAG? The rfcs demand none (because the header filed is SP-terminated) and also in practice there is not always a colon in it. So following this rule would break RFC compatibility and also probably break a lot more real-world cases than it fixes. > if there's no colon I'll do > whatever I want but crashing". But it's guessing, against RFCs, and I > don't really think a syslog parser should play fortune-telling. > > > I guess that the gconf folks will not want to change their format, because > > that, too could potentially also break a lot of things (log parsers!). > > I'll try to follow this up to gconf guys, so that they know that any log > parsing of their messages is necessarily lacking *crucial* > information. Anyways, gconf is the least important application to me, > and I see some services around here showing the same symptoms. > > > A solution within rsyslog configuration > > In my scenario, the message has gone through several syslog relays and > the correct host information is lost before it comes to my service, so > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". Brings up an interesting thought (for another time) - it may make sense to de-multiplex these "last message repeated N times", but that's quite some effort... Rainer > > Thanks for the clarifications. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Fri Jul 17 15:54:40 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 17 Jul 2009 07:54:40 -0600 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <4255c2570907170654i7f46cfb8la4440d4bf0838bd1@mail.gmail.com> On Fri, Jul 17, 2009 at 07:05, Luis Fernando Mu?oz Mej?as wrote: > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". It was these precise messages from *BSD that drove me to using the fromhost-ip property as opposed to whatever the sender "says" they are. It breaks down in the face of relay hosts, but when I get to the point of needing those I plan on relying on secondary parsing. From rgerhards at hq.adiscon.com Mon Jul 20 16:02:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Jul 2009 16:02:55 +0200 Subject: [rsyslog] rsyslog status Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBE8@GRFEXC.intern.adiscon.com> Hi all, contrary to what I originally thought, I have worked "a bit" on the queue engine again. I couldn't resist to unleash at least some of the potential that the now gone-away restart-type HUP offered. I am probably through with a round of changes, and it looks rather well. The code has been greatly simplified, what I hope will also enhance its stability and make finding threading bugs much easier. All in all, the new engine looks much cleaner than the previous one. I also gained a bit more performance, but that is not so much of a difference. At the high end, you may say an increase in steady message rate by maybe one or two percent. The overhead of running multiple action queues has been reduced. The new code will probably be released tomorrow (will do some final checks), but it is available in the git master branch right now. I will now see if I begin to look into the set of performance monitoring tools or postpone that after my vacation and look at threaded inputs (at least for an initial pilot). I just thought I let you know. Rainer From tbergfeld at hq.adiscon.com Wed Jul 29 08:08:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 29 Jul 2009 08:08:15 +0200 Subject: [rsyslog] rsyslog 5.1.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC41@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 5.1.3, a member of the v5-development branch. This version offers a change in the architecture, queue now always has at least one worker thread if not running in direct mode. Previous versions could run without any active workers. This simplifies the code at a very small expense. See v5 compatibility note document for more in-depth discussion. There are also some bug fixes like a fix for a minor static memory leak while reading configuration. Furthermore there is the ability added to terminate input modules not via pthread_cancel but an alternate approach via pthread_kill. This is somewhat safer as we do not need to think about the cancel-safeness of all libraries we use. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-168.phtml Changelog: http://www.rsyslog.com/Article390.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Jul 30 03:24:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 29 Jul 2009 18:24:12 -0700 (PDT) Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf Message-ID: I have the following in my config file :programname, isequal, "iaalog" ~ :programname, isequal, "plug-gw" ~ *.* @192.168.210.246:514;TraditionalFormat the first time a log entry comes along that has a program name of iaalog rsyslog crashes 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 21:09:05|history|1428800' 6183.244047218:40f00950: Message has legacy syslog format. 6183.244051641:40f00950: Called action, logging to builtin-file 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx 6183.244062030:40f00950: entering actionCalldoAction(), state: itx 6183.244066591:40f00950: file to log to: /var/log/messages 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy 6183.244086454:40f00950: action call returned 0 6183.244091070:40f00950: Filter: check for property 'programname' (value 'iaalog') isequal 'iaalog': TRUE 6183.244096297:40f00950: Called action, logging to builtin-discard 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx 6183.244106591:40f00950: entering actionCalldoAction(), state: itx 6183.244109759:40f00950: 6183.244113010:40f00950: action call returned -2002 David Lang From toppiprc at gmail.com Thu Jul 30 04:02:37 2009 From: toppiprc at gmail.com (=?UTF-8?B?6JSh6LaF?=) Date: Thu, 30 Jul 2009 10:02:37 +0800 Subject: [rsyslog] how log non-ascii characters into mysql Message-ID: Hi, I want to record some non-ascii (e.g. UTF8) text using rsyslog. When the message is written to file, it is recorded as I logged, so it can be decoded and read well. But if the message is written to mysql, I can't decode it, even through I have changed charset configurations of the mysql database, tables and columns. What has happened before the messages were sent to mysql? Another problem, suppose I know the encodings of messages from all devices, and I want to transform them into some uniform one before saving. How can I achive it? Best regards Cai Chao From rgerhards at hq.adiscon.com Thu Jul 30 11:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 11:14:57 +0200 Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC52@GRFEXC.intern.adiscon.com> David, thanks for bringing this up. Actually, there were two problems in the code: number one was the segfault but number two was that no message was ever discarded. I have fixed both, but the second fix might cause some regression. The testbench (now extended to do a basic test of the discard functionality) runs through without problems, but obviously it does not yet cover all potential cases [and it will take more time until it really does...]. I will now leave for my vacation very shortly, so I cannot do any more in-depth testing and need to hope that this does not case too bad regressions. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, July 30, 2009 3:24 AM > To: rsyslog-users > Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf > > I have the following in my config file > > :programname, isequal, "iaalog" ~ > :programname, isequal, "plug-gw" ~ > *.* @192.168.210.246:514;TraditionalFormat > > the first time a log entry comes along that has a program name of iaalog > rsyslog crashes > > 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg > '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 > 21:09:05|history|1428800' > 6183.244047218:40f00950: Message has legacy syslog format. > 6183.244051641:40f00950: Called action, logging to builtin-file > 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx > 6183.244062030:40f00950: entering actionCalldoAction(), state: itx > 6183.244066591:40f00950: file to log to: /var/log/messages > 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 > 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 > 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes > 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy > 6183.244086454:40f00950: action call returned 0 > 6183.244091070:40f00950: Filter: check for property 'programname' (value > 'iaalog') isequal 'iaalog': TRUE > 6183.244096297:40f00950: Called action, logging to builtin-discard > 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx > 6183.244106591:40f00950: entering actionCalldoAction(), state: itx > 6183.244109759:40f00950: > 6183.244113010:40f00950: action call returned -2002 > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 12:03:33 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 12:03:33 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> Message-ID: <4A716FF5.8060608@redhat.com> Hi, just wanted to let you know that rsyslog-4.2.0 is now in rawhide and will be included in Fedora 12. Tomas On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > Hi Peter, > > that sounds quite interesting. I'll see that I get the v4-stable out > tomorrow. > > Thanks for following up... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >> Sent: Monday, June 22, 2009 3:53 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi folks, >> >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) >> >> At first we need v4 stable. If we have stable v4 we will push it into > rawhide >> and F11. This must be done ASAP, I see deadline in mid-July. >> >> Peter. >> >> >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>> Hi all, >>> >>> I guess RHEL6 will be based on Fedora 11 & 12. >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>> It's sure that RHEL6 includes v3 at least. >>> >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>> http://fedoraproject.org/wiki/Schedule >>> If v4 is stable and requested to be included in F12, v4 will be >>> shipped in F12. >>> >>> Best Rio. >>> >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>> That a good question and I, too, would like to know the answer. In >>>> any case, >>>> I hope it will not be v2. Giving that RHEL is a conservative >>>> distribution >>>> (and it has to be for the target base), I would assume that if they >>>> go for a >>>> newer version, it is v3. >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>> To: rsyslog at lists.adiscon.com >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>> >>>>> Hello, >>>>> is there any info about what rsyslog version will be available in >>>>> RHEL >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>> the >>>>> time "RHEL6 package freeze" will be in stable status? >>>>> >>>>> Thank you, >>>>> Jiri H. >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Jul 30 12:56:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 12:56:26 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Hi, thanks for sharing this excellent news! If I may try to ponder you for some more information... Does this increase the chance that a recent version of rsyslog will be part of the next version of RHEL? ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Thursday, July 30, 2009 12:04 PM > To: rsyslog-users > Subject: Re: [rsyslog] RSyslog version in RHEL 6 > > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas > > On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > > Hi Peter, > > > > that sounds quite interesting. I'll see that I get the v4-stable out > > tomorrow. > > > > Thanks for following up... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec > >> Sent: Monday, June 22, 2009 3:53 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 > >> > >> Hi folks, > >> > >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) > >> > >> At first we need v4 stable. If we have stable v4 we will push it into > > rawhide > >> and F11. This must be done ASAP, I see deadline in mid-July. > >> > >> Peter. > >> > >> > >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: > >>> Hi all, > >>> > >>> I guess RHEL6 will be based on Fedora 11 & 12. > >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. > >>> It's sure that RHEL6 includes v3 at least. > >>> > >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. > >>> http://fedoraproject.org/wiki/Schedule > >>> If v4 is stable and requested to be included in F12, v4 will be > >>> shipped in F12. > >>> > >>> Best Rio. > >>> > >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: > >>>> That a good question and I, too, would like to know the answer. In > >>>> any case, > >>>> I hope it will not be v2. Giving that RHEL is a conservative > >>>> distribution > >>>> (and it has to be for the target base), I would assume that if they > >>>> go for a > >>>> newer version, it is v3. > >>>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka > >>>>> Sent: Saturday, June 20, 2009 9:48 PM > >>>>> To: rsyslog at lists.adiscon.com > >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 > >>>>> > >>>>> Hello, > >>>>> is there any info about what rsyslog version will be available in > >>>>> RHEL > >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in > >>>>> the > >>>>> time "RHEL6 package freeze" will be in stable status? > >>>>> > >>>>> Thank you, > >>>>> Jiri H. > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 13:22:08 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 13:22:08 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Message-ID: <4A718260.8090806@redhat.com> As far as I know, this version will be the one to go into RHEL 6.0. Tomas On 07/30/2009 12:56 PM, Rainer Gerhards wrote: > Hi, > > thanks for sharing this excellent news! If I may try to ponder you for some > more information... Does this increase the chance that a recent version of > rsyslog will be part of the next version of RHEL? ;) > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >> Sent: Thursday, July 30, 2009 12:04 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi, >> >> just wanted to let you know that rsyslog-4.2.0 is now in rawhide and >> will be included in Fedora 12. >> >> Tomas >> >> On 06/22/2009 02:24 PM, Rainer Gerhards wrote: >>> Hi Peter, >>> >>> that sounds quite interesting. I'll see that I get the v4-stable out >>> tomorrow. >>> >>> Thanks for following up... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >>>> Sent: Monday, June 22, 2009 3:53 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >>>> >>>> Hi folks, >>>> >>>> rsyslog maintainer is on vacation at the moment so I'm answering this Q. > :) >>>> At first we need v4 stable. If we have stable v4 we will push it into >>> rawhide >>>> and F11. This must be done ASAP, I see deadline in mid-July. >>>> >>>> Peter. >>>> >>>> >>>> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>>>> Hi all, >>>>> >>>>> I guess RHEL6 will be based on Fedora 11 & 12. >>>>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>>>> It's sure that RHEL6 includes v3 at least. >>>>> >>>>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>>>> http://fedoraproject.org/wiki/Schedule >>>>> If v4 is stable and requested to be included in F12, v4 will be >>>>> shipped in F12. >>>>> >>>>> Best Rio. >>>>> >>>>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>>>> That a good question and I, too, would like to know the answer. In >>>>>> any case, >>>>>> I hope it will not be v2. Giving that RHEL is a conservative >>>>>> distribution >>>>>> (and it has to be for the target base), I would assume that if they >>>>>> go for a >>>>>> newer version, it is v3. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>>>> To: rsyslog at lists.adiscon.com >>>>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>>>> >>>>>>> Hello, >>>>>>> is there any info about what rsyslog version will be available in >>>>>>> RHEL >>>>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>>>> the >>>>>>> time "RHEL6 package freeze" will be in stable status? >>>>>>> >>>>>>> Thank you, >>>>>>> Jiri H. >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Jul 30 13:29:11 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 13:29:11 +0200 Subject: [rsyslog] merge failures Message-ID: Hi, just noticed this in v4-stable grep "<<<<" * -R tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am Seems like a merge error. Haven't checked the other branches. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Jul 30 14:23:23 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 14:23:23 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <4A716FF5.8060608@redhat.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com> <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: 2009/7/30 Tomas Heinrich : > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas Hi Tomas, that's good news. FWIW the current version in Debian unstable/testing is also 4.2.0, so squeeze will also ship with at least 4.2.x and Ubuntu also made the switch to rsyslog. Their next release 9.10 will have rsyslog 4.2.0 as default syslog daemon. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Jul 30 15:43:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 15:43:21 +0200 Subject: [rsyslog] merge failures References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC5E@GRFEXC.intern.adiscon.com> Thanks Fixed in v4-stable now, will propagate to the other branches over time... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, July 30, 2009 1:29 PM > To: rsyslog-users > Subject: [rsyslog] merge failures > > Hi, > just noticed this in v4-stable > > grep "<<<<" * -R > tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am > > Seems like a merge error. > > Haven't checked the other branches. > > Cheers, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 00:42:14 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:42:14 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > ubuntu 9.04 correction, debian 5.0, not ubuntu (forgot which machine I was compiling on) master also fails with the same error, v4-stable has no problem. David Lang > zlib version > /* zlib.h -- interface of the 'zlib' general purpose compression library > version 1.2.3.3, October 2nd, 2006 > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > compression library - development > > > error > > In file included from zlibw.h:28, > from stream.h:72, > from obj.h:50, > from rsyslog.h:482, > from stringbuf.c:39: > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gzseek64' > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > '__attribute__' before 'gztell64' > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > before 'off64_t' > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > before 'off64_t' > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 00:50:13 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Jun 2009 15:50:13 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: Message-ID: On Tue, 30 Jun 2009, david at lang.hm wrote: > On Tue, 30 Jun 2009, david at lang.hm wrote: > >> ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. and the error remains the same with configure --disable-zlib David Lang > David Lang > > >> zlib version >> /* zlib.h -- interface of the 'zlib' general purpose compression library >> version 1.2.3.3, October 2nd, 2006 >> >> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >> compression library - development >> >> >> error >> >> In file included from zlibw.h:28, >> from stream.h:72, >> from obj.h:50, >> from rsyslog.h:482, >> from stringbuf.c:39: >> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gzseek64' >> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >> '__attribute__' before 'gztell64' >> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >> before 'off64_t' >> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >> before 'off64_t' >> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 09:13:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 09:13:29 +0200 Subject: [rsyslog] v5-devel won't compile References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Looks like a problem "with" Debian. I guess I defined some types that zlib also defines to the same names. I have to admit that Michael Biebl told me, but it somehow slipped my mind... Will test on Debian today and fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 12:42 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Tue, 30 Jun 2009, david at lang.hm wrote: > > > ubuntu 9.04 > > correction, debian 5.0, not ubuntu (forgot which machine I was compiling > on) > > master also fails with the same error, v4-stable has no problem. > > David Lang > > > > zlib version > > /* zlib.h -- interface of the 'zlib' general purpose compression library > > version 1.2.3.3, October 2nd, 2006 > > > > ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > compression library - development > > > > > > error > > > > In file included from zlibw.h:28, > > from stream.h:72, > > from obj.h:50, > > from rsyslog.h:482, > > from stringbuf.c:39: > > /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gzseek64' > > /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > '__attribute__' before 'gztell64' > > /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > > before 'off64_t' > > /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > > before 'off64_t' > > make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory `/usr/src/rsyslog' > > make: *** [all] Error 2 > > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:10:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:10:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Looks like a problem "with" Debian. I guess I defined some types that zlib > also defines to the same names. I have to admit that Michael Biebl told me, > but it somehow slipped my mind... Will test on Debian today and fix. master and v5-devel also complained that I didn't have the zlib development libraries installed (unable to find zlib.h), after I installed that it gave me the error I posted below. v4 didn't complain about the missing library to start with as I posted in another message, I was suprised the the problem didn't go away when I configured with --disable-zlib David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 12:42 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> On Tue, 30 Jun 2009, david at lang.hm wrote: >> >>> ubuntu 9.04 >> >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling >> on) >> >> master also fails with the same error, v4-stable has no problem. >> >> David Lang >> >> >>> zlib version >>> /* zlib.h -- interface of the 'zlib' general purpose compression library >>> version 1.2.3.3, October 2nd, 2006 >>> >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>> compression library - development >>> >>> >>> error >>> >>> In file included from zlibw.h:28, >>> from stream.h:72, >>> from obj.h:50, >>> from rsyslog.h:482, >>> from stringbuf.c:39: >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gzseek64' >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>> '__attribute__' before 'gztell64' >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' >>> before 'off64_t' >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:14:37 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:14:37 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:11 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Looks like a problem "with" Debian. I guess I defined some types that zlib > > also defines to the same names. I have to admit that Michael Biebl told me, > > but it somehow slipped my mind... Will test on Debian today and fix. > > master and v5-devel also complained that I didn't have the zlib > development libraries installed (unable to find zlib.h), after I installed > that it gave me the error I posted below. I have begun to look into it. Debian has a slightly newer version of zlib than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem occurs. I now need to look at the root cause, but it is well hidden ;) > > v4 didn't complain about the missing library to start with > > as I posted in another message, I was suprised the the problem didn't go > away when I configured with --disable-zlib That's a bug in omfile, it obviously does not yet properly reflect this switch. Will fix that one, too. Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Wednesday, July 01, 2009 12:42 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > >> > >>> ubuntu 9.04 > >> > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > >> on) > >> > >> master also fails with the same error, v4-stable has no problem. > >> > >> David Lang > >> > >> > >>> zlib version > >>> /* zlib.h -- interface of the 'zlib' general purpose compression library > >>> version 1.2.3.3, October 2nd, 2006 > >>> > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>> compression library - development > >>> > >>> > >>> error > >>> > >>> In file included from zlibw.h:28, > >>> from stream.h:72, > >>> from obj.h:50, > >>> from rsyslog.h:482, > >>> from stringbuf.c:39: > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gzseek64' > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>> '__attribute__' before 'gztell64' > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or '...' > >>> before 'off64_t' > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>> make[1]: *** [all-recursive] Error 1 > >>> make[1]: Leaving directory `/usr/src/rsyslog' > >>> make: *** [all] Error 2 > >>> > >>> > >>> David Lang > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 10:35:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:35:56 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Of course (and as always), the problem rooted in rsyslog. The commit says it all: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 308f383bec1b8be8 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, July 01, 2009 10:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, July 01, 2009 10:11 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > > > Looks like a problem "with" Debian. I guess I defined some types that > zlib > > > also defines to the same names. I have to admit that Michael Biebl told > me, > > > but it somehow slipped my mind... Will test on Debian today and fix. > > > > master and v5-devel also complained that I didn't have the zlib > > development libraries installed (unable to find zlib.h), after I installed > > that it gave me the error I posted below. > > I have begun to look into it. Debian has a slightly newer version of zlib > than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > occurs. I now need to look at the root cause, but it is well hidden ;) > > > > > v4 didn't complain about the missing library to start with > > > > as I posted in another message, I was suprised the the problem didn't go > > away when I configured with --disable-zlib > > That's a bug in omfile, it obviously does not yet properly reflect this > switch. Will fix that one, too. > > Rainer > > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > >> Sent: Wednesday, July 01, 2009 12:42 AM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] v5-devel won't compile > > >> > > >> On Tue, 30 Jun 2009, david at lang.hm wrote: > > >> > > >>> ubuntu 9.04 > > >> > > >> correction, debian 5.0, not ubuntu (forgot which machine I was compiling > > >> on) > > >> > > >> master also fails with the same error, v4-stable has no problem. > > >> > > >> David Lang > > >> > > >> > > >>> zlib version > > >>> /* zlib.h -- interface of the 'zlib' general purpose compression > library > > >>> version 1.2.3.3, October 2nd, 2006 > > >>> > > >>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > > >>> compression library - development > > >>> > > >>> > > >>> error > > >>> > > >>> In file included from zlibw.h:28, > > >>> from stream.h:72, > > >>> from obj.h:50, > > >>> from rsyslog.h:482, > > >>> from stringbuf.c:39: > > >>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gzseek64' > > >>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > > >>> '__attribute__' before 'gztell64' > > >>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > '...' > > >>> before 'off64_t' > > >>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > > >>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > > >>> make[1]: *** [all-recursive] Error 1 > > >>> make[1]: Leaving directory `/usr/src/rsyslog' > > >>> make: *** [all] Error 2 > > >>> > > >>> > > >>> David Lang > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Jul 1 10:53:15 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 01:53:15 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 I'll test it when I get into the office. when I tried to do it on my laptop I ran into the testbench failure again, even with --disable-testbench CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/bash: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 config.log: $ ./configure --disable-testbench David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 10:56:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 10:56:08 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAA0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 10:53 AM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Of course (and as always), the problem rooted in rsyslog. The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > > 308f383bec1b8be8 > > I'll test it when I get into the office. when I tried to do it on my > laptop I ran into the testbench failure again, even with > --disable-testbench > > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/bash: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > Mmhhh... I did my testing on a debian 5.0 machine without javadev and with --enable-testbench=no. Seems to make a difference (I am still not deep inside autoconf...). > > config.log: $ ./configure --disable-testbench > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined some types that > >> zlib > >>>> also defines to the same names. I have to admit that Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem > >> occurs. I now need to look at the root cause, but it is well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 1 15:18:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 15:18:35 +0200 Subject: [rsyslog] performance optimization (milestone) done Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have reached a milestone with my optimizations. My goal of reducing malloc/free calls has now been reached. I need to revisit priorities, but I will probably move away from performance optimization for a while now (maybe some smaller things that I stumble across, but not a real effort). As I have written recently, there is still ample potential for optimization. I will "just" probably not explore it at this point in time. However, I would like to ask one question that could lead to a new effort (in v5). Looking at the queue engine, I see that managing the queue, especially in ultra-reliable mode, requires a lot of effort, what also means a lot of CPU time. If we relax reliability conditions, we could definitely save some time here (probably a real lot!). So my question is what would be the least reliability that you need for a non-audit, high-performance scenario. I can envision that it is sufficient to have this: All messages handed over to the queue will be processed if no system failure occurs and if there is space available inside the queue. All messages are queued in main memory. If messages are tried to be enqueued when the queue is full, these messages will be lost. Message loss victims will be strictly based on order of enqueue (queue full condition) and not severity or any other property (evaluating that would take time, again). Also, it is acceptable to lose unprocessed messages during rsyslogd shutdown, if they cannot be processed within the (configurable) shutdown intervals. Would this make sense for sufficiently important use cases? If so, I could see that (over time!) I implement a very fast queue driver based on that relaxed criteria. With it, I think I could also create a lock-free version (but not immediately), which would definitely be a big performance gain. Feedback is appreciated. The optimized builds are currently in the git master and v5-devel branches, new releases are due soon (but will see that I do at least a bit more doc work before releasing them). Rainer From david at lang.hm Wed Jul 1 18:58:55 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 09:58:55 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I just wanted to let you know that I have reached a milestone with my > optimizations. My goal of reducing malloc/free calls has now been reached. I > need to revisit priorities, but I will probably move away from performance > optimization for a while now (maybe some smaller things that I stumble > across, but not a real effort). > > As I have written recently, there is still ample potential for optimization. > I will "just" probably not explore it at this point in time. > > However, I would like to ask one question that could lead to a new effort (in > v5). Looking at the queue engine, I see that managing the queue, especially > in ultra-reliable mode, requires a lot of effort, what also means a lot of > CPU time. > > If we relax reliability conditions, we could definitely save some time here > (probably a real lot!). So my question is what would be the least reliability > that you need for a non-audit, high-performance scenario. I can envision that > it is sufficient to have this: > > All messages handed over to the queue will be processed if no system failure > occurs and if there is space available inside the queue. All messages are > queued in main memory. If messages are tried to be enqueued when the queue is > full, these messages will be lost. Message loss victims will be strictly > based on order of enqueue (queue full condition) and not severity or any > other property (evaluating that would take time, again). Also, it is > acceptable to lose unprocessed messages during rsyslogd shutdown, if they > cannot be processed within the (configurable) shutdown intervals. > > Would this make sense for sufficiently important use cases? If so, I could > see that (over time!) I implement a very fast queue driver based on that > relaxed criteria. With it, I think I could also create a lock-free version > (but not immediately), which would definitely be a big performance gain. how is this different than what happens today? David Lang > Feedback is appreciated. > > The optimized builds are currently in the git master and v5-devel branches, > new releases are due soon (but will see that I do at least a bit more doc > work before releasing them). > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 19:37:27 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 10:37:27 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: back on the debian 5 system I get the following error with v5-devel commit 6322343eca1084b509386a94c1bcf2ca819f1220 gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o rsyslogd-omfwd.o rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread ../runtime/.libs/librsyslog.a -ldl -lrt rsyslogd-syslogd.o: In function `mainloop': /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to `execScheduled' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtExit': /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference to `rulesetClassExit' /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference to `ruleClassExit' ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In function `rsrtInit': /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference to `propClassInit' /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference to `ruleClassInit' /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference to `rulesetClassInit' ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In function `objClassInit': /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to `apcClassInit' collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/rsyslog/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 On Wed, 1 Jul 2009, Rainer Gerhards wrote: > Date: Wed, 1 Jul 2009 10:35:56 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > Of course (and as always), the problem rooted in rsyslog. The commit says it > all: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff39b46630d95a8d8 > 308f383bec1b8be8 > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Wednesday, July 01, 2009 10:15 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 10:11 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>> >>>> Looks like a problem "with" Debian. I guess I defined some types that >> zlib >>>> also defines to the same names. I have to admit that Michael Biebl told >> me, >>>> but it somehow slipped my mind... Will test on Debian today and fix. >>> >>> master and v5-devel also complained that I didn't have the zlib >>> development libraries installed (unable to find zlib.h), after I > installed >>> that it gave me the error I posted below. >> >> I have begun to look into it. Debian has a slightly newer version of zlib >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's header the problem >> occurs. I now need to look at the root cause, but it is well hidden ;) >> >>> >>> v4 didn't complain about the missing library to start with >>> >>> as I posted in another message, I was suprised the the problem didn't go >>> away when I configured with --disable-zlib >> >> That's a bug in omfile, it obviously does not yet properly reflect this >> switch. Will fix that one, too. >> >> Rainer >> >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>> >>>>>> ubuntu 9.04 >>>>> >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > compiling >>>>> on) >>>>> >>>>> master also fails with the same error, v4-stable has no problem. >>>>> >>>>> David Lang >>>>> >>>>> >>>>>> zlib version >>>>>> /* zlib.h -- interface of the 'zlib' general purpose compression >> library >>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>> >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>> compression library - development >>>>>> >>>>>> >>>>>> error >>>>>> >>>>>> In file included from zlibw.h:28, >>>>>> from stream.h:72, >>>>>> from obj.h:50, >>>>>> from rsyslog.h:482, >>>>>> from stringbuf.c:39: >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gzseek64' >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', ';', 'asm' or >>>>>> '__attribute__' before 'gztell64' >>>>>> /usr/include/zlib.h:1368: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> /usr/include/zlib.h:1369: error: expected declaration specifiers or >> '...' >>>>>> before 'off64_t' >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>> make[1]: *** [all-recursive] Error 1 >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>> make: *** [all] Error 2 >>>>>> >>>>>> >>>>>> David Lang >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:24:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:24:17 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, runtime/apc.c) are missing. I have just checked, they are in the git branch. Also I have no problems compiling that version, I tried it on Debian this morning during the zlib fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 7:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > back on the debian 5 system I get the following error with v5-devel > commit 6322343eca1084b509386a94c1bcf2ca819f1220 > > > gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > rsyslogd-omfwd.o > rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > ../runtime/.libs/librsyslog.a -ldl -lrt > rsyslogd-syslogd.o: In function `mainloop': > /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > `execScheduled' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtExit': > /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > to `rulesetClassExit' > /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > to `ruleClassExit' > ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > function `rsrtInit': > /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > to `propClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > to `ruleClassInit' > /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > to `rulesetClassInit' > ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > function `objClassInit': > /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > `apcClassInit' > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory `/usr/src/rsyslog/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > > > > On Wed, 1 Jul 2009, Rainer > Gerhards wrote: > > > Date: Wed, 1 Jul 2009 10:35:56 +0200 > > From: Rainer Gerhards > > Reply-To: rsyslog-users > > To: rsyslog-users > > Subject: Re: [rsyslog] v5-devel won't compile > > > > Of course (and as always), the problem rooted in rsyslog. > The commit says it > > all: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > 39b46630d95a8d8 > > 308f383bec1b8be8 > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Wednesday, July 01, 2009 10:15 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] v5-devel won't compile > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>> Sent: Wednesday, July 01, 2009 10:11 AM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] v5-devel won't compile > >>> > >>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>> > >>>> Looks like a problem "with" Debian. I guess I defined > some types that > >> zlib > >>>> also defines to the same names. I have to admit that > Michael Biebl told > >> me, > >>>> but it somehow slipped my mind... Will test on Debian > today and fix. > >>> > >>> master and v5-devel also complained that I didn't have the zlib > >>> development libraries installed (unable to find zlib.h), after I > > installed > >>> that it gave me the error I posted below. > >> > >> I have begun to look into it. Debian has a slightly newer > version of zlib > >> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > header the problem > >> occurs. I now need to look at the root cause, but it is > well hidden ;) > >> > >>> > >>> v4 didn't complain about the missing library to start with > >>> > >>> as I posted in another message, I was suprised the the > problem didn't go > >>> away when I configured with --disable-zlib > >> > >> That's a bug in omfile, it obviously does not yet properly > reflect this > >> switch. Will fix that one, too. > >> > >> Rainer > >> > >>> > >>> David Lang > >>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>> > >>>>>> ubuntu 9.04 > >>>>> > >>>>> correction, debian 5.0, not ubuntu (forgot which machine I was > > compiling > >>>>> on) > >>>>> > >>>>> master also fails with the same error, v4-stable has no problem. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>>> zlib version > >>>>>> /* zlib.h -- interface of the 'zlib' general purpose > compression > >> library > >>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>> > >>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>> compression library - development > >>>>>> > >>>>>> > >>>>>> error > >>>>>> > >>>>>> In file included from zlibw.h:28, > >>>>>> from stream.h:72, > >>>>>> from obj.h:50, > >>>>>> from rsyslog.h:482, > >>>>>> from stringbuf.c:39: > >>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gzseek64' > >>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > ';', 'asm' or > >>>>>> '__attribute__' before 'gztell64' > >>>>>> /usr/include/zlib.h:1368: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> /usr/include/zlib.h:1369: error: expected declaration > specifiers or > >> '...' > >>>>>> before 'off64_t' > >>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>> make: *** [all] Error 2 > >>>>>> > >>>>>> > >>>>>> David Lang > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Jul 1 22:29:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Jul 2009 22:29:08 +0200 Subject: [rsyslog] performance optimization (milestone) done References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 6:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] performance optimization (milestone) done > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I just wanted to let you know that I have reached a > milestone with my > > optimizations. My goal of reducing malloc/free calls has > now been reached. I > > need to revisit priorities, but I will probably move away > from performance > > optimization for a while now (maybe some smaller things > that I stumble > > across, but not a real effort). > > > > As I have written recently, there is still ample potential > for optimization. > > I will "just" probably not explore it at this point in time. > > > > However, I would like to ask one question that could lead > to a new effort (in > > v5). Looking at the queue engine, I see that managing the > queue, especially > > in ultra-reliable mode, requires a lot of effort, what also > means a lot of > > CPU time. > > > > If we relax reliability conditions, we could definitely > save some time here > > (probably a real lot!). So my question is what would be the > least reliability > > that you need for a non-audit, high-performance scenario. I > can envision that > > it is sufficient to have this: > > > > All messages handed over to the queue will be processed if > no system failure > > occurs and if there is space available inside the queue. > All messages are > > queued in main memory. If messages are tried to be enqueued > when the queue is > > full, these messages will be lost. Message loss victims > will be strictly > > based on order of enqueue (queue full condition) and not > severity or any > > other property (evaluating that would take time, again). Also, it is > > acceptable to lose unprocessed messages during rsyslogd > shutdown, if they > > cannot be processed within the (configurable) shutdown intervals. > > > > Would this make sense for sufficiently important use cases? > If so, I could > > see that (over time!) I implement a very fast queue driver > based on that > > relaxed criteria. With it, I think I could also create a > lock-free version > > (but not immediately), which would definitely be a big > performance gain. > > how is this different than what happens today? Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, persistence, reliability, discard messages based on priority, slow down sender, rate limiting ... stuff. All nice and useful, but takes time... What I described above would be the sole capability, (almost) none of these bells and whistles (except those that are handled in the action engine, I do not know out of my head which these are, but a low subset of what I said). Rainer > > David Lang > > > Feedback is appreciated. > > > > The optimized builds are currently in the git master and > v5-devel branches, > > new releases are due soon (but will see that I do at least > a bit more doc > > work before releasing them). > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:51:36 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: > This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, > runtime/apc.c) are missing. I have just checked, they are in the git branch. > Also I have no problems compiling that version, I tried it on Debian this > morning during the zlib fix. hmm, those files were definantly missing. I had done a git checkout origin/v5-devel, I then did a git checkout -f origin/v5-devel and the files are there. now it is dieing in the configure step :-( checking for FSSTND support... yes ./configure: line 10307: syntax error near unexpected token `GNUTLS,' ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= 1.4.0)' make: *** [config.status] Error 2 I tried doing the autoreconf command, and it isn't happy #autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 my next step is going to be to nuke the entire directory and try a checkout again. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 7:37 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] v5-devel won't compile >> >> back on the debian 5 system I get the following error with v5-devel >> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >> >> >> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >> rsyslogd-omfwd.o >> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >> ../runtime/.libs/librsyslog.a -ldl -lrt >> rsyslogd-syslogd.o: In function `mainloop': >> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >> `execScheduled' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtExit': >> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >> to `rulesetClassExit' >> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >> to `ruleClassExit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >> function `rsrtInit': >> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >> to `propClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >> to `ruleClassInit' >> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >> to `rulesetClassInit' >> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >> function `objClassInit': >> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >> `apcClassInit' >> collect2: ld returned 1 exit status >> make[2]: *** [rsyslogd] Error 1 >> make[2]: Leaving directory `/usr/src/rsyslog/tools' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory `/usr/src/rsyslog' >> make: *** [all] Error 2 >> >> >> >> On Wed, 1 Jul 2009, Rainer >> Gerhards wrote: >> >>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>> From: Rainer Gerhards >>> Reply-To: rsyslog-users >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> Of course (and as always), the problem rooted in rsyslog. >> The commit says it >>> all: >>> >>> >> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >> 39b46630d95a8d8 >>> 308f383bec1b8be8 >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>> >>>>>> Looks like a problem "with" Debian. I guess I defined >> some types that >>>> zlib >>>>>> also defines to the same names. I have to admit that >> Michael Biebl told >>>> me, >>>>>> but it somehow slipped my mind... Will test on Debian >> today and fix. >>>>> >>>>> master and v5-devel also complained that I didn't have the zlib >>>>> development libraries installed (unable to find zlib.h), after I >>> installed >>>>> that it gave me the error I posted below. >>>> >>>> I have begun to look into it. Debian has a slightly newer >> version of zlib >>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >> header the problem >>>> occurs. I now need to look at the root cause, but it is >> well hidden ;) >>>> >>>>> >>>>> v4 didn't complain about the missing library to start with >>>>> >>>>> as I posted in another message, I was suprised the the >> problem didn't go >>>>> away when I configured with --disable-zlib >>>> >>>> That's a bug in omfile, it obviously does not yet properly >> reflect this >>>> switch. Will fix that one, too. >>>> >>>> Rainer >>>> >>>>> >>>>> David Lang >>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>> >>>>>>>> ubuntu 9.04 >>>>>>> >>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>> compiling >>>>>>> on) >>>>>>> >>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>>> zlib version >>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >> compression >>>> library >>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>> >>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>> compression library - development >>>>>>>> >>>>>>>> >>>>>>>> error >>>>>>>> >>>>>>>> In file included from zlibw.h:28, >>>>>>>> from stream.h:72, >>>>>>>> from obj.h:50, >>>>>>>> from rsyslog.h:482, >>>>>>>> from stringbuf.c:39: >>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >> ';', 'asm' or >>>>>>>> '__attribute__' before 'gztell64' >>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >> specifiers or >>>> '...' >>>>>>>> before 'off64_t' >>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>> make: *** [all] Error 2 >>>>>>>> >>>>>>>> >>>>>>>> David Lang >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 22:52:57 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:52:57 -0700 (PDT) Subject: [rsyslog] performance optimization (milestone) done In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAAD@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAF@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, July 01, 2009 6:59 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] performance optimization (milestone) done >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> Hi all, >>> >>> I just wanted to let you know that I have reached a >> milestone with my >>> optimizations. My goal of reducing malloc/free calls has >> now been reached. I >>> need to revisit priorities, but I will probably move away >> from performance >>> optimization for a while now (maybe some smaller things >> that I stumble >>> across, but not a real effort). >>> >>> As I have written recently, there is still ample potential >> for optimization. >>> I will "just" probably not explore it at this point in time. >>> >>> However, I would like to ask one question that could lead >> to a new effort (in >>> v5). Looking at the queue engine, I see that managing the >> queue, especially >>> in ultra-reliable mode, requires a lot of effort, what also >> means a lot of >>> CPU time. >>> >>> If we relax reliability conditions, we could definitely >> save some time here >>> (probably a real lot!). So my question is what would be the >> least reliability >>> that you need for a non-audit, high-performance scenario. I >> can envision that >>> it is sufficient to have this: >>> >>> All messages handed over to the queue will be processed if >> no system failure >>> occurs and if there is space available inside the queue. >> All messages are >>> queued in main memory. If messages are tried to be enqueued >> when the queue is >>> full, these messages will be lost. Message loss victims >> will be strictly >>> based on order of enqueue (queue full condition) and not >> severity or any >>> other property (evaluating that would take time, again). Also, it is >>> acceptable to lose unprocessed messages during rsyslogd >> shutdown, if they >>> cannot be processed within the (configurable) shutdown intervals. >>> >>> Would this make sense for sufficiently important use cases? >> If so, I could >>> see that (over time!) I implement a very fast queue driver >> based on that >>> relaxed criteria. With it, I think I could also create a >> lock-free version >>> (but not immediately), which would definitely be a big >> performance gain. >> >> how is this different than what happens today? > > Puuuuhhh... In various ways, we have all this go to disk, low/high watermark, > persistence, reliability, discard messages based on priority, slow down > sender, rate limiting ... stuff. All nice and useful, but takes time... > > What I described above would be the sole capability, (almost) none of these > bells and whistles (except those that are handled in the action engine, I do > not know out of my head which these are, but a low subset of what I said). what you described as the new, limited capabilities sounds like what I would expect from a syslog daemon. David Lang From david at lang.hm Wed Jul 1 22:58:59 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 13:58:59 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. still no luck, secdev:/usr/src/rsyslog# rm -r * secdev:/usr/src/rsyslog# ls -a . .. .deps .git .gitignore .libs secdev:/usr/src/rsyslog# ls .libs lmtcpclt.la lmtcpclt.lai lmtcpclt_la-tcpclt.o lmtcpclt.so lmtcpsrv.la lmtcpsrv.lai lmtcpsrv_la-tcpsrv.o lmtcpsrv_la-tcps_sess.o lmtcpsrv.so secdev:/usr/src/rsyslog# !git git checkout -f origin/v5-devel Checking out files: 100% (497/497), done. HEAD is now at 6322343... Merge branch 'master' into v5-devel secdev:/usr/src/rsyslog# ls runtime/rule* runtime/rule.c runtime/rule.h runtime/ruleset.c runtime/ruleset.h secdev:/usr/src/rsyslog# autoreconf -fvi autoreconf2.50: Entering directory `.' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal --force -I m4 autoreconf2.50: configure.ac: tracing autoreconf2.50: configure.ac: not using Libtool autoreconf2.50: running: /usr/bin/autoconf --force configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 secdev:/usr/src/rsyslog# autoreconf -V autoreconf (GNU Autoconf) 2.61 Copyright (C) 2006 Free Software Foundation, Inc. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Written by David J. MacKenzie and Akim Demaille. --- Autoconf 2.50 chosen by Debian wrapper script. For information and tuning advice see autoconf(1). > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:02:23 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:02:23 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: never mind, pkg-config was not installed on this server (and I've got _no_ idea how I was sucessfully compiling the v4-stable branch on this box) David Lang On Wed, 1 Jul 2009, david at lang.hm wrote: > Date: Wed, 1 Jul 2009 13:51:36 -0700 (PDT) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >> runtime/apc.c) are missing. I have just checked, they are in the git branch. >> Also I have no problems compiling that version, I tried it on Debian this >> morning during the zlib fix. > > hmm, those files were definantly missing. > > I had done a git checkout origin/v5-devel, I then did a git checkout -f > origin/v5-devel and the files are there. > > now it is dieing in the configure step :-( > > checking for FSSTND support... yes > ./configure: line 10307: syntax error near unexpected token `GNUTLS,' > ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > 1.4.0)' > make: *** [config.status] Error 2 > > > I tried doing the autoreconf command, and it isn't happy > > #autoreconf -fvi > autoreconf2.50: Entering directory `.' > autoreconf2.50: configure.ac: not using Gettext > autoreconf2.50: running: aclocal --force -I m4 > autoreconf2.50: configure.ac: tracing > autoreconf2.50: configure.ac: not using Libtool > autoreconf2.50: running: /usr/bin/autoconf --force > configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > > my next step is going to be to nuke the entire directory and try a > checkout again. > > David Lang > > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com >>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>> Sent: Wednesday, July 01, 2009 7:37 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] v5-devel won't compile >>> >>> back on the debian 5 system I get the following error with v5-devel >>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>> >>> >>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>> rsyslogd-omfwd.o >>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>> ../runtime/.libs/librsyslog.a -ldl -lrt >>> rsyslogd-syslogd.o: In function `mainloop': >>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>> `execScheduled' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtExit': >>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>> to `rulesetClassExit' >>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>> to `ruleClassExit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>> function `rsrtInit': >>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>> to `propClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>> to `ruleClassInit' >>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>> to `rulesetClassInit' >>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>> function `objClassInit': >>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>> `apcClassInit' >>> collect2: ld returned 1 exit status >>> make[2]: *** [rsyslogd] Error 1 >>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory `/usr/src/rsyslog' >>> make: *** [all] Error 2 >>> >>> >>> >>> On Wed, 1 Jul 2009, Rainer >>> Gerhards wrote: >>> >>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>> From: Rainer Gerhards >>>> Reply-To: rsyslog-users >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> Of course (and as always), the problem rooted in rsyslog. >>> The commit says it >>>> all: >>>> >>>> >>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>> 39b46630d95a8d8 >>>> 308f383bec1b8be8 >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>> >>>>>>> Looks like a problem "with" Debian. I guess I defined >>> some types that >>>>> zlib >>>>>>> also defines to the same names. I have to admit that >>> Michael Biebl told >>>>> me, >>>>>>> but it somehow slipped my mind... Will test on Debian >>> today and fix. >>>>>> >>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>> development libraries installed (unable to find zlib.h), after I >>>> installed >>>>>> that it gave me the error I posted below. >>>>> >>>>> I have begun to look into it. Debian has a slightly newer >>> version of zlib >>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>> header the problem >>>>> occurs. I now need to look at the root cause, but it is >>> well hidden ;) >>>>> >>>>>> >>>>>> v4 didn't complain about the missing library to start with >>>>>> >>>>>> as I posted in another message, I was suprised the the >>> problem didn't go >>>>>> away when I configured with --disable-zlib >>>>> >>>>> That's a bug in omfile, it obviously does not yet properly >>> reflect this >>>>> switch. Will fix that one, too. >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>> To: rsyslog-users >>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>> >>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>> >>>>>>>>> ubuntu 9.04 >>>>>>>> >>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>> compiling >>>>>>>> on) >>>>>>>> >>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>>> zlib version >>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>> compression >>>>> library >>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>> >>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>> compression library - development >>>>>>>>> >>>>>>>>> >>>>>>>>> error >>>>>>>>> >>>>>>>>> In file included from zlibw.h:28, >>>>>>>>> from stream.h:72, >>>>>>>>> from obj.h:50, >>>>>>>>> from rsyslog.h:482, >>>>>>>>> from stringbuf.c:39: >>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>> ';', 'asm' or >>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>> specifiers or >>>>> '...' >>>>>>>>> before 'off64_t' >>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>> make: *** [all] Error 2 >>>>>>>>> >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Jul 1 23:22:51 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Jul 2009 14:22:51 -0700 (PDT) Subject: [rsyslog] v5-devel won't compile In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 1 Jul 2009, david at lang.hm wrote: > never mind, pkg-config was not installed on this server (and I've got _no_ > idea how I was sucessfully compiling the v4-stable branch on this box) that wasn't it either, it turns out that the libtool package is needed as well. disabling the testbench doesn't seem to work for me (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' Making all in tests make[2]: Entering directory `/usr/src/rsyslog/tests' CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java /bin/sh: line 6: javac: command not found make[2]: *** [classcheck.stamp] Error 127 make[2]: Leaving directory `/usr/src/rsyslog/tests' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/rsyslog' make: *** [all] Error 2 secdev:/usr/src/rsyslog# history |tail 1408 ./configure 1409 ./configure --help 1410 ./configure --enable-imfile 1411 make 1412 ./configure --help |grep test 1413 ./configure --enable-imfile --enable-testbench=no 1414 make 1415 make clean 1416 make 1417 history |tail I'm now installing java.. David Lang > On Wed, 1 Jul 2009, david at lang.hm wrote: > >> >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >> >>> This sounds like a couple of files (runtime/rule.c, runtime/ruleset.c, >>> runtime/apc.c) are missing. I have just checked, they are in the git branch. >>> Also I have no problems compiling that version, I tried it on Debian this >>> morning during the zlib fix. >> >> hmm, those files were definantly missing. >> >> I had done a git checkout origin/v5-devel, I then did a git checkout -f >> origin/v5-devel and the files are there. >> >> now it is dieing in the configure step :-( >> >> checking for FSSTND support... yes >> ./configure: line 10307: syntax error near unexpected token `GNUTLS,' >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= >> 1.4.0)' >> make: *** [config.status] Error 2 >> >> >> I tried doing the autoreconf command, and it isn't happy >> >> #autoreconf -fvi >> autoreconf2.50: Entering directory `.' >> autoreconf2.50: configure.ac: not using Gettext >> autoreconf2.50: running: aclocal --force -I m4 >> autoreconf2.50: configure.ac: tracing >> autoreconf2.50: configure.ac: not using Libtool >> autoreconf2.50: running: /usr/bin/autoconf --force >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC >> If this token and others are legitimate, please use >> m4_pattern_allow. >> See the Autoconf documentation. >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 >> >> my next step is going to be to nuke the entire directory and try a >> checkout again. >> >> David Lang >> >> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>> Sent: Wednesday, July 01, 2009 7:37 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] v5-devel won't compile >>>> >>>> back on the debian 5 system I get the following error with v5-devel >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 >>>> >>>> >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o >>>> rsyslogd-omfwd.o >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread >>>> ../runtime/.libs/librsyslog.a -ldl -lrt >>>> rsyslogd-syslogd.o: In function `mainloop': >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to >>>> `execScheduled' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtExit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference >>>> to `rulesetClassExit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference >>>> to `ruleClassExit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In >>>> function `rsrtInit': >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference >>>> to `propClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference >>>> to `ruleClassInit' >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference >>>> to `rulesetClassInit' >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In >>>> function `objClassInit': >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to >>>> `apcClassInit' >>>> collect2: ld returned 1 exit status >>>> make[2]: *** [rsyslogd] Error 1 >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' >>>> make[1]: *** [all-recursive] Error 1 >>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>> make: *** [all] Error 2 >>>> >>>> >>>> >>>> On Wed, 1 Jul 2009, Rainer >>>> Gerhards wrote: >>>> >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 >>>>> From: Rainer Gerhards >>>>> Reply-To: rsyslog-users >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>> >>>>> Of course (and as always), the problem rooted in rsyslog. >>>> The commit says it >>>>> all: >>>>> >>>>> >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff >>>> 39b46630d95a8d8 >>>>> 308f383bec1b8be8 >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>> >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: >>>>>>> >>>>>>>> Looks like a problem "with" Debian. I guess I defined >>>> some types that >>>>>> zlib >>>>>>>> also defines to the same names. I have to admit that >>>> Michael Biebl told >>>>>> me, >>>>>>>> but it somehow slipped my mind... Will test on Debian >>>> today and fix. >>>>>>> >>>>>>> master and v5-devel also complained that I didn't have the zlib >>>>>>> development libraries installed (unable to find zlib.h), after I >>>>> installed >>>>>>> that it gave me the error I posted below. >>>>>> >>>>>> I have begun to look into it. Debian has a slightly newer >>>> version of zlib >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's >>>> header the problem >>>>>> occurs. I now need to look at the root cause, but it is >>>> well hidden ;) >>>>>> >>>>>>> >>>>>>> v4 didn't complain about the missing library to start with >>>>>>> >>>>>>> as I posted in another message, I was suprised the the >>>> problem didn't go >>>>>>> away when I configured with --disable-zlib >>>>>> >>>>>> That's a bug in omfile, it obviously does not yet properly >>>> reflect this >>>>>> switch. Will fix that one, too. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile >>>>>>>>> >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: >>>>>>>>> >>>>>>>>>> ubuntu 9.04 >>>>>>>>> >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which machine I was >>>>> compiling >>>>>>>>> on) >>>>>>>>> >>>>>>>>> master also fails with the same error, v4-stable has no problem. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>> >>>>>>>>>> zlib version >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose >>>> compression >>>>>> library >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 >>>>>>>>>> >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 >>>>>>>>>> compression library - development >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> error >>>>>>>>>> >>>>>>>>>> In file included from zlibw.h:28, >>>>>>>>>> from stream.h:72, >>>>>>>>>> from obj.h:50, >>>>>>>>>> from rsyslog.h:482, >>>>>>>>>> from stringbuf.c:39: >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gzseek64' >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', >>>> ';', 'asm' or >>>>>>>>>> '__attribute__' before 'gztell64' >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration >>>> specifiers or >>>>>> '...' >>>>>>>>>> before 'off64_t' >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' >>>>>>>>>> make[1]: *** [all-recursive] Error 1 >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' >>>>>>>>>> make: *** [all] Error 2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> David Lang >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Jul 2 07:57:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 07:57:04 +0200 Subject: [rsyslog] v5-devel won't compile References: <9B6E2A8877C38245BFB15CC491A11DA706FA96@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9C@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FA9F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FAAE@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAB0@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, July 01, 2009 11:23 PM > To: rsyslog-users > Subject: Re: [rsyslog] v5-devel won't compile > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > never mind, pkg-config was not installed on this server > (and I've got _no_ > > idea how I was sucessfully compiling the v4-stable branch > on this box) > > that wasn't it either, it turns out that the libtool package > is needed as > well. > > disabling the testbench doesn't seem to work for me Did you use "--enable-testbench=no"? This definitely works for me... > > (cd .libs && rm -f imfile.la && ln -s ../imfile.la imfile.la) > make[2]: Leaving directory `/usr/src/rsyslog/plugins/imfile' > Making all in tests > make[2]: Entering directory `/usr/src/rsyslog/tests' > CLASSPATH=..:./..:$CLASSPATH javac -d .. DiagTalker.java > /bin/sh: line 6: javac: command not found > make[2]: *** [classcheck.stamp] Error 127 > make[2]: Leaving directory `/usr/src/rsyslog/tests' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/src/rsyslog' > make: *** [all] Error 2 > secdev:/usr/src/rsyslog# history |tail > 1408 ./configure > 1409 ./configure --help > 1410 ./configure --enable-imfile > 1411 make > 1412 ./configure --help |grep test > 1413 ./configure --enable-imfile --enable-testbench=no > 1414 make > 1415 make clean > 1416 make > 1417 history |tail > > > I'm now installing java.. > > David Lang > > > On Wed, 1 Jul 2009, david at lang.hm wrote: > > > >> > >> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >> > >>> This sounds like a couple of files (runtime/rule.c, > runtime/ruleset.c, > >>> runtime/apc.c) are missing. I have just checked, they are > in the git branch. > >>> Also I have no problems compiling that version, I tried > it on Debian this > >>> morning during the zlib fix. > >> > >> hmm, those files were definantly missing. > >> > >> I had done a git checkout origin/v5-devel, I then did a > git checkout -f > >> origin/v5-devel and the files are there. > >> > >> now it is dieing in the configure step :-( > >> > >> checking for FSSTND support... yes > >> ./configure: line 10307: syntax error near unexpected > token `GNUTLS,' > >> ./configure: line 10307: ` PKG_CHECK_MODULES(GNUTLS, gnutls >= > >> 1.4.0)' > >> make: *** [config.status] Error 2 > >> > >> > >> I tried doing the autoreconf command, and it isn't happy > >> > >> #autoreconf -fvi > >> autoreconf2.50: Entering directory `.' > >> autoreconf2.50: configure.ac: not using Gettext > >> autoreconf2.50: running: aclocal --force -I m4 > >> autoreconf2.50: configure.ac: tracing > >> autoreconf2.50: configure.ac: not using Libtool > >> autoreconf2.50: running: /usr/bin/autoconf --force > >> configure.ac:19: error: possibly undefined macro: AC_DISABLE_STATIC > >> If this token and others are legitimate, please use > >> m4_pattern_allow. > >> See the Autoconf documentation. > >> configure.ac:20: error: possibly undefined macro: AC_PROG_LIBTOOL > >> autoreconf2.50: /usr/bin/autoconf failed with exit status: 1 > >> > >> my next step is going to be to nuke the entire directory and try a > >> checkout again. > >> > >> David Lang > >> > >> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com > >>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >>>> Sent: Wednesday, July 01, 2009 7:37 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>> > >>>> back on the debian 5 system I get the following error > with v5-devel > >>>> commit 6322343eca1084b509386a94c1bcf2ca819f1220 > >>>> > >>>> > >>>> gcc -g -O2 -W -Wall -Wformat-security -Wshadow -Wcast-align > >>>> -Wpointer-arith -Wmissing-format-attribute -g -o rsyslogd > >>>> rsyslogd-syslogd.o rsyslogd-omshell.o rsyslogd-omusrmsg.o > >>>> rsyslogd-omfwd.o > >>>> rsyslogd-omfile.o rsyslogd-omdiscard.o rsyslogd-iminternal.o > >>>> rsyslogd-pidfile.o -Wl,--export-dynamic -lz -lpthread > >>>> ../runtime/.libs/librsyslog.a -ldl -lrt > >>>> rsyslogd-syslogd.o: In function `mainloop': > >>>> /usr/src/rsyslog/tools/syslogd.c:2832: undefined reference to > >>>> `execScheduled' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtExit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:216: undefined reference > >>>> to `rulesetClassExit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:217: undefined reference > >>>> to `ruleClassExit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-rsyslog.o): In > >>>> function `rsrtInit': > >>>> /usr/src/rsyslog/runtime/rsyslog.c:151: undefined reference > >>>> to `propClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:175: undefined reference > >>>> to `ruleClassInit' > >>>> /usr/src/rsyslog/runtime/rsyslog.c:177: undefined reference > >>>> to `rulesetClassInit' > >>>> ../runtime/.libs/librsyslog.a(librsyslog_la-obj.o): In > >>>> function `objClassInit': > >>>> /usr/src/rsyslog/runtime/obj.c:1317: undefined reference to > >>>> `apcClassInit' > >>>> collect2: ld returned 1 exit status > >>>> make[2]: *** [rsyslogd] Error 1 > >>>> make[2]: Leaving directory `/usr/src/rsyslog/tools' > >>>> make[1]: *** [all-recursive] Error 1 > >>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>> make: *** [all] Error 2 > >>>> > >>>> > >>>> > >>>> On Wed, 1 Jul 2009, Rainer > >>>> Gerhards wrote: > >>>> > >>>>> Date: Wed, 1 Jul 2009 10:35:56 +0200 > >>>>> From: Rainer Gerhards > >>>>> Reply-To: rsyslog-users > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>> > >>>>> Of course (and as always), the problem rooted in rsyslog. > >>>> The commit says it > >>>>> all: > >>>>> > >>>>> > >>>> http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=f76881fff > >>>> 39b46630d95a8d8 > >>>>> 308f383bec1b8be8 > >>>>> > >>>>> Rainer > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>>>>> Sent: Wednesday, July 01, 2009 10:15 AM > >>>>>> To: rsyslog-users > >>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>> Sent: Wednesday, July 01, 2009 10:11 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>> > >>>>>>> On Wed, 1 Jul 2009, Rainer Gerhards wrote: > >>>>>>> > >>>>>>>> Looks like a problem "with" Debian. I guess I defined > >>>> some types that > >>>>>> zlib > >>>>>>>> also defines to the same names. I have to admit that > >>>> Michael Biebl told > >>>>>> me, > >>>>>>>> but it somehow slipped my mind... Will test on Debian > >>>> today and fix. > >>>>>>> > >>>>>>> master and v5-devel also complained that I didn't > have the zlib > >>>>>>> development libraries installed (unable to find > zlib.h), after I > >>>>> installed > >>>>>>> that it gave me the error I posted below. > >>>>>> > >>>>>> I have begun to look into it. Debian has a slightly newer > >>>> version of zlib > >>>>>> than Fedora (1.2.3.3 vs 1.2.3) and with that version's > >>>> header the problem > >>>>>> occurs. I now need to look at the root cause, but it is > >>>> well hidden ;) > >>>>>> > >>>>>>> > >>>>>>> v4 didn't complain about the missing library to start with > >>>>>>> > >>>>>>> as I posted in another message, I was suprised the the > >>>> problem didn't go > >>>>>>> away when I configured with --disable-zlib > >>>>>> > >>>>>> That's a bug in omfile, it obviously does not yet properly > >>>> reflect this > >>>>>> switch. Will fix that one, too. > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>>> Rainer > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>>>>>>> Sent: Wednesday, July 01, 2009 12:42 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] v5-devel won't compile > >>>>>>>>> > >>>>>>>>> On Tue, 30 Jun 2009, david at lang.hm wrote: > >>>>>>>>> > >>>>>>>>>> ubuntu 9.04 > >>>>>>>>> > >>>>>>>>> correction, debian 5.0, not ubuntu (forgot which > machine I was > >>>>> compiling > >>>>>>>>> on) > >>>>>>>>> > >>>>>>>>> master also fails with the same error, v4-stable > has no problem. > >>>>>>>>> > >>>>>>>>> David Lang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> zlib version > >>>>>>>>>> /* zlib.h -- interface of the 'zlib' general purpose > >>>> compression > >>>>>> library > >>>>>>>>>> version 1.2.3.3, October 2nd, 2006 > >>>>>>>>>> > >>>>>>>>>> ii zlib1g-dev 1:1.2.3.3.dfsg-12 > >>>>>>>>>> compression library - development > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> error > >>>>>>>>>> > >>>>>>>>>> In file included from zlibw.h:28, > >>>>>>>>>> from stream.h:72, > >>>>>>>>>> from obj.h:50, > >>>>>>>>>> from rsyslog.h:482, > >>>>>>>>>> from stringbuf.c:39: > >>>>>>>>>> /usr/include/zlib.h:1366: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gzseek64' > >>>>>>>>>> /usr/include/zlib.h:1367: error: expected '=', ',', > >>>> ';', 'asm' or > >>>>>>>>>> '__attribute__' before 'gztell64' > >>>>>>>>>> /usr/include/zlib.h:1368: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> /usr/include/zlib.h:1369: error: expected declaration > >>>> specifiers or > >>>>>> '...' > >>>>>>>>>> before 'off64_t' > >>>>>>>>>> make[2]: *** [librsyslog_la-stringbuf.lo] Error 1 > >>>>>>>>>> make[2]: Leaving directory `/usr/src/rsyslog/runtime' > >>>>>>>>>> make[1]: *** [all-recursive] Error 1 > >>>>>>>>>> make[1]: Leaving directory `/usr/src/rsyslog' > >>>>>>>>>> make: *** [all] Error 2 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> David Lang > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>> http://www.rsyslog.com > >>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From dirk.schulz at kinzesberg.de Thu Jul 2 10:34:45 2009 From: dirk.schulz at kinzesberg.de (Dirk H. Schulz) Date: Thu, 02 Jul 2009 10:34:45 +0200 Subject: [rsyslog] Rsyslog on NetBSD 5 Message-ID: <4A4C7125.6090109@kinzesberg.de> Hi folks, I am running Rsyslog on RedHat/CentOS and Debian/Ubuntu for quite a while now and am happy with it. Now I would like to test it on NetBSD 5, but I do not find anything on that out there on the web. NetBSD themselves do not offer it in package source, and there is no report on that on the web. Anyone on the list using rsyslog on NetBSD (5)? Any hints or help? Or is it just compile and run? Thanks in advance for any help or link to related information. Dirk From rgerhards at hq.adiscon.com Thu Jul 2 15:29:22 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 15:29:22 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Hi all, with the next released of rsyslog v4/v5, I will introduce so-called "rulesets". They are effectively a sequence of rules that can directly be bound to an input (as long as the input supports this feature, currently not all do). I hope that this mechanism will simplify configuration. It will also enhance performance in most cases (as less filter logic is involved). A howto with detail description is available here: http://www.rsyslog.com/doc-multi_ruleset.html Rainer From tbergfeld at hq.adiscon.com Thu Jul 2 16:11:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 2 Jul 2009 16:11:15 +0200 Subject: [rsyslog] rsyslog 3.22.1 (v3-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACB@GRFEXC.intern.adiscon.com> Hi all, rsyslog 3.22.1, a member of the v3-stable branch, has been released today. The release mainly consists of bugfixes like a fix for an invalid error message issued if $inlcudeConfig was on an empty set of files. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-163.phtml Changelog: http://www.rsyslog.com/Article381.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From aoz.syn at gmail.com Thu Jul 2 17:13:35 2009 From: aoz.syn at gmail.com (RB) Date: Thu, 2 Jul 2009 09:13:35 -0600 Subject: [rsyslog] v4/v5: new mult-ruleset support In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > I hope that this mechanism will simplify configuration. It will also enhance > performance in most cases (as less filter logic is involved). A howto with > detail description is available here: Very pleasing; this is an excellent step toward some of the enhanced configuration semantics I miss from syslog-ng. Perhaps I'm alone here, but I feel that the current syntax is detrimental to complex configurations, largely due to [understandable] compatibility adherence. Oh, to be a college student again and able to take on a SoC project to write a secondary configuration parser. From rgerhards at hq.adiscon.com Thu Jul 2 17:17:19 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Jul 2009 17:17:19 +0200 Subject: [rsyslog] v4/v5: new mult-ruleset support References: <9B6E2A8877C38245BFB15CC491A11DA706FAC6@GRFEXC.intern.adiscon.com> <4255c2570907020813m352356bby93d8d0235c9a1229@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FACC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Thursday, July 02, 2009 5:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] v4/v5: new mult-ruleset support > > On Thu, Jul 2, 2009 at 07:29, Rainer Gerhards wrote: > > I hope that this mechanism will simplify configuration. It will also enhance > > performance in most cases (as less filter logic is involved). A howto with > > detail description is available here: > > Very pleasing; this is an excellent step toward some of the enhanced > configuration semantics I miss from syslog-ng. Yeah, I started with the goal of full compatibility, but some complex things really get hard in that mode. > Perhaps I'm alone > here, but I feel that the current syntax is detrimental to complex > configurations, me, too ;) >largely due to [understandable] compatibility > adherence. Oh, to be a college student again and able to take on a > SoC project to write a secondary configuration parser. The script engine should bring this ... when I am finally able to go to it (doesn't look like I am anytime soon...). Rainer From tbergfeld at hq.adiscon.com Fri Jul 3 11:38:04 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 3 Jul 2009 11:38:04 +0200 Subject: [rsyslog] rsyslog 4.5.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FAE8@GRFEXC.intern.adiscon.com> Hi all, Today, we have released rsyslog 4.5.0, a member of the development branch. This version offers improved performance and reduced memory requirements of the msg object. It further provides you with better config error messages, added capability to fsync() queue disc files for enhanced reliability, new configuration commands and stricter parsing of the hostname in rfc3164 mode. There are also some bug fixes included and the max value for $DynaFileCacheSize has been reduced to a more suitable value. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-164.phtml Changelog: http://www.rsyslog.com/Article380.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Mon Jul 6 14:03:23 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Jul 2009 14:03:23 +0200 Subject: [rsyslog] git branch changes Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB08@GRFEXC.intern.adiscon.com> Hi all, I made a mistake last Friday when creating the new releases. I accidently merged the v5-devel branch into the master branch, which was v4-devel up to then. I now looked at the result and think it is best to drop the v5-devel and create a new v4-devel branch (I can identify the wrong merge and will base v4-devel on it). This hopefully prevents any other such mistake in the future... Somehow my mind is on "master is always the most current". So please use v4-devel for the v4 tree and master for the v5-tree. Once v5 has become / confirmed to be sufficiently stable, I'll stop development in v4 and we will go back to the stable/beta/devel versions with only one active development version. I am right now preparing the new branches, will send a short mail when I am done. I just thought I let you know asap that something went wrong. Sorry for the hassle, Rainer From tbergfeld at hq.adiscon.com Wed Jul 8 16:48:09 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 8 Jul 2009 16:48:09 +0200 Subject: [rsyslog] rsyslog 5.1.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB43@GRFEXC.intern.adiscon.com> Hi all, We have released rsyslog 5.1.2, a new version of the development branch. This version offers improved performance and includes important fixes like a fix for an abort condition when RecvFrom was not set and message reduction was on, this happened e.g. with imuxsock. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-166.phtml Changelog: http://www.rsyslog.com/Article386.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From rgerhards at hq.adiscon.com Thu Jul 9 17:56:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Jul 2009 17:56:36 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Sorry folks, have not worked on this topic since March - but I never forget ;) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, March 04, 2009 8:17 AM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Ok, here is a diff that works. > I have integrated David's UDP spoofing patch into v5-devel (master branch) as a separate output module (named omudpspoof). I have extracted the necessary functionality from omfwd and it now works quite well. However, the configuration is not yet as typically found in rsyslog (not that I find it very elegant, but consistency is a plus ;)). Also, doc is missing. My next steps will be to address these issues. However, for those interested, I just wanted to tell you that the git master branch now contains a version capable of spoofing (I have verified this with both an rsyslog receiver as well as Wireshark - hopefully I can also add an automated test for this functionality, but this is not trivial). > it cycles the source IP address from 32000-42000 (since we are just > sending, and not creating a normal socket this should not matter) > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > to use it create a template that puts the hostname-ip ahead of what you > want to send, similar to > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > %HOSTNAME% %syslogtag%%msg%\n" > > *.* @10.0.0.100;TraditionalFwdFormat > > the one problem right now is that any logs sent from the local box will go > out with a source IP of 127.0.0.1 > > I wasted a bit of time trying to setup filters to use a different template > if $myhostname == $fromhost, but apparently the filtering doesn't allow > comparing two properties I overlooked this in march. The script-based filter engine does not even know that what is being compared are properties, so there is no restriction in what can be compared. So this must either have been a config issue or a bug. I just wanted to tell you should you come into a similar situation again. >, and then I realized that you have a very > high-performance name cache now, so you could easily replace my trivial > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > line with a call to the name lookup and then the %fromhost-ip% could be > replaced by %fromhost% in the template and everything would work sanely > (assuming forward and reverse name resolution are sane ;-) And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > the IP layer options, but I don't know exactly what yet. Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand shows up. Even then, I consider it to be low priority except given very good reasoning ;) Rainer > > I wanted to float this first to see what you think before spending much > more time on it. > > David Lang From doherty at crystal.harvard.edu Thu Jul 9 23:05:30 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Thu, 9 Jul 2009 17:05:30 -0400 Subject: [rsyslog] High CPU utlization, and memory usage Message-ID: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Hi, I recently deployed rsyslog on a server. I'm using a central server that receives logs from currently just 4 other machines. It's using TLS. It's a pretty basic setup, I didn't do anything fancy. Twice this week the central server has had issues, and when I logged in to check, I saw that rsyslog was using all the swap (2GB) and using over 300MB of the 512MB RAM, plus using 70% or more CPU time. I'm running version 4.2.0 Can you give me some idea where to start looking for what's causing this? It seems to run fine for a day or so, and then over a very short amount of time, maybe 2-60 minutes, the memory and cpu usage spikes. Here's a snippet from the /etc/rsyslog.conf if that helps. Thanks. --Peter $ModLoad immark $ModLoad imklog $ModLoad imuxsock # local messages $ModLoad imtcp # TCP listener $MarkMessagePeriod 1200 # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerRun 10514 # start up listener at port 10514 From david at lang.hm Thu Jul 9 23:35:30 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 9 Jul 2009 14:35:30 -0700 (PDT) Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: On Thu, 9 Jul 2009, Peter Doherty wrote: > Hi, I recently deployed rsyslog on a server. I'm using a central > server that receives logs from currently just 4 other machines. It's > using TLS. It's a pretty basic setup, I didn't do anything fancy. > > Twice this week the central server has had issues, and when I logged > in to check, I saw that rsyslog was using all the swap (2GB) and using > over 300MB of the 512MB RAM, plus using 70% or more CPU time. one thing when examining the load is to keep in mind that rsyslog uses multiple threads, in linux hit 'H' in top to have it show the individual threads. it's very possible that the high cpu load is due to the encryption overhead one thing that I do when testing is that once I identify which thread is eating a significant amount of cpu I do 'strace -p ' for that thread for a few seconds, looking at that output I can ususally make a fair guess at what the thread is busy working on. using that much ram would leave me guessing that the ability to write the log file stopped, and the queue is filling up. David Lang > I'm running version 4.2.0 > Can you give me some idea where to start looking for what's causing > this? It seems to run fine for a day or so, and then over a very > short amount of time, maybe 2-60 minutes, the memory and cpu usage > spikes. > > Here's a snippet from the /etc/rsyslog.conf if that helps. > > Thanks. > --Peter > > > $ModLoad immark > $ModLoad imklog > $ModLoad imuxsock # local messages > $ModLoad imtcp # TCP listener > > $MarkMessagePeriod 1200 > > # make gtls driver the default > $DefaultNetstreamDriver gtls > > # certificate files > $DefaultNetstreamDriverCAFile /usr/share/tls/ca.pem > $DefaultNetstreamDriverCertFile /usr/share/tls/server-cert.pem > $DefaultNetstreamDriverKeyFile /usr/share/tls/server-key.pem > > > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode > $InputTCPServerRun 10514 # start up listener at port 10514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Jul 10 14:07:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 10 Jul 2009 14:07:24 +0200 Subject: [rsyslog] UDP source forging. References: <4255c2570902231929sebf2f0cqf9f840389e6d6a0b@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44FCBE@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44FCC2@grfint2.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E8E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71E9A@GRFEXC.intern.adiscon.com><1235670387.28865.2.camel@rf10up.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FB4A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB66@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that I have completed this module now. It is part of the regular master branch. Some basic doc is available here: http://www.rsyslog.com/doc-omudpspoof.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, July 09, 2009 5:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] UDP source forging. > > Sorry folks, have not worked on this topic since March - but I never forget > ;) > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, March 04, 2009 8:17 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] UDP source forging. > > > > Ok, here is a diff that works. > > > > I have integrated David's UDP spoofing patch into v5-devel (master branch) as > a separate output module (named omudpspoof). I have extracted the necessary > functionality from omfwd and it now works quite well. However, the > configuration is not yet as typically found in rsyslog (not that I find it > very elegant, but consistency is a plus ;)). Also, doc is missing. My next > steps will be to address these issues. However, for those interested, I just > wanted to tell you that the git master branch now contains a version capable > of spoofing (I have verified this with both an rsyslog receiver as well as > Wireshark - hopefully I can also add an automated test for this > functionality, but this is not trivial). > > > it cycles the source IP address from 32000-42000 (since we are just > > sending, and not creating a normal socket this should not matter) > > > > it needs LIBS = /usr/lib/libnet.a in the Makefile in tools > > > > to use it create a template that puts the hostname-ip ahead of what you > > want to send, similar to > > > > $template TraditionalFwdFormat,"%fromhost-ip% <%pri%>%timegenerated% > > %HOSTNAME% %syslogtag%%msg%\n" > > > > *.* @10.0.0.100;TraditionalFwdFormat > > > > the one problem right now is that any logs sent from the local box will go > > out with a source IP of 127.0.0.1 > > > > I wasted a bit of time trying to setup filters to use a different template > > if $myhostname == $fromhost, but apparently the filtering doesn't allow > > comparing two properties > > I overlooked this in march. The script-based filter engine does not even know > that what is being compared are properties, so there is no restriction in > what can be compared. So this must either have been a config issue or a bug. > I just wanted to tell you should you come into a similar situation again. > > >, and then I realized that you have a very > > high-performance name cache now, so you could easily replace my trivial > > inet_pton(AF_INET, source_text_ip, &(source_ip.sin_addr)); > > line with a call to the name lookup and then the %fromhost-ip% could be > > replaced by %fromhost% in the template and everything would work sanely > > (assuming forward and reverse name resolution are sane ;-) > > And another point to stress: rsyslogd does *not* yet have a high-performance > cache. All it does is cache the last host (and only the last host). This > works exceptionally well if a large bunch of messages arrive from the same > host, but "cache" performance can easily be thrashed with multiple senders. > All in all, in practice, it works reasonably well, as on a busy system at > least a couple of messages are usually from the same sender. I plan to add a > real cache some time later, personally I would hope to see it this summer. > > > > > I haven't tried to do IPv6 yet, I know that it requires more effort to set > > the IP layer options, but I don't know exactly what yet. > > Omudpspoof is kept IPv4 only and I plan to work on IPv6 only if real demand > shows up. Even then, I consider it to be low priority except given very good > reasoning ;) > > Rainer > > > > > I wanted to float this first to see what you think before spending much > > more time on it. > > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From doherty at crystal.harvard.edu Fri Jul 10 16:04:27 2009 From: doherty at crystal.harvard.edu (Peter Doherty) Date: Fri, 10 Jul 2009 10:04:27 -0400 Subject: [rsyslog] High CPU utlization, and memory usage In-Reply-To: References: <37A1933B-D78A-465D-994D-57D632DD0787@crystal.harvard.edu> Message-ID: <47B915CC-ACD1-496C-B91F-597EC84831E8@crystal.harvard.edu> On Jul 9, 2009, at 5:35 PM, david at lang.hm wrote: > > one thing when examining the load is to keep in mind that rsyslog uses > multiple threads, in linux hit 'H' in top to have it show the > individual > threads. > > it's very possible that the high cpu load is due to the encryption > overhead > > one thing that I do when testing is that once I identify which > thread is > eating a significant amount of cpu I do 'strace -p ' for that > thread > for a few seconds, looking at that output I can ususally make a fair > guess > at what the thread is busy working on. > > using that much ram would leave me guessing that the ability to > write the > log file stopped, and the queue is filling up. > > David Lang I'll have to do a little further testing, but I've got a hunch what caused this state. One of the computers sometimes gets into a state where it starts flooding its syslog with errors from a program. On the order of hundreds of messages a minute. I can correct this one case, but I can't guarantee some other system won't ever have some error that causes it to start flooding it's logs. Are there settings in rsyslog that can control this? Essentially I need something that will prevent a DoS style attack. Something that drops messages from the queue if they come in too fast from a certain host? Or I often see messages from syslogd systems which just say "Last message repeated n times" Can I enable that functionality in rsyslog? --Peter From ko at ibl.fr Fri Jul 10 17:04:25 2009 From: ko at ibl.fr (ko) Date: Fri, 10 Jul 2009 17:04:25 +0200 Subject: [rsyslog] One file per host logging Message-ID: <4A575879.3030308@ibl.fr> Hi, I'm a new-by in rsyslog. I want to do a centralized syslog server, and I want to know if it's possible to do a per host logging, I mean : all log comming from 192.168.1.1 going to /var/log/server1.log all log comming from 192.168.1.2 going to /var/log/server2.log ... and so on... Thanks for your help K. From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 10 17:17:53 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 10 Jul 2009 17:17:53 +0200 Subject: [rsyslog] One file per host logging In-Reply-To: <4A575879.3030308@ibl.fr> References: <4A575879.3030308@ibl.fr> Message-ID: <200907101717.56354.Luis.Fernando.Munoz.Mejias@cern.ch> Ko, > > I'm a new-by in rsyslog. I want to do a centralized syslog server, and > I want to know if it's possible to do a per host logging, I mean : Indeed you can. Just specify the file name as a template. For instance: $template FileNamePerHost,"/var/log/%HOSTNAME%.log" And then, store everything on the files specified by the template: *.* FileNamePerHost You can even have folders per day, just review the documentation on templates and properties. Cheers -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Mon Jul 13 14:17:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 14:17:04 +0200 Subject: [rsyslog] rsyslog - what's next? Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Hi all, I think we made some really good progress with rsyslog's capabilities (feature set & performance) in the past couple of month. I also think I have reached a milestone. As such, I've taken a bit time off to think about where to head to next. And as I'd like to have this available as a handy reference, I've blogged about. So if you are interested in what I intend to focus on the next time (maybe two to three month), please have a look at my blog post: http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html Of course, urgent things will have priority over the general goal, but I am pretty sure the planned effort will have a very positive effect on rsyslog's capabilities and qualities even in the medium term. As always, comments and questions are welcome. Thanks, Rainer From david at lang.hm Mon Jul 13 19:09:32 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 10:09:32 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 13 Jul 2009, Rainer Gerhards wrote: > Hi all, > > I think we made some really good progress with rsyslog's capabilities > (feature set & performance) in the past couple of month. I also think I have > reached a milestone. As such, I've taken a bit time off to think about where > to head to next. And as I'd like to have this available as a handy reference, > I've blogged about. > > So if you are interested in what I intend to focus on the next time (maybe > two to three month), please have a look at my blog post: > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > Of course, urgent things will have priority over the general goal, but I am > pretty sure the planned effort will have a very positive effect on rsyslog's > capabilities and qualities even in the medium term. > > As always, comments and questions are welcome. I don't think that you need to do much for UDP recieve performance. as long as it doesn't need to do DNS lookups it can recieve and insert into a memory queue at full gig-E speeds. the only think you may want to do here is to extend the DNS cache so that instead of only caching the last think you looked up, you cache everything until a restart or HUP (ideally the HUP should be a configurable option) the machines I had on order with the 10G interfaces were misconfigured by the vendor, so I don't have the 10G cards yet :-( when I get them I'll do more testing and see how much faster I can push rsyslog. David Lang From aoz.syn at gmail.com Mon Jul 13 19:38:17 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 11:38:17 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> On Mon, Jul 13, 2009 at 11:09, wrote: > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) I haven't looked at the DNS cache code, but unless you're also caching and handling the records' TTL, blindly caching DNS records within the app is a dark road to go down. Some apps (namely browsers) do it anyway, but get away with it by setting their internal TTL significantly lower than that of a typical record. What kind of performance hit are you actually seeing with DNS lookups? From david at lang.hm Mon Jul 13 20:11:02 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 11:11:02 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 11:09, wrote: >> is to extend the DNS cache so that instead of only caching the last think >> you looked up, you cache everything until a restart or HUP (ideally the >> HUP should be a configurable option) > > I haven't looked at the DNS cache code, but unless you're also caching > and handling the records' TTL, blindly caching DNS records within the > app is a dark road to go down. Some apps (namely browsers) do it > anyway, but get away with it by setting their internal TTL > significantly lower than that of a typical record. for Internet access I completely agree with you, but for a log server you don't have IPs changing names very frequently. As a result it becomes practical to cache the lookup unconditionally until a restart/HUP. even in a 'highly dynamic' environment you are usually adding servers instead of changing the names of IP addresses. note that caching the lookup at all can be disabled via a config option > What kind of performance hit are you actually seeing with DNS lookups? 6 months ago it was a factor of 10x to 100x (it's probably more now since rsyslog has gotten faster) the problem is that to do a name lookup requires a LOT of steps first the name resolver library looks several places on the filesystem for various config files then it reads /etc/hosts, parses it, and looks for the name/IP then it makes a network connection to the nameserver, sends a message, waits for the reciever to parse the message and look it up in it's datastore, then send the message back to the requester who needs to parse the message and if you network packet gets dropped due to congestion on the network, you wait 30 seconds and retry. doing all of this for every UDP syslog packet that you recieve results in a _lot_ of system calls, and a lot of delays. even if you have the name in the /etc/hosts file, the overhead of looking in the filesystem and parsing the file is significant. without doing DNS lookups, rsyslog is able to recieve UDP logs at almost 400,000 per second (effectivly Gig-E wire speed with 256 byte log messages), _very_ few DNS servers can handle requests at that speed. in fact, at that request rate, you use about half of a Gig-E connection just for the DNS requests (more if you request more data, like what the TTL would be, or don't give it the best name the first time and need to make multiple requests with different domains attached or something like that) David Lang From aoz.syn at gmail.com Mon Jul 13 21:43:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 13:43:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> Message-ID: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> On Mon, Jul 13, 2009 at 12:11, wrote: > for Internet access I completely agree with you, but for a log server you > don't have IPs changing names very frequently. As a result it becomes > practical to cache the lookup unconditionally until a restart/HUP. even in > a 'highly dynamic' environment you are usually adding servers instead of > changing the names of IP addresses. The problem with that approach is that it assumes it is acceptable to interrupt service for any trivial DNS change. How do you propose to deal withDNS-level availability tools like round-robin or Cisco's GSS? There are many approaches that use DNS as both a scalability and a redundancy tool that TTL-agnostic caching will not interact well with. > the problem is that to do a name lookup requires a LOT of steps I'm well aware of the series of steps, but am unconvinced caching belongs in the application. Have you tried using local caching (nscd) or segregating the DNS traffic from the production syslog traffic? Most enterprise-grade configurations (for whatever app) I've seen use a separate interface for administration and metadata like reverse lookups anyway. > even if you have the name in the /etc/hosts file, the overhead of looking > in the filesystem and parsing the file is significant. How significant? Unless the host is poorly configured, /etc/hosts is going to be in the filesystem cache and presented at near-memory speeds until it gets invalidated or evicted. > without doing DNS lookups, rsyslog is able to recieve UDP logs at almost > 400,000 per second (effectivly Gig-E wire speed with 256 byte log > messages), _very_ few DNS servers can handle requests at that speed. in To clarify since I'm not looking at the code right now: are you saying rsyslog performs a blocking lookup per-packet? If so that's certainly a sub-optimal approach, but there are often better ways (namely delayed resolution) to fix that than breaking RFC compatibility. > fact, at that request rate, you use about half of a Gig-E connection just > for the DNS requests (more if you request more data, like what the TTL > would be, or don't give it the best name the first time and need to make > multiple requests with different domains attached or something like that) Your example invalidates itself - the TTL comes with the query whether or not you use it, and any caching mechanism (TTL-sensitive or not) will shortcut such sub-optimal queries after the first one. Caching is a good thing as long as it's done in a compliant manner or documented in a fashion that clearly identifies it as a broken (if performant) approach. From ktm at rice.edu Mon Jul 13 21:37:57 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 13 Jul 2009 14:37:57 -0500 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> Message-ID: <20090713193757.GB4988@it.is.rice.edu> On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > Hi all, > > > > I think we made some really good progress with rsyslog's capabilities > > (feature set & performance) in the past couple of month. I also think I have > > reached a milestone. As such, I've taken a bit time off to think about where > > to head to next. And as I'd like to have this available as a handy reference, > > I've blogged about. > > > > So if you are interested in what I intend to focus on the next time (maybe > > two to three month), please have a look at my blog post: > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > Of course, urgent things will have priority over the general goal, but I am > > pretty sure the planned effort will have a very positive effect on rsyslog's > > capabilities and qualities even in the medium term. > > > > As always, comments and questions are welcome. > > I don't think that you need to do much for UDP recieve performance. as > long as it doesn't need to do DNS lookups it can recieve and insert into a > memory queue at full gig-E speeds. the only think you may want to do here > is to extend the DNS cache so that instead of only caching the last think > you looked up, you cache everything until a restart or HUP (ideally the > HUP should be a configurable option) > > the machines I had on order with the 10G interfaces were misconfigured by > the vendor, so I don't have the 10G cards yet :-( when I get them I'll do > more testing and see how much faster I can push rsyslog. > > David Lang I agree with David. DNS lookups can really sap both bandwidth and CPU. I am not sure that caching everything since a restart is a good idea though. Even if you could do a lookup, cache the TTL and IP and then relookup after the TTL that would be great. Even having it re-query the DNS in a worker thread to sweep through the cached values periodically would really help performance. Regards, Ken From rgerhards at hq.adiscon.com Mon Jul 13 22:25:07 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Jul 2009 22:25:07 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <20090713193757.GB4988@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB80@GRFEXC.intern.adiscon.com> One important point to stress is that rsyslog does NOT yet have a real DNS cache! Let me quote my mail from July, 9th[1]: == And another point to stress: rsyslogd does *not* yet have a high-performance cache. All it does is cache the last host (and only the last host). This works exceptionally well if a large bunch of messages arrive from the same host, but "cache" performance can easily be thrashed with multiple senders. All in all, in practice, it works reasonably well, as on a busy system at least a couple of messages are usually from the same sender. I plan to add a real cache some time later, personally I would hope to see it this summer. == Adding a the cache is not really a focus activity - I think it could be done in a couple of days. I also personally do not see an issue with obying TTL - or discarding entries every five minutes or so. In my POV, it doesn't really matter if they are quickly discarded. If there is low traffic, it doesn't make a difference at all. If we handle hundereds of thousands message per second, one lockup every few billion of messages doesn't really matter either. The compare operation is relatively fast and will not add significantly to the runtime (at least I'd guess that). In regard to imudp, I think there is a couple of more optimizations possible. I'd expect that the biggest hit (after DNS lookup) is switching from select to epoll. Re-using property copying is another one (reduces expensive memory writes). A batching interface seems also possible. Enhanced pipelining is another possibility. I also think that the current solution will gain very different results based on the communication patterns. So rather than (educated) guessing, I would prefer to have some good instrumentation to get a grip on all this variants. Also, imudp is just a sample, there are more areas that could benefit from a focussed analytic view. Plus that performance tools can be used to stress-testing and thus finding otherwise hard to find bugs. Thus I think it is worth spending some time on an improved toolset (and will pay off in the long term). Rainer [1] http://lists.adiscon.net/pipermail/rsyslog/2009-July/002432.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Monday, July 13, 2009 9:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 10:09:32AM -0700, david at lang.hm wrote: > > On Mon, 13 Jul 2009, Rainer Gerhards wrote: > > > > > Hi all, > > > > > > I think we made some really good progress with rsyslog's > capabilities > > > (feature set & performance) in the past couple of month. > I also think I have > > > reached a milestone. As such, I've taken a bit time off > to think about where > > > to head to next. And as I'd like to have this available > as a handy reference, > > > I've blogged about. > > > > > > So if you are interested in what I intend to focus on the > next time (maybe > > > two to three month), please have a look at my blog post: > > > > > > http://blog.gerhards.net/2009/07/rsyslog-where-are-we.html > > > > > > Of course, urgent things will have priority over the > general goal, but I am > > > pretty sure the planned effort will have a very positive > effect on rsyslog's > > > capabilities and qualities even in the medium term. > > > > > > As always, comments and questions are welcome. > > > > I don't think that you need to do much for UDP recieve > performance. as > > long as it doesn't need to do DNS lookups it can recieve > and insert into a > > memory queue at full gig-E speeds. the only think you may > want to do here > > is to extend the DNS cache so that instead of only caching > the last think > > you looked up, you cache everything until a restart or HUP > (ideally the > > HUP should be a configurable option) > > > > the machines I had on order with the 10G interfaces were > misconfigured by > > the vendor, so I don't have the 10G cards yet :-( when I > get them I'll do > > more testing and see how much faster I can push rsyslog. > > > > David Lang > > I agree with David. DNS lookups can really sap both bandwidth and CPU. > I am not sure that caching everything since a restart is a good idea > though. Even if you could do a lookup, cache the TTL and IP and then > relookup after the TTL that would be great. Even having it re-query > the DNS in a worker thread to sweep through the cached values > periodically > would really help performance. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Jul 13 22:38:52 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 13:38:52 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 12:11, wrote: >> for Internet access I completely agree with you, but for a log server you >> don't have IPs changing names very frequently. As a result it becomes >> practical to cache the lookup unconditionally until a restart/HUP. even in >> a 'highly dynamic' environment you are usually adding servers instead of >> changing the names of IP addresses. > > The problem with that approach is that it assumes it is acceptable to > interrupt service for any trivial DNS change. How do you propose to > deal withDNS-level availability tools like round-robin or Cisco's GSS? > There are many approaches that use DNS as both a scalability and a > redundancy tool that TTL-agnostic caching will not interact well with. HUP does not interrupt services. >> the problem is that to do a name lookup requires a LOT of steps > > I'm well aware of the series of steps, but am unconvinced caching > belongs in the application. Have you tried using local caching (nscd) > or segregating the DNS traffic from the production syslog traffic? > Most enterprise-grade configurations (for whatever app) I've seen use > a separate interface for administration and metadata like reverse > lookups anyway. yes, doing a local DNS cacheing server does not avoid all the local filesystem lookups and host file parsing. it does speed up the network connection. >> even if you have the name in the /etc/hosts file, the overhead of looking >> in the filesystem and parsing the file is significant. > > How significant? Unless the host is poorly configured, /etc/hosts is > going to be in the filesystem cache and presented at near-memory > speeds until it gets invalidated or evicted. the system calls to access it (and to first lookup if it should be accessed at all) take enough time to be significant at these speeds. >> without doing DNS lookups, rsyslog is able to recieve UDP logs at almost >> 400,000 per second (effectivly Gig-E wire speed with 256 byte log >> messages), _very_ few DNS servers can handle requests at that speed. in > > To clarify since I'm not looking at the code right now: are you saying > rsyslog performs a blocking lookup per-packet? If so that's certainly > a sub-optimal approach, but there are often better ways (namely > delayed resolution) to fix that than breaking RFC compatibility. correct, it does all input processing in a single thread. so if that thread is stalled doing a lookup it's not processing other packets. when the data is first inserted into the message queue it is complete with all lookups done. >> fact, at that request rate, you use about half of a Gig-E connection just >> for the DNS requests (more if you request more data, like what the TTL >> would be, or don't give it the best name the first time and need to make >> multiple requests with different domains attached or something like that) > > Your example invalidates itself - the TTL comes with the query whether > or not you use it, and any caching mechanism (TTL-sensitive or not) > will shortcut such sub-optimal queries after the first one. > > Caching is a good thing as long as it's done in a compliant manner or > documented in a fashion that clearly identifies it as a broken (if > performant) approach. you want the ability to cache the name lookup no matter what method is used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include a TTL. I agree that the details of how the cache works should be documented. David Lang From aoz.syn at gmail.com Mon Jul 13 23:24:10 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 15:24:10 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> Message-ID: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> On Mon, Jul 13, 2009 at 14:38, wrote: > HUP does not interrupt services. I'm sorry; last October you noted that for certain configurations SIGHUP will drop memory-queued messages. In addition to the other notes in the thread (tearing down TCP connections, etc.), it sounds an awful lot like a service interruption. Has that since changed for 3.x or 4.2? > the system calls to access it (and to first lookup if it should be > accessed at all) take enough time to be significant at these speeds. May I gently prod you to substantiate this statement? I don't doubt that it makes calls to external libraries, and that those calls are likely more expensive than internal resolution, but "significant" is not significant without numbers to back it up. Even a simple speed comparison for a few million packets between 'no resolution' and 'in /etc/hosts with a cold cache' would be useful. > you want the ability to cache the name lookup no matter what method is > used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > a TTL. I have no problem expanding the scope of discussion to resolution methods beyond DNS (which was the original premise), but if a name service does not provide an explicit TTL, it must be assumed as an implicit '0', which blind caching will break. That said, each of those methods still provides some [internal] method of caching and invalidation that, to be audit-grade correct, rsyslog will either have to replicate or trust. I still can't see a problem with having something to the effect of a "$BreakNSCache" configuration option, but by default a syslog daemon should play as strictly by the rules as possible. Then again, I'm RB, not RG. ;) From david at lang.hm Mon Jul 13 23:58:33 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 14:58:33 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 14:38, wrote: >> HUP does not interrupt services. > > I'm sorry; last October you noted that for certain configurations > SIGHUP will drop memory-queued messages. In addition to the other > notes in the thread (tearing down TCP connections, etc.), it sounds an > awful lot like a service interruption. Has that since changed for 3.x > or 4.2? yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not do a complete teardown, halt, and restart, but instead just closes and re-opens files and connections so that no long messages are lost in a restart. this matches the traditional use of HUP to get syslogd to release the files it's writing to so that they can be rotated away. it doesn't re-read the config file due to the fact that the rsyslog config file is so complex and can significantly alter the software by loading modules. >> the system calls to access it (and to first lookup if it should be >> accessed at all) take enough time to be significant at these speeds. > > May I gently prod you to substantiate this statement? I don't doubt > that it makes calls to external libraries, and that those calls are > likely more expensive than internal resolution, but "significant" is > not significant without numbers to back it up. Even a simple speed > comparison for a few million packets between 'no resolution' and 'in > /etc/hosts with a cold cache' would be useful. I would have to do a new set of tests to get you concrete numbers, but back in roughly the october timeframe last year I was doing extensive tests on this and with hosts files vs. no resolution I was seeing 4x or more differences (with a tiny, 5-line hosts file to parse) at the time I think I discussed it on a long performance thread on the website. at the time I did quite a number of strace dumps to show how much time was being eaten up in the system calls. RG has done a LOT of cleanup since then (to the point that the failsafe audit mode is in the ballpark of being as fast as the in-memory mode was back then) >> you want the ability to cache the name lookup no matter what method is >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include >> a TTL. > > I have no problem expanding the scope of discussion to resolution > methods beyond DNS (which was the original premise), but if a name > service does not provide an explicit TTL, it must be assumed as an > implicit '0', which blind caching will break. That said, each of > those methods still provides some [internal] method of caching and > invalidation that, to be audit-grade correct, rsyslog will either have > to replicate or trust. I still can't see a problem with having > something to the effect of a "$BreakNSCache" configuration option, but > by default a syslog daemon should play as strictly by the rules as > possible. well, if you are really wanting audit-grade logging, will you use anything other than the IP address (or what is already in the message)? any lookup that you do is a potential for corruption. I'm fine with the default being to do a lookup for each item. I just want a way to avoid it, and if I can do that with a cache instead of having to disable DNS, I will do so. it's very common for servers to need to disable DNS lookups for their logs. do a quick search for apache dns performance and you will find that it's a very common problem people have there if they enable lookups on the sender's IP address. > Then again, I'm RB, not RG. ;) more people speaking up is good. I have strong opinions, and real problems to address, but RG needs to hear from people other than me. David Lang From aoz.syn at gmail.com Tue Jul 14 05:12:57 2009 From: aoz.syn at gmail.com (RB) Date: Mon, 13 Jul 2009 21:12:57 -0600 Subject: [rsyslog] rsyslog - what's next? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> On Mon, Jul 13, 2009 at 15:58, wrote: > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. Hopefully some of the tooling Rainer is considering will better enable performance monitoring and we can test the effect of some of these suggested changes. I don't completely discount strace as a performance testing tool, but it's inherently intrusive, making Heisenberg quite happy. > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address Indeed; I've generally eschewed DNS for such critical pieces of infrastructure, instead opting for resolution during analysis and correlation with other logs. That largely stems from network security work and the associated disdain for using DNS entries for rules. Even so, I'm curious whether a properly tuned library-integrated cache like nscd (as opposed to a local caching resolver) would provide a sufficient performance increase in the interim. I spent a couple of hours this evening digging through the related code with the intent of starting work on delayed resolution, moving the lookup to runtime/msg.c:getHOSTNAME. It might not be the right place, but would have the advantage of only resolving hosts you explicitly request it for. Unfortunately, the AllowedSender pragma is a sticking point, forcing per-packet resolution; otherwise, it seemed a pretty trivial conversion. I don't know how much (if any) performance increase that approach would grant, but it would at least move a blocking call out of that single thread. I may experiment with adding a "NoAllowedSenderDNS" later, but it's beyond my time flexibility right now. From david at lang.hm Tue Jul 14 06:54:37 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Jul 2009 21:54:37 -0700 (PDT) Subject: [rsyslog] rsyslog - what's next? In-Reply-To: <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: On Mon, 13 Jul 2009, RB wrote: > On Mon, Jul 13, 2009 at 15:58, wrote: >> at the time I did quite a number of strace dumps to show how much time was >> being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. absolutly, especially when you enable timing of the calls !!! (a gettimeofday call before and after evey syscall) but if the program hasn't been examined to try and reduce the number of syscalls it makes, strace is a good place to start. once you get down to a minimum of nessasary syscalls, strace stops being useful. >> well, if you are really wanting audit-grade logging, will you use anything >> other than the IP address (or what is already in the message)? any lookup >> that you do is a potential for corruption. >> >> I'm fine with the default being to do a lookup for each item. I just want >> a way to avoid it, and if I can do that with a cache instead of having to >> disable DNS, I will do so. >> >> it's very common for servers to need to disable DNS lookups for their >> logs. do a quick search for apache dns performance and you will find that >> it's a very common problem people have there if they enable lookups on the >> sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. I'll try and get a chance to look into that > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. interesting thought. David Lang From rgerhards at hq.adiscon.com Tue Jul 14 07:17:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 07:17:24 +0200 Subject: [rsyslog] rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB82@GRFEXC.intern.adiscon.com> I am digesting this thread (just got up ;)), but let me spell out something explicitely: the traditional perception of what is fast and what is slow is no longer valid if you aim at a rate of several hunderd-thousand messages per second. For example, it then makes a difference if you (memory) write a single 32 byte buffer per record or not. It may be in the region of 100 to 500 records per second, but these differences sum up. So trying to avoid such writes, even a the costs of (currently much cheaper) reads is something to aim at. I'll see that I get together a more elaborate description inside my blog, probably it is useful. I learned a lot of lessons the past nine month ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, July 14, 2009 6:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 15:58, wrote: > >> at the time I did quite a number of strace dumps to show > how much time was > >> being eaten up in the system calls. > > > > Hopefully some of the tooling Rainer is considering will > better enable > > performance monitoring and we can test the effect of some of these > > suggested changes. I don't completely discount strace as a > > performance testing tool, but it's inherently intrusive, making > > Heisenberg quite happy. > > absolutly, especially when you enable timing of the calls !!! (a > gettimeofday call before and after evey syscall) > > but if the program hasn't been examined to try and reduce the > number of > syscalls it makes, strace is a good place to start. > > once you get down to a minimum of nessasary syscalls, strace > stops being > useful. > > >> well, if you are really wanting audit-grade logging, will > you use anything > >> other than the IP address (or what is already in the > message)? any lookup > >> that you do is a potential for corruption. > >> > >> I'm fine with the default being to do a lookup for each > item. I just want > >> a way to avoid it, and if I can do that with a cache > instead of having to > >> disable DNS, I will do so. > >> > >> it's very common for servers to need to disable DNS > lookups for their > >> logs. do a quick search for apache dns performance and you > will find that > >> it's a very common problem people have there if they > enable lookups on the > >> sender's IP address > > > > Indeed; I've generally eschewed DNS for such critical pieces of > > infrastructure, instead opting for resolution during analysis and > > correlation with other logs. That largely stems from > network security > > work and the associated disdain for using DNS entries for > rules. Even > > so, I'm curious whether a properly tuned library-integrated > cache like > > nscd (as opposed to a local caching resolver) would provide a > > sufficient performance increase in the interim. > > I'll try and get a chance to look into that > > > I spent a couple of hours this evening digging through the related > > code with the intent of starting work on delayed resolution, moving > > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > > place, but would have the advantage of only resolving hosts you > > explicitly request it for. Unfortunately, the > AllowedSender pragma is > > a sticking point, forcing per-packet resolution; otherwise, > it seemed > > a pretty trivial conversion. I don't know how much (if any) > > performance increase that approach would grant, but it > would at least > > move a blocking call out of that single thread. I may > experiment with > > adding a "NoAllowedSenderDNS" later, but it's beyond my time > > flexibility right now. > > interesting thought. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 10:37:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 10:37:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, July 13, 2009 11:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, 13 Jul 2009, RB wrote: > > > On Mon, Jul 13, 2009 at 14:38, wrote: > >> HUP does not interrupt services. > > > > I'm sorry; last October you noted that for certain configurations > > SIGHUP will drop memory-queued messages. In addition to the other > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > awful lot like a service interruption. Has that since changed for 3.x > > or 4.2? > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP not > do a complete teardown, halt, and restart, but instead just closes and > re-opens files and connections so that no long messages are lost in a > restart. And that is the default setting! > > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. For v5, I actually think of adding a "rsyslogd restart deamon" for those that insist on traditional HUP semantics. That deamon would lie dormant in memory until it receives HUP, in which case it does a shutdown and restart of the real rsyslogd. With that, I could cleanup a lot of complexity. One major issue is that under some circumstances I need to cancel output threads when they do not finish their processing within the allotted timeouts. Canceling threads is always a bit dangerous, but there currently is no always-reliable way around this. To make it happen, a lot of enable/disable cancel calls, including cancel cleanup handlers are made throughout the code. If we would not have restart-type HUPs, I could simply cancel those threads and not care about resources not being cleaned up (the process cleanup will take care of that). So I could also remove all the enable/disable cancel logic and its helpers. Of course, this is not something done as quickly as I write those lines and it must be made sure that any inconsistence does not affect the rest of the shutdown. But without those annoying restart-type HUPs, it is much simpler to do... > > >> the system calls to access it (and to first lookup if it should be > >> accessed at all) take enough time to be significant at these speeds. > > > > May I gently prod you to substantiate this statement? I don't doubt > > that it makes calls to external libraries, and that those calls are > > likely more expensive than internal resolution, but "significant" is > > not significant without numbers to back it up. Even a simple speed > > comparison for a few million packets between 'no resolution' and 'in > > /etc/hosts with a cold cache' would be useful. > > I would have to do a new set of tests to get you concrete numbers, but > back in roughly the october timeframe last year I was doing extensive > tests on this and with hosts files vs. no resolution I was seeing 4x or > more differences (with a tiny, 5-line hosts file to parse) > > at the time I think I discussed it on a long performance thread on the > website. > > at the time I did quite a number of strace dumps to show how much time was > being eaten up in the system calls. RG has done a LOT of cleanup since > then (to the point that the failsafe audit mode is in the ballpark of > being as fast as the in-memory mode was back then) > > >> you want the ability to cache the name lookup no matter what method is > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not include > >> a TTL. > > > > I have no problem expanding the scope of discussion to resolution > > methods beyond DNS (which was the original premise), but if a name > > service does not provide an explicit TTL, it must be assumed as an > > implicit '0', which blind caching will break. That said, each of > > those methods still provides some [internal] method of caching and > > invalidation that, to be audit-grade correct, rsyslog will either have > > to replicate or trust. I still can't see a problem with having > > something to the effect of a "$BreakNSCache" configuration option, but > > by default a syslog daemon should play as strictly by the rules as > > possible. > > well, if you are really wanting audit-grade logging, will you use anything > other than the IP address (or what is already in the message)? any lookup > that you do is a potential for corruption. > > I'm fine with the default being to do a lookup for each item. I just want > a way to avoid it, and if I can do that with a cache instead of having to > disable DNS, I will do so. > > it's very common for servers to need to disable DNS lookups for their > logs. do a quick search for apache dns performance and you will find that > it's a very common problem people have there if they enable lookups on the > sender's IP address. > > > Then again, I'm RB, not RG. ;) > > more people speaking up is good. I have strong opinions, and real problems > to address, but RG needs to hear from people other than me. Definitely! I often simply do not realize where a need is, and I always often make mistakes or have wrong perceptions. If nobody objects, I am lost and so is the project ;) Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 11:02:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 11:02:40 +0200 Subject: [rsyslog] DNS performance - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <4255c2570907132012n502d3f5dt1b063f59f1a7e2b7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB8B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Tuesday, July 14, 2009 5:13 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog - what's next? > > On Mon, Jul 13, 2009 at 15:58, wrote: > > at the time I did quite a number of strace dumps to show how much time was > > being eaten up in the system calls. > > Hopefully some of the tooling Rainer is considering will better enable > performance monitoring and we can test the effect of some of these > suggested changes. I don't completely discount strace as a > performance testing tool, but it's inherently intrusive, making > Heisenberg quite happy. That's exactly part of the "new tools" idea, but obviously this instrumentation will also affect the overall picture. But I think I can do the base counters with a few atomic increments, so the effect should be not that bad. The real work shall then be done in the front-end. > > > well, if you are really wanting audit-grade logging, will you use anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on the > > sender's IP address > > Indeed; I've generally eschewed DNS for such critical pieces of > infrastructure, instead opting for resolution during analysis and > correlation with other logs. That largely stems from network security > work and the associated disdain for using DNS entries for rules. Even > so, I'm curious whether a properly tuned library-integrated cache like > nscd (as opposed to a local caching resolver) would provide a > sufficient performance increase in the interim. > > I spent a couple of hours this evening digging through the related > code with the intent of starting work on delayed resolution, moving > the lookup to runtime/msg.c:getHOSTNAME. It might not be the right > place, but would have the advantage of only resolving hosts you > explicitly request it for. Unfortunately, the AllowedSender pragma is > a sticking point, forcing per-packet resolution; otherwise, it seemed > a pretty trivial conversion. I don't know how much (if any) > performance increase that approach would grant, but it would at least > move a blocking call out of that single thread. I may experiment with > adding a "NoAllowedSenderDNS" later, but it's beyond my time > flexibility right now. That's indeed an interesting thought. The allowedSenders is not really that much of a problem. First of all, not all inputs support them (one may argue they should, but one may also argue that a firewall is a much better place to block traffic...). Secondly, $AllowedSender is not used in most configurations. Rsyslog can detect that the sender name is not needed and in this case can actually defer the name lookup. Of course, that doesn't help in situations where the lookup is required, but it helps in many configurations. This is also the spirit of some of my recent performance enhancements: not all work equally well under all circumstances. But I have tried to make only those suffer that actually need the feature that bears the cost. I hope to extend that idea to the queue, where an ultra-fast mode may automatically be selected if the parameters indicate that is possible. But, again, some tooling to test this and its affects would be quite useful. Rainer From iamsayan at gmail.com Tue Jul 14 17:52:53 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:52:53 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Sorry if this is out of context. > this matches the traditional use of HUP to get syslogd to release the > files it's writing to so that they can be rotated away. > > it doesn't re-read the config file due to the fact that the rsyslog config > file is so complex and can significantly alter the software by loading > modules. I am still tempted to remove the non-restart type of HUP. Actually, it is the root cause for a lot of complexities (read performance bottlenecks) inside the engine. And all this for something that is not strictly needed. However, there are always many opponents when I intend to remove the traditional HUP behavior. Is there any other way to make the rsyslog process reread the conf file other than a restart? This is the only reason I use the traditional SIGHUP processing, I send HUP after I alter the rsyslog.conf file. On Tue, Jul 14, 2009 at 4:37 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Monday, July 13, 2009 11:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog - what's next? > > > > On Mon, 13 Jul 2009, RB wrote: > > > > > On Mon, Jul 13, 2009 at 14:38, wrote: > > >> HUP does not interrupt services. > > > > > > I'm sorry; last October you noted that for certain configurations > > > SIGHUP will drop memory-queued messages. In addition to the other > > > notes in the thread (tearing down TCP connections, etc.), it sounds an > > > awful lot like a service interruption. Has that since changed for 3.x > > > or 4.2? > > > > yes, for 4.2 there is the option of 'HUPisRestart no', which makes HUP > not > > do a complete teardown, halt, and restart, but instead just closes and > > re-opens files and connections so that no long messages are lost in a > > restart. > > And that is the default setting! > > > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. > However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > For v5, I actually think of adding a "rsyslogd restart deamon" for those > that > insist on traditional HUP semantics. That deamon would lie dormant in > memory > until it receives HUP, in which case it does a shutdown and restart of the > real rsyslogd. With that, I could cleanup a lot of complexity. > > One major issue is that under some circumstances I need to cancel output > threads when they do not finish their processing within the allotted > timeouts. Canceling threads is always a bit dangerous, but there currently > is > no always-reliable way around this. To make it happen, a lot of > enable/disable cancel calls, including cancel cleanup handlers are made > throughout the code. If we would not have restart-type HUPs, I could simply > cancel those threads and not care about resources not being cleaned up (the > process cleanup will take care of that). So I could also remove all the > enable/disable cancel logic and its helpers. Of course, this is not > something > done as quickly as I write those lines and it must be made sure that any > inconsistence does not affect the rest of the shutdown. But without those > annoying restart-type HUPs, it is much simpler to do... > > > > >> the system calls to access it (and to first lookup if it should be > > >> accessed at all) take enough time to be significant at these speeds. > > > > > > May I gently prod you to substantiate this statement? I don't doubt > > > that it makes calls to external libraries, and that those calls are > > > likely more expensive than internal resolution, but "significant" is > > > not significant without numbers to back it up. Even a simple speed > > > comparison for a few million packets between 'no resolution' and 'in > > > /etc/hosts with a cold cache' would be useful. > > > > I would have to do a new set of tests to get you concrete numbers, but > > back in roughly the october timeframe last year I was doing extensive > > tests on this and with hosts files vs. no resolution I was seeing 4x or > > more differences (with a tiny, 5-line hosts file to parse) > > > > at the time I think I discussed it on a long performance thread on the > > website. > > > > at the time I did quite a number of strace dumps to show how much time > was > > being eaten up in the system calls. RG has done a LOT of cleanup since > > then (to the point that the failsafe audit mode is in the ballpark of > > being as fast as the in-memory mode was back then) > > > > >> you want the ability to cache the name lookup no matter what method is > > >> used. only DNS provides a TTL, hosts files, NIS, LDAP, etc do not > include > > >> a TTL. > > > > > > I have no problem expanding the scope of discussion to resolution > > > methods beyond DNS (which was the original premise), but if a name > > > service does not provide an explicit TTL, it must be assumed as an > > > implicit '0', which blind caching will break. That said, each of > > > those methods still provides some [internal] method of caching and > > > invalidation that, to be audit-grade correct, rsyslog will either have > > > to replicate or trust. I still can't see a problem with having > > > something to the effect of a "$BreakNSCache" configuration option, but > > > by default a syslog daemon should play as strictly by the rules as > > > possible. > > > > well, if you are really wanting audit-grade logging, will you use > anything > > other than the IP address (or what is already in the message)? any lookup > > that you do is a potential for corruption. > > > > I'm fine with the default being to do a lookup for each item. I just want > > a way to avoid it, and if I can do that with a cache instead of having to > > disable DNS, I will do so. > > > > it's very common for servers to need to disable DNS lookups for their > > logs. do a quick search for apache dns performance and you will find that > > it's a very common problem people have there if they enable lookups on > the > > sender's IP address. > > > > > Then again, I'm RB, not RG. ;) > > > > more people speaking up is good. I have strong opinions, and real > problems > > to address, but RG needs to hear from people other than me. > > Definitely! I often simply do not realize where a need is, and I always > often > make mistakes or have wrong perceptions. If nobody objects, I am lost and > so > is the project ;) > > Rainer > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 17:55:48 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 17:55:48 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Sorry if this is out of context. > > > this matches the traditional use of HUP to get syslogd to release the > > files it's writing to so that they can be rotated away. > > > > it doesn't re-read the config file due to the fact that the rsyslog config > > file is so complex and can significantly alter the software by loading > > modules. > > I am still tempted to remove the non-restart type of HUP. Actually, it is > the > root cause for a lot of complexities (read performance bottlenecks) inside > the engine. And all this for something that is not strictly needed. However, > there are always many opponents when I intend to remove the traditional HUP > behavior. > > > Is there any other way to make the rsyslog process reread the conf file > other than a restart? > This is the only reason I use the traditional SIGHUP processing, I send HUP > after I alter the rsyslog.conf file. No, but I always fail to see the big difference between typing $ kill -HUP ?cat /var/run/rsyslog.log? And $ /etc/rc.d/rsyslogd restart ;) Use the later one, and you do not need to use sighup. Even better, you can now use HUP for a lightweight "close the files only" log rotation thing ;) Rainer From iamsayan at gmail.com Tue Jul 14 17:58:58 2009 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Tue, 14 Jul 2009 11:58:58 -0400 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com> <4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com> <4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com> <4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com> <87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> Message-ID: <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Yes exactly, I have changed the script to do just that recently , I was just wondering whether this is the reason why you get so much resistance when you want to deprecate the traditional HUP processing... On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:53 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > next? > > > > Sorry if this is out of context. > > > > > this matches the traditional use of HUP to get syslogd to release the > > > files it's writing to so that they can be rotated away. > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > config > > > file is so complex and can significantly alter the software by loading > > > modules. > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > the > > root cause for a lot of complexities (read performance bottlenecks) > inside > > the engine. And all this for something that is not strictly needed. > However, > > there are always many opponents when I intend to remove the traditional > HUP > > behavior. > > > > > > Is there any other way to make the rsyslog process reread the conf file > > other than a restart? > > This is the only reason I use the traditional SIGHUP processing, I send > HUP > > after I alter the rsyslog.conf file. > > > No, but I always fail to see the big difference between typing > > $ kill -HUP ?cat /var/run/rsyslog.log? > > And > > $ /etc/rc.d/rsyslogd restart > > ;) > > Use the later one, and you do not need to use sighup. Even better, you can > now use HUP for a lightweight "close the files only" log rotation thing ;) > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Jul 14 18:06:38 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:06:38 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com> <87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Tuesday, July 14, 2009 5:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > Yes exactly, I have changed the script to do just that recently , I was just > wondering whether this is the reason why you get so much resistance when you > want to deprecate the traditional HUP processing... I think it is maily a "tradition" argument along the lines "but I am used to...". Still hard to beat if everybody complains ;) I don't see any technicaly reasons, especially as a restart-type restart IS a restart (nothing less), just carried out in the most complex way ;) Rainer > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > wrote: > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > next? > > > > > > Sorry if this is out of context. > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > files it's writing to so that they can be rotated away. > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > config > > > > file is so complex and can significantly alter the software by loading > > > > modules. > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it is > > > the > > > root cause for a lot of complexities (read performance bottlenecks) > > inside > > > the engine. And all this for something that is not strictly needed. > > However, > > > there are always many opponents when I intend to remove the traditional > > HUP > > > behavior. > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > other than a restart? > > > This is the only reason I use the traditional SIGHUP processing, I send > > HUP > > > after I alter the rsyslog.conf file. > > > > > > No, but I always fail to see the big difference between typing > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > And > > > > $ /etc/rc.d/rsyslogd restart > > > > ;) > > > > Use the later one, and you do not need to use sighup. Even better, you can > > now use HUP for a lightweight "close the files only" log rotation thing ;) > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Jul 14 18:17:59 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 14 Jul 2009 18:17:59 +0200 Subject: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? References: <9B6E2A8877C38245BFB15CC491A11DA706FB76@GRFEXC.intern.adiscon.com><4255c2570907131038x3b003dces1ab6a2b56973fb89@mail.gmail.com><4255c2570907131243l22781adbr9649e9da6fa7bbd1@mail.gmail.com><4255c2570907131424r5ce51f6arff7649d886fefb54@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB8A@GRFEXC.intern.adiscon.com><87a1ae540907140852s1e679cdo5c0b34eaf21efd7@mail.gmail.com><9B6E2A8877C38245BFB15CC491A11DA706FB95@GRFEXC.intern.adiscon.com><87a1ae540907140858s7c49df09i6dcf59bbdfbcd7d9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FB98@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FB99@GRFEXC.intern.adiscon.com> Oh, and I forgot to mention that recent builds loudly complain about config errors on stderr when you do a real restart bit silently need to dump these error messages to their respective bins (where experience tells nobody sees them...) when doing restart-type HUP. Maybe the result of this discussion is that I should really drop that misfeature in v5. Any violent opponents? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, July 14, 2009 6:07 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Tuesday, July 14, 2009 5:59 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's next? > > > > Yes exactly, I have changed the script to do just that recently , I was > just > > wondering whether this is the reason why you get so much resistance when > you > > want to deprecate the traditional HUP processing... > > I think it is maily a "tradition" argument along the lines "but I am used > to...". Still hard to beat if everybody complains ;) I don't see any > technicaly reasons, especially as a restart-type restart IS a restart > (nothing less), just carried out in the most complex way ;) > > Rainer > > > > > > > On Tue, Jul 14, 2009 at 11:55 AM, Rainer Gerhards > > wrote: > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > > > Sent: Tuesday, July 14, 2009 5:53 PM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] HUP restart or not - was: RE: rsyslog - what's > > > next? > > > > > > > > Sorry if this is out of context. > > > > > > > > > this matches the traditional use of HUP to get syslogd to release the > > > > > files it's writing to so that they can be rotated away. > > > > > > > > > > it doesn't re-read the config file due to the fact that the rsyslog > > > config > > > > > file is so complex and can significantly alter the software by > loading > > > > > modules. > > > > > > > > I am still tempted to remove the non-restart type of HUP. Actually, it > is > > > > the > > > > root cause for a lot of complexities (read performance bottlenecks) > > > inside > > > > the engine. And all this for something that is not strictly needed. > > > However, > > > > there are always many opponents when I intend to remove the traditional > > > HUP > > > > behavior. > > > > > > > > > > > > Is there any other way to make the rsyslog process reread the conf file > > > > other than a restart? > > > > This is the only reason I use the traditional SIGHUP processing, I send > > > HUP > > > > after I alter the rsyslog.conf file. > > > > > > > > > No, but I always fail to see the big difference between typing > > > > > > $ kill -HUP ?cat /var/run/rsyslog.log? > > > > > > And > > > > > > $ /etc/rc.d/rsyslogd restart > > > > > > ;) > > > > > > Use the later one, and you do not need to use sighup. Even better, you > can > > > now use HUP for a lightweight "close the files only" log rotation thing > ;) > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Jul 15 12:06:53 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Jul 2009 12:06:53 +0200 Subject: [rsyslog] HUP Processing - restart type will go away Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBAA@GRFEXC.intern.adiscon.com> Hi all, based on yesterday's discussion, and review of the (partly new) arguments, I have decided to go through the hassle of finally deprecating the restart-type HUP. I have changed the v3 doc so that it no longer explicitly recommends using HUP to apply config changes. I have added a compatibility doc to v4 which explains why HUP will go away and when. I invite you to read this document: http://www.rsyslog.com/doc-v4compatibility.html With the recent v4-devel (4.5.1, just released), I have also changed the default of $HUPisRestart to "off". In v5, I have added another compatibility document. Finally, I will remove the restart-type HUP in v5 soon, starting with the directive go away and later actually removing the (lots of) code that support it (and finally getting the benefits). Rainer From tbergfeld at hq.adiscon.com Wed Jul 15 14:30:23 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 15 Jul 2009 14:30:23 +0200 Subject: [rsyslog] rsyslog 4.5.1 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBB1@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 4.5.1, a member of the development branch. This version offers improved performannce. There were also some bugfixes like a fix for a potential segfault when zip-compressed syslog records were received. It further provides the ability for the TCP output action to "rebind" its send socket after sending n messages (actually, it re-opens the connection, the name is used because this is a concept very similiar to $ActionUDPRebindInterval). This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-167.phtml Changelog: http://www.rsyslog.com/Article388.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From alex at segv.de Wed Jul 15 11:20:30 2009 From: alex at segv.de (Alexander Elbs) Date: Wed, 15 Jul 2009 11:20:30 +0200 Subject: [rsyslog] Use precision timestamps Message-ID: <20090715092030.GC19320@segv.de> Hi, when using syslog(3) an application can send log messages via /dev/log to rsyslog and then to e.g. a file. If I enable high precision timestamps in rsyslog the log messages have a more precise timestamp. However there is some delay between the application generating a log message and rsyslog adding the timestamp. So why settle for less? :) (Well it is a distributed application, i.e. several processes and computers. So to debug interactions between the parts the correct ordering and timing is very important to me.) I wrote some code that opens /dev/log itself and sends the new format directly. This works very nice and I get the timestamps I want. Example code: -------------- #!/usr/bin/python import socket log = socket.socket( socket.AF_UNIX, socket.SOCK_DGRAM ) log.connect( "/dev/log" ) # VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG # PRI: 23 (==local7) * 8 + 4 (==warning) = 188 log.send( "<188>1 2009-07-15T09:45:12.463435Z mycomputer TEST_CLIENT 12345 SOME_PACKAGE This is a test message" ) log.close() -------------- However I have a few questions: - Is there some library code I could use that accepts high precision timestamps? Some kind of successor to syslog(3). - Is there a recommended way to detect if the syslog daemon will accept the new format? Currently this could mean checking if rsyslogd is listening on /dev/log or someone else. Otherwise the logging code needs to fall back to the old format that is understood by any syslog daemon (and use only second resolution). Mfg Alexander Elbs -- Alexander Elbs *** eMail alex at segv.de From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 14:40:42 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 14:40:42 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <200907161440.43049.Luis.Fernando.Munoz.Mejias@cern.ch> Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Thu Jul 16 15:31:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Jul 2009 15:31:17 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? Message-ID: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Just a quick note: there is no such thing as a whitespace inside a syslog tag (read rfc3164, 5424). The message is severely malformed and i do not see any way how rsyslog could guess the intention of the sender correctly. Have a look at the doc set, there is a full doc page with elaborate info on such cases. (i dont have it at hand right now) rainer ----- Urspr?ngliche Nachricht ----- Von: "Luis Fernando Mu?oz Mej?as" An: "rsyslog at lists.adiscon.com" Gesendet: 16.07.09 14:41 Betreff: [rsyslog] Syslogtags with whitespaces misparsed? Hello, world Some programs intruduce spaces as part of their syslog tags. For instance, a message from gconfd includes the tag "gconfd", and then the user and a PID, like this: 2009-07-16T00:58:45+02:00 gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 The bad news is that the "gconfd" is interpreted as the host name. I'd expect that line to be: 2009-07-16T00:58:45+02:00 HOSTNAME gconfd (foo-26702): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0 And thus, I'm losing the precious information of who produced the message. We have a funny mix of syslogd (most nodes) and rsyslog (3.X and 4.Y for central log services, mainly) here. I wonder if it's a configuration problem, an rsyslog bug (the first word of the syslogtag is displacing the host name) or an unavoidable problem of communicating rsyslog and syslog. Anyone knows what is going on? Thanks. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From Luis.Fernando.Munoz.Mejias at cern.ch Thu Jul 16 15:48:31 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?utf-8?q?Mu=C3=B1oz_Mej=C3=ADas?=) Date: Thu, 16 Jul 2009 15:48:31 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> Message-ID: <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > Just a quick note: there is no such thing as a whitespace inside a syslog > tag (read rfc3164, 5424). Thanks a lot, I'll tell people around here. Now I know who I must piss off. :) > The message is severely malformed and i do not see any way how rsyslog > could guess the intention of the sender correctly Indeed, rsyslog is not responsible for misbehaving senders, I just needed to clarify the source of my problem. Cheers. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 10:13:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 10:13:02 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Just let me say that I try really hard to make rsyslog guess right for even the most malformed mesage format. The real issue is that there have been no standards for many years, and so there is a lot of incompatible formats out there. In this case, however, I do really not see how I could handle that intelligently within the parser (except if we create per-host custom parsers, something on my list but not even begun to work on). I guess that the gconf folks will not want to change their format, because that, too could potentially also break a lot of things (log parsers!). A solution within rsyslog configuration could be to check the hostname and apply some special template (which then utilizes the property replacer to extract the "right" information) when the hostname is "gconf". That, of course, means that no host ever must be named "gconf", but I think you can enforce that ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Luis > Fernando Mu?oz Mej?as > Sent: Thursday, July 16, 2009 3:49 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > El Jueves, 16 de Julio de 2009 15:31, Rainer Gerhards escribi?: > > Just a quick note: there is no such thing as a whitespace > inside a syslog > > tag (read rfc3164, 5424). > > Thanks a lot, I'll tell people around here. Now I know who I must piss > off. :) > > > The message is severely malformed and i do not see any way > how rsyslog > > could guess the intention of the sender correctly > > Indeed, rsyslog is not responsible for misbehaving senders, I just > needed to clarify the source of my problem. > > Cheers. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From Luis.Fernando.Munoz.Mejias at cern.ch Fri Jul 17 15:05:38 2009 From: Luis.Fernando.Munoz.Mejias at cern.ch (Luis Fernando =?iso-8859-1?q?Mu=F1oz_Mej=EDas?=) Date: Fri, 17 Jul 2009 15:05:38 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> Message-ID: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Rainer, > In this case, however, I do really not see how I could handle that > intelligently within the parser My guesswork, as you call it on the document you reference is something like "the first word after the timestamp must be the host name, and from there to the first colon it's all syslogtag; if there's no colon I'll do whatever I want but crashing". But it's guessing, against RFCs, and I don't really think a syslog parser should play fortune-telling. > I guess that the gconf folks will not want to change their format, because > that, too could potentially also break a lot of things (log parsers!). I'll try to follow this up to gconf guys, so that they know that any log parsing of their messages is necessarily lacking *crucial* information. Anyways, gconf is the least important application to me, and I see some services around here showing the same symptoms. > A solution within rsyslog configuration In my scenario, the message has gone through several syslog relays and the correct host information is lost before it comes to my service, so there is no way to configure rsyslog to solve it. Another funny example is syslog's habit of saying "last message repeated N times". These messages don't have a colon or anything useful to delimit the application name. In this case, I receive *lots* of messages from a host called "last". Thanks for the clarifications. -- Luis Fernando Mu?oz Mej?as Luis.Fernando.Munoz.Mejias at cern.ch From rgerhards at hq.adiscon.com Fri Jul 17 15:42:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Jul 2009 15:42:10 +0200 Subject: [rsyslog] Syslogtags with whitespaces misparsed? References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com><200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch><9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBCC@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luis Fernando Mu?oz Mej?as > Sent: Friday, July 17, 2009 3:06 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Syslogtags with whitespaces misparsed? > > Rainer, > > > In this case, however, I do really not see how I could handle that > > intelligently within the parser > > My guesswork, as you call it on the document you reference is something > like "the first word after the timestamp must be the host name, Just to clarify so that everyone sees where the issues are ;) First off, how do you know if there is a hostname? If the message is lacking a hostname, the TAG will become it. Rsyslog assumes that a hostname is present, but knows it is a TAG instead if a character that is not valid inside the hostname. In this example "gconf" there is no way to know this is not a valid hostname. > and from > there to the first colon it's all syslogtag; Next actually not-sosubtle issue: But what do you do if there is no colon at the end of the syslog TAG? The rfcs demand none (because the header filed is SP-terminated) and also in practice there is not always a colon in it. So following this rule would break RFC compatibility and also probably break a lot more real-world cases than it fixes. > if there's no colon I'll do > whatever I want but crashing". But it's guessing, against RFCs, and I > don't really think a syslog parser should play fortune-telling. > > > I guess that the gconf folks will not want to change their format, because > > that, too could potentially also break a lot of things (log parsers!). > > I'll try to follow this up to gconf guys, so that they know that any log > parsing of their messages is necessarily lacking *crucial* > information. Anyways, gconf is the least important application to me, > and I see some services around here showing the same symptoms. > > > A solution within rsyslog configuration > > In my scenario, the message has gone through several syslog relays and > the correct host information is lost before it comes to my service, so > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". Brings up an interesting thought (for another time) - it may make sense to de-multiplex these "last message repeated N times", but that's quite some effort... Rainer > > Thanks for the clarifications. > -- > Luis Fernando Mu?oz Mej?as > Luis.Fernando.Munoz.Mejias at cern.ch > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Fri Jul 17 15:54:40 2009 From: aoz.syn at gmail.com (RB) Date: Fri, 17 Jul 2009 07:54:40 -0600 Subject: [rsyslog] Syslogtags with whitespaces misparsed? In-Reply-To: <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> References: <000a01ca061a$086d9ebb$100013ac@intern.adiscon.com> <200907161548.31737.Luis.Fernando.Munoz.Mejias@cern.ch> <9B6E2A8877C38245BFB15CC491A11DA706FBC6@GRFEXC.intern.adiscon.com> <200907171505.38813.Luis.Fernando.Munoz.Mejias@cern.ch> Message-ID: <4255c2570907170654i7f46cfb8la4440d4bf0838bd1@mail.gmail.com> On Fri, Jul 17, 2009 at 07:05, Luis Fernando Mu?oz Mej?as wrote: > there is no way to configure rsyslog to solve it. Another funny example > is syslog's habit of saying "last message repeated N times". These > messages don't have a colon or anything useful to delimit the > application name. In this case, I receive *lots* of messages from a host > called "last". It was these precise messages from *BSD that drove me to using the fromhost-ip property as opposed to whatever the sender "says" they are. It breaks down in the face of relay hosts, but when I get to the point of needing those I plan on relying on secondary parsing. From rgerhards at hq.adiscon.com Mon Jul 20 16:02:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Jul 2009 16:02:55 +0200 Subject: [rsyslog] rsyslog status Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FBE8@GRFEXC.intern.adiscon.com> Hi all, contrary to what I originally thought, I have worked "a bit" on the queue engine again. I couldn't resist to unleash at least some of the potential that the now gone-away restart-type HUP offered. I am probably through with a round of changes, and it looks rather well. The code has been greatly simplified, what I hope will also enhance its stability and make finding threading bugs much easier. All in all, the new engine looks much cleaner than the previous one. I also gained a bit more performance, but that is not so much of a difference. At the high end, you may say an increase in steady message rate by maybe one or two percent. The overhead of running multiple action queues has been reduced. The new code will probably be released tomorrow (will do some final checks), but it is available in the git master branch right now. I will now see if I begin to look into the set of performance monitoring tools or postpone that after my vacation and look at threaded inputs (at least for an initial pilot). I just thought I let you know. Rainer From tbergfeld at hq.adiscon.com Wed Jul 29 08:08:15 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 29 Jul 2009 08:08:15 +0200 Subject: [rsyslog] rsyslog 5.1.3 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC41@GRFEXC.intern.adiscon.com> Hi All, We have just released rsyslog 5.1.3, a member of the v5-development branch. This version offers a change in the architecture, queue now always has at least one worker thread if not running in direct mode. Previous versions could run without any active workers. This simplifies the code at a very small expense. See v5 compatibility note document for more in-depth discussion. There are also some bug fixes like a fix for a minor static memory leak while reading configuration. Furthermore there is the ability added to terminate input modules not via pthread_cancel but an alternate approach via pthread_kill. This is somewhat safer as we do not need to think about the cancel-safeness of all libraries we use. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-168.phtml Changelog: http://www.rsyslog.com/Article390.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Thu Jul 30 03:24:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 29 Jul 2009 18:24:12 -0700 (PDT) Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf Message-ID: I have the following in my config file :programname, isequal, "iaalog" ~ :programname, isequal, "plug-gw" ~ *.* @192.168.210.246:514;TraditionalFormat the first time a log entry comes along that has a program name of iaalog rsyslog crashes 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 21:09:05|history|1428800' 6183.244047218:40f00950: Message has legacy syslog format. 6183.244051641:40f00950: Called action, logging to builtin-file 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx 6183.244062030:40f00950: entering actionCalldoAction(), state: itx 6183.244066591:40f00950: file to log to: /var/log/messages 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy 6183.244086454:40f00950: action call returned 0 6183.244091070:40f00950: Filter: check for property 'programname' (value 'iaalog') isequal 'iaalog': TRUE 6183.244096297:40f00950: Called action, logging to builtin-discard 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: 0x1926600 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx 6183.244106591:40f00950: entering actionCalldoAction(), state: itx 6183.244109759:40f00950: 6183.244113010:40f00950: action call returned -2002 David Lang From toppiprc at gmail.com Thu Jul 30 04:02:37 2009 From: toppiprc at gmail.com (=?UTF-8?B?6JSh6LaF?=) Date: Thu, 30 Jul 2009 10:02:37 +0800 Subject: [rsyslog] how log non-ascii characters into mysql Message-ID: Hi, I want to record some non-ascii (e.g. UTF8) text using rsyslog. When the message is written to file, it is recorded as I logged, so it can be decoded and read well. But if the message is written to mysql, I can't decode it, even through I have changed charset configurations of the mysql database, tables and columns. What has happened before the messages were sent to mysql? Another problem, suppose I know the encodings of messages from all devices, and I want to transform them into some uniform one before saving. How can I achive it? Best regards Cai Chao From rgerhards at hq.adiscon.com Thu Jul 30 11:14:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 11:14:57 +0200 Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC52@GRFEXC.intern.adiscon.com> David, thanks for bringing this up. Actually, there were two problems in the code: number one was the segfault but number two was that no message was ever discarded. I have fixed both, but the second fix might cause some regression. The testbench (now extended to do a basic test of the discard functionality) runs through without problems, but obviously it does not yet cover all potential cases [and it will take more time until it really does...]. I will now leave for my vacation very shortly, so I cannot do any more in-depth testing and need to hope that this does not case too bad regressions. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, July 30, 2009 3:24 AM > To: rsyslog-users > Subject: [rsyslog] 5.1.3 problems with conditions in rsyslog.conf > > I have the following in my config file > > :programname, isequal, "iaalog" ~ > :programname, isequal, "plug-gw" ~ > *.* @192.168.210.246:514;TraditionalFormat > > the first time a log entry comes along that has a program name of iaalog > rsyslog crashes > > 6183.244020652:40f00950: msg parser: flags 30, from '192.168.210.6', msg > '<5>Jul 29 18:09:43 192.168.242.178 iaalog[112682]: AIB|leaderscu|2009/07/29 > 21:09:05|history|1428800' > 6183.244047218:40f00950: Message has legacy syslog format. > 6183.244051641:40f00950: Called action, logging to builtin-file > 6183.244055509:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244058853:40f00950: Action 0x1923b90 transitioned to state: itx > 6183.244062030:40f00950: entering actionCalldoAction(), state: itx > 6183.244066591:40f00950: file to log to: /var/log/messages > 6183.244070038:40f00950: doWrite, pData->pStrm 0x1923560, lenBuf 100 > 6183.244073820:40f00950: strm 0x1923560: file 5 flush, buflen 100 > 6183.244079133:40f00950: strm 0x1923560: file 5 write wrote 100 bytes > 6183.244083241:40f00950: Action 0x1923b90 transitioned to state: rdy > 6183.244086454:40f00950: action call returned 0 > 6183.244091070:40f00950: Filter: check for property 'programname' (value > 'iaalog') isequal 'iaalog': TRUE > 6183.244096297:40f00950: Called action, logging to builtin-discard > 6183.244100144:40f00950: submitBatch: i:0, batch size 1, to process 1, pMsg: > 0x1926600 > 6183.244103422:40f00950: Action 0x19241a0 transitioned to state: itx > 6183.244106591:40f00950: entering actionCalldoAction(), state: itx > 6183.244109759:40f00950: > 6183.244113010:40f00950: action call returned -2002 > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 12:03:33 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 12:03:33 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> Message-ID: <4A716FF5.8060608@redhat.com> Hi, just wanted to let you know that rsyslog-4.2.0 is now in rawhide and will be included in Fedora 12. Tomas On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > Hi Peter, > > that sounds quite interesting. I'll see that I get the v4-stable out > tomorrow. > > Thanks for following up... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >> Sent: Monday, June 22, 2009 3:53 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi folks, >> >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) >> >> At first we need v4 stable. If we have stable v4 we will push it into > rawhide >> and F11. This must be done ASAP, I see deadline in mid-July. >> >> Peter. >> >> >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>> Hi all, >>> >>> I guess RHEL6 will be based on Fedora 11 & 12. >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>> It's sure that RHEL6 includes v3 at least. >>> >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>> http://fedoraproject.org/wiki/Schedule >>> If v4 is stable and requested to be included in F12, v4 will be >>> shipped in F12. >>> >>> Best Rio. >>> >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>> That a good question and I, too, would like to know the answer. In >>>> any case, >>>> I hope it will not be v2. Giving that RHEL is a conservative >>>> distribution >>>> (and it has to be for the target base), I would assume that if they >>>> go for a >>>> newer version, it is v3. >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>> To: rsyslog at lists.adiscon.com >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>> >>>>> Hello, >>>>> is there any info about what rsyslog version will be available in >>>>> RHEL >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>> the >>>>> time "RHEL6 package freeze" will be in stable status? >>>>> >>>>> Thank you, >>>>> Jiri H. >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Jul 30 12:56:26 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 12:56:26 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Hi, thanks for sharing this excellent news! If I may try to ponder you for some more information... Does this increase the chance that a recent version of rsyslog will be part of the next version of RHEL? ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich > Sent: Thursday, July 30, 2009 12:04 PM > To: rsyslog-users > Subject: Re: [rsyslog] RSyslog version in RHEL 6 > > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas > > On 06/22/2009 02:24 PM, Rainer Gerhards wrote: > > Hi Peter, > > > > that sounds quite interesting. I'll see that I get the v4-stable out > > tomorrow. > > > > Thanks for following up... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec > >> Sent: Monday, June 22, 2009 3:53 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 > >> > >> Hi folks, > >> > >> rsyslog maintainer is on vacation at the moment so I'm answering this Q. :) > >> > >> At first we need v4 stable. If we have stable v4 we will push it into > > rawhide > >> and F11. This must be done ASAP, I see deadline in mid-July. > >> > >> Peter. > >> > >> > >> On Monday 22 June 2009 03:44:56 am ? ?? wrote: > >>> Hi all, > >>> > >>> I guess RHEL6 will be based on Fedora 11 & 12. > >>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. > >>> It's sure that RHEL6 includes v3 at least. > >>> > >>> Feature Freeze of Fedora 12 is planned on 2009-07-28. > >>> http://fedoraproject.org/wiki/Schedule > >>> If v4 is stable and requested to be included in F12, v4 will be > >>> shipped in F12. > >>> > >>> Best Rio. > >>> > >>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: > >>>> That a good question and I, too, would like to know the answer. In > >>>> any case, > >>>> I hope it will not be v2. Giving that RHEL is a conservative > >>>> distribution > >>>> (and it has to be for the target base), I would assume that if they > >>>> go for a > >>>> newer version, it is v3. > >>>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka > >>>>> Sent: Saturday, June 20, 2009 9:48 PM > >>>>> To: rsyslog at lists.adiscon.com > >>>>> Subject: [rsyslog] RSyslog version in RHEL 6 > >>>>> > >>>>> Hello, > >>>>> is there any info about what rsyslog version will be available in > >>>>> RHEL > >>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in > >>>>> the > >>>>> time "RHEL6 package freeze" will be in stable status? > >>>>> > >>>>> Thank you, > >>>>> Jiri H. > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Thu Jul 30 13:22:08 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Thu, 30 Jul 2009 13:22:08 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com><44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com><9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706FC59@GRFEXC.intern.adiscon.com> Message-ID: <4A718260.8090806@redhat.com> As far as I know, this version will be the one to go into RHEL 6.0. Tomas On 07/30/2009 12:56 PM, Rainer Gerhards wrote: > Hi, > > thanks for sharing this excellent news! If I may try to ponder you for some > more information... Does this increase the chance that a recent version of > rsyslog will be part of the next version of RHEL? ;) > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Tomas Heinrich >> Sent: Thursday, July 30, 2009 12:04 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >> >> Hi, >> >> just wanted to let you know that rsyslog-4.2.0 is now in rawhide and >> will be included in Fedora 12. >> >> Tomas >> >> On 06/22/2009 02:24 PM, Rainer Gerhards wrote: >>> Hi Peter, >>> >>> that sounds quite interesting. I'll see that I get the v4-stable out >>> tomorrow. >>> >>> Thanks for following up... >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Peter Vrabec >>>> Sent: Monday, June 22, 2009 3:53 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] RSyslog version in RHEL 6 >>>> >>>> Hi folks, >>>> >>>> rsyslog maintainer is on vacation at the moment so I'm answering this Q. > :) >>>> At first we need v4 stable. If we have stable v4 we will push it into >>> rawhide >>>> and F11. This must be done ASAP, I see deadline in mid-July. >>>> >>>> Peter. >>>> >>>> >>>> On Monday 22 June 2009 03:44:56 am ? ?? wrote: >>>>> Hi all, >>>>> >>>>> I guess RHEL6 will be based on Fedora 11 & 12. >>>>> Fedora 11 was shipped with rsyslog-3.21.11-1.fc11. >>>>> It's sure that RHEL6 includes v3 at least. >>>>> >>>>> Feature Freeze of Fedora 12 is planned on 2009-07-28. >>>>> http://fedoraproject.org/wiki/Schedule >>>>> If v4 is stable and requested to be included in F12, v4 will be >>>>> shipped in F12. >>>>> >>>>> Best Rio. >>>>> >>>>> On 2009/06/21, at 18:58, Rainer Gerhards wrote: >>>>>> That a good question and I, too, would like to know the answer. In >>>>>> any case, >>>>>> I hope it will not be v2. Giving that RHEL is a conservative >>>>>> distribution >>>>>> (and it has to be for the target base), I would assume that if they >>>>>> go for a >>>>>> newer version, it is v3. >>>>>> >>>>>> Rainer >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Jir? Hlinka >>>>>>> Sent: Saturday, June 20, 2009 9:48 PM >>>>>>> To: rsyslog at lists.adiscon.com >>>>>>> Subject: [rsyslog] RSyslog version in RHEL 6 >>>>>>> >>>>>>> Hello, >>>>>>> is there any info about what rsyslog version will be available in >>>>>>> RHEL >>>>>>> 6? I suppose it will be 3.x, but im not sure, maybe 4.x version in >>>>>>> the >>>>>>> time "RHEL6 package freeze" will be in stable status? >>>>>>> >>>>>>> Thank you, >>>>>>> Jiri H. >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Jul 30 13:29:11 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 13:29:11 +0200 Subject: [rsyslog] merge failures Message-ID: Hi, just noticed this in v4-stable grep "<<<<" * -R tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am Seems like a merge error. Haven't checked the other branches. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Jul 30 14:23:23 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 30 Jul 2009 14:23:23 +0200 Subject: [rsyslog] RSyslog version in RHEL 6 In-Reply-To: <4A716FF5.8060608@redhat.com> References: <9B6E2A8877C38245BFB15CC491A11DA706F9E5@GRFEXC.intern.adiscon.com> <44669843-6B67-444D-B34F-CCE2A4FE12B5@rio.st> <200906221553.10791.pvrabec@redhat.com> <9B6E2A8877C38245BFB15CC491A11DA706F9EF@GRFEXC.intern.adiscon.com> <4A716FF5.8060608@redhat.com> Message-ID: 2009/7/30 Tomas Heinrich : > Hi, > > just wanted to let you know that rsyslog-4.2.0 is now in rawhide and > will be included in Fedora 12. > > Tomas Hi Tomas, that's good news. FWIW the current version in Debian unstable/testing is also 4.2.0, so squeeze will also ship with at least 4.2.x and Ubuntu also made the switch to rsyslog. Their next release 9.10 will have rsyslog 4.2.0 as default syslog daemon. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Jul 30 15:43:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 30 Jul 2009 15:43:21 +0200 Subject: [rsyslog] merge failures References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FC5E@GRFEXC.intern.adiscon.com> Thanks Fixed in v4-stable now, will propagate to the other branches over time... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, July 30, 2009 1:29 PM > To: rsyslog-users > Subject: [rsyslog] merge failures > > Hi, > just noticed this in v4-stable > > grep "<<<<" * -R > tests/Makefile.am:<<<<<<< HEAD:tests/Makefile.am > > Seems like a merge error. > > Haven't checked the other branches. > > Cheers, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com