From david at lang.hm Tue Feb 1 07:03:50 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 31 Jan 2011 22:03:50 -0800 (PST) Subject: [rsyslog] notes on parser development with a couple of questions Message-ID: the rsyslog git repository is configured to not allow you to check in Makefile.am but you must have a file by that name in your plugins/pm* directory question: should this be forced manually with git add -f filename? or is there some other way that this should be done. the parser modules get hold of the string, _after_ it has been sanitized, so instead of tab characters you have #011 strings to work with. question: how can the parser module tell what options were used in receiving the log message? I am trying to write a snare log receiver module (which sends tab separated logs), how can I tell if tabs have been sanitized when I'm looking at the message? David Lang From rgerhards at hq.adiscon.com Tue Feb 1 07:25:40 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 07:25:40 +0100 Subject: [rsyslog] notes on parser development with a couple of questions References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB01@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, February 01, 2011 7:04 AM > To: rsyslog-users > Subject: [rsyslog] notes on parser development with a couple of > questions > > the rsyslog git repository is configured to not allow you to check in > Makefile.am but you must have a file by that name in your plugins/pm* > directory > > question: should this be forced manually with git add -f filename? or > is > there some other way that this should be done. > I guess you are mixing s/t up: Makefile and Makefile.in are not accepted, but Makefile.am is (at least in my git and the central repository). > > > the parser modules get hold of the string, _after_ it has been > sanitized, > so instead of tab characters you have #011 strings to work with. > > question: how can the parser module tell what options were used in > receiving the log message? I am trying to write a snare log receiver > module (which sends tab separated logs), how can I tell if tabs have > been > sanitized when I'm looking at the message? You can turn the sanitizion off, and also pri parsing. Here are the settings: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=38f556a2a3 9418fb5ddea3ffff41f55f448fe36c;hb=HEAD#l63 Return RS_RET_NO (actually anything else but OK), and that will not happen. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Tue Feb 1 08:08:01 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 31 Jan 2011 23:08:01 -0800 (PST) Subject: [rsyslog] notes on parser development with a couple of questions In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB01@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB01@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 1 Feb 2011, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, February 01, 2011 7:04 AM >> To: rsyslog-users >> Subject: [rsyslog] notes on parser development with a couple of >> questions >> >> the rsyslog git repository is configured to not allow you to check in >> Makefile.am but you must have a file by that name in your plugins/pm* >> directory >> >> question: should this be forced manually with git add -f filename? or >> is >> there some other way that this should be done. >> > > I guess you are mixing s/t up: Makefile and Makefile.in are not accepted, but > Makefile.am is (at least in my git and the central repository). hmm, you seem to be right. I thought I had tried that. oh well. the pmaixforwardedfrom submission is missing the Makefile.am file >> the parser modules get hold of the string, _after_ it has been >> sanitized, >> so instead of tab characters you have #011 strings to work with. >> >> question: how can the parser module tell what options were used in >> receiving the log message? I am trying to write a snare log receiver >> module (which sends tab separated logs), how can I tell if tabs have >> been >> sanitized when I'm looking at the message? > > You can turn the sanitizion off, and also pri parsing. Here are the settings: > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=38f556a2a3 > 9418fb5ddea3ffff41f55f448fe36c;hb=HEAD#l63 > > Return RS_RET_NO (actually anything else but OK), and that will not happen. I'm not wanting to force the user to turn the settings off, I just want to be able to detect if the user has turned them off or not. the easiest thing would be if I could get the message pre-sanitization and then after I finish mucking with it, allow it to fall through and get sanitized later, but I doubt that's possible. David Lang From david at lang.hm Tue Feb 1 08:23:52 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 31 Jan 2011 23:23:52 -0800 (PST) Subject: [rsyslog] snare parser Message-ID: this parser is designed to accept snare messages and turn them into messages that can be more easily digested by rsyslog. it looks for messages in either the raw snare format hostnametagotherstuff and change it to hostname tag otherstuff this then has the and timestamp set by the normal parser or in the 'syslog' format date hostname tagotherstuff and change it to date hostname tag otherstuff I currently have it support two tags, MSWinEventLog and LinuxKAudit, I am looking to find all the tags that snare agents generate to add them in. this is working for me, but I would like to see others test it as well. David Lang -------------- next part -------------- A non-text attachment was scrubbed... Name: pmsnare.patch Type: text/x-diff Size: 10963 bytes Desc: URL: From rgerhards at hq.adiscon.com Tue Feb 1 08:41:27 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 08:41:27 +0100 Subject: [rsyslog] notes on parser development with a couple of questions References: <9B6E2A8877C38245BFB15CC491A11DA71DDB01@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB02@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, February 01, 2011 8:08 AM > To: rsyslog-users > Subject: Re: [rsyslog] notes on parser development with a couple of > questions > > On Tue, 1 Feb 2011, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Tuesday, February 01, 2011 7:04 AM > >> To: rsyslog-users > >> Subject: [rsyslog] notes on parser development with a couple of > >> questions > >> > >> the rsyslog git repository is configured to not allow you to check > in > >> Makefile.am but you must have a file by that name in your > plugins/pm* > >> directory > >> > >> question: should this be forced manually with git add -f filename? > or > >> is > >> there some other way that this should be done. > >> > > > > I guess you are mixing s/t up: Makefile and Makefile.in are not > accepted, but > > Makefile.am is (at least in my git and the central repository). > > hmm, you seem to be right. I thought I had tried that. oh well. > > the pmaixforwardedfrom submission is missing the Makefile.am file Can you pls mail it ;) > >> the parser modules get hold of the string, _after_ it has been > >> sanitized, > >> so instead of tab characters you have #011 strings to work with. > >> > >> question: how can the parser module tell what options were used in > >> receiving the log message? I am trying to write a snare log receiver > >> module (which sends tab separated logs), how can I tell if tabs have > >> been > >> sanitized when I'm looking at the message? > > > > You can turn the sanitizion off, and also pri parsing. Here are the > settings: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=38f5 > 56a2a3 > > 9418fb5ddea3ffff41f55f448fe36c;hb=HEAD#l63 > > > > Return RS_RET_NO (actually anything else but OK), and that will not > happen. > > I'm not wanting to force the user to turn the settings off, I just want > to > be able to detect if the user has turned them off or not. > > the easiest thing would be if I could get the message pre-sanitization > and then after I finish mucking with it, allow it to fall through and > get > sanitized later, but I doubt that's possible. Ah, I did not explain well enough. This is NOT a user-setting! It just says if your module wants to have messages sanitized or not. So it tells the core about your module's needs. Most importantly, it will not affect any other module. But, for obvious reasons, when any module is run that requires sanitation, this is carried out by the core and can not be undone after that. So modul execution order is important. You want your module to load first, and instruct the core to not use sanitazion. Then you do whatever you want to the message. When the next parser (e.g. legacy syslog) is run, it has the sanitation flag set and the core will *then* call the sanitizer. The user setting is reflected only inside the sanitizer. So what the flag in you code says is that you are not "compatible" with sanitation, that is you do not want it for your needs. I guess this is exactly what you need. I hope this clarifies. Rainer From david at lang.hm Tue Feb 1 09:17:01 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 00:17:01 -0800 (PST) Subject: [rsyslog] makefile.am for pmaixforwardedfrom Message-ID: this plus the patch I sent over the weekend should be everything needed to make this work. David Lang -------------- next part -------------- pkglib_LTLIBRARIES = pmaixforwardedfrom.la pmaixforwardedfrom_la_SOURCES = pmaixforwardedfrom.c pmaixforwardedfrom_la_CPPFLAGS = $(RSRT_CFLAGS) $(PTHREADS_CFLAGS) -I ../../tools pmaixforwardedfrom_la_LDFLAGS = -module -avoid-version pmaixforwardedfrom_la_LIBADD = EXTRA_DIST = From david at lang.hm Tue Feb 1 09:21:15 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 00:21:15 -0800 (PST) Subject: [rsyslog] notes on parser development with a couple of questions In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB02@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB01@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB02@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 1 Feb 2011, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> On Tue, 1 Feb 2011, Rainer Gerhards wrote: >> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>> Sent: Tuesday, February 01, 2011 7:04 AM >>>> To: rsyslog-users >>>> Subject: [rsyslog] notes on parser development with a couple of >>>> questions >>>> >>>> the rsyslog git repository is configured to not allow you to check >> in >>>> Makefile.am but you must have a file by that name in your >> plugins/pm* >>>> directory >>>> >>>> question: should this be forced manually with git add -f filename? >> or >>>> is >>>> there some other way that this should be done. >>>> >>> >>> I guess you are mixing s/t up: Makefile and Makefile.in are not >> accepted, but >>> Makefile.am is (at least in my git and the central repository). >> >> hmm, you seem to be right. I thought I had tried that. oh well. >> >> the pmaixforwardedfrom submission is missing the Makefile.am file > > Can you pls mail it ;) sent >>>> the parser modules get hold of the string, _after_ it has been >>>> sanitized, >>>> so instead of tab characters you have #011 strings to work with. >>>> >>>> question: how can the parser module tell what options were used in >>>> receiving the log message? I am trying to write a snare log receiver >>>> module (which sends tab separated logs), how can I tell if tabs have >>>> been >>>> sanitized when I'm looking at the message? >>> >>> You can turn the sanitizion off, and also pri parsing. Here are the >> settings: >>> >>> >> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/pmrfc3164.c;h=38f5 >> 56a2a3 >>> 9418fb5ddea3ffff41f55f448fe36c;hb=HEAD#l63 >>> >>> Return RS_RET_NO (actually anything else but OK), and that will not >> happen. >> >> I'm not wanting to force the user to turn the settings off, I just want >> to >> be able to detect if the user has turned them off or not. >> >> the easiest thing would be if I could get the message pre-sanitization >> and then after I finish mucking with it, allow it to fall through and >> get >> sanitized later, but I doubt that's possible. > > Ah, I did not explain well enough. This is NOT a user-setting! It just says > if your module wants to have messages sanitized or not. So it tells the core > about your module's needs. Most importantly, it will not affect any other > module. But, for obvious reasons, when any module is run that requires > sanitation, this is carried out by the core and can not be undone after that. > So modul execution order is important. > > You want your module to load first, and instruct the core to not use > sanitazion. Then you do whatever you want to the message. When the next > parser (e.g. legacy syslog) is run, it has the sanitation flag set and the > core will *then* call the sanitizer. > > The user setting is reflected only inside the sanitizer. > > So what the flag in you code says is that you are not "compatible" with > sanitation, that is you do not want it for your needs. I guess this is > exactly what you need. > > I hope this clarifies. this makes sense. I'll have to think about this for a bit. I hate the thought of building something that won't work depending on module loading order (or evaluation order), especially something subtle like this. the snare module I sent should work with the sanitization, If I have some way to detect that the sanitization has not taken place there is one #define that would need to be changed. will the config validation step detect if a parser that is 'incompatible' with sanitization is loaded after a module that uses sanitization? If that is detected it would make me much more willing to do things this way. David Lang From rgerhards at hq.adiscon.com Tue Feb 1 11:22:04 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 11:22:04 +0100 Subject: [rsyslog] makefile.am for pmaixforwardedfrom References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB05@GRFEXC.intern.adiscon.com> FYI: I have just merged that patch into v5-devel. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, February 01, 2011 9:17 AM > To: rsyslog-users > Subject: [rsyslog] makefile.am for pmaixforwardedfrom > > this plus the patch I sent over the weekend should be everything needed > to > make this work. > > David Lang From rgerhards at hq.adiscon.com Tue Feb 1 11:27:08 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 11:27:08 +0100 Subject: [rsyslog] Feedback on ommysql requested References: <20101202155126.GZ19162@aart.is.rice.edu> <4D0BE506.8090000@hostdime.com><9B6E2A8877C38245BFB15CC491A11DA71DD93D@GRFEXC.intern.adiscon.com> <4D45D678.7090303@hostdime.com><9B6E2A8877C38245BFB15CC491A11DA71DDAFC@GRFEXC.intern.adiscon.com> <4D472A97.4010300@hostdime.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB07@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ariel P. > Sent: Monday, January 31, 2011 10:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback on ommysql requested > > The patch for doc/ommysql.html has been submitted via the bug tracker. many thanks! I just merged it. Rainer > > Ariel P. > Server Security Analyst > HostDime.com, Inc. > > On 2011-01-31 11:00, Rainer Gerhards wrote: > > sorry, I had forgotten the merge after my vacation. Now merged. It > would be > > good if you could also provide a patch for the doc file. > > > > Thanks again! > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Ariel P. > >> Sent: Sunday, January 30, 2011 10:22 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Feedback on ommysql requested > >> > >> Hi all, > >> > >> Any word on the merging of this patch into the main code base, given > >> that no further opposing arguments were given in the mailing list? > >> > >> Ariel P. > >> Server Security Analyst > >> HostDime.com, Inc. > >> > >> > >> On 2010-12-22 13:07, Rainer Gerhards wrote: > >>> Hi all, > >>> > >>> thanks to Ariel for the hard work. I, too, think that all concerns > >> are > >>> addressed. I intend to merge the patch when I am back in January if > I > >> don't > >>> here any good argument why I should not ;) > >>> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Ariel P. > >>>> Sent: Friday, December 17, 2010 11:33 PM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: Re: [rsyslog] Feedback on ommysql requested > >>>> > >>>> I have created a new version of my patch, which creates two > >>>> configuration settings for ommysql. > >>>> The first setting sets the path for the 'my.cnf' file. If this > >> setting > >>>> is not specified in the configuration file, the mysql library is > not > >>>> informed of a file path (current behavior) > >>>> The second setting is used to change the section of the 'my.cnf' > >> file > >>>> to > >>>> use, which defaults to "client" if not specified in the > >> configuration > >>>> file (this is mysql standard behavior). > >>>> > >>>> Please see [ http://bugzilla.adiscon.com/show_bug.cgi?id=213#c3 ] > >> for > >>>> details. > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Tue Feb 1 15:23:34 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Tue, 1 Feb 2011 15:23:34 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 6.1.3, the new v6-devel. This release brings a number of enhancements. First of all, it provides exciting new functionality in the form or an experimental MongoDB output plugin and various message parsers. The latter greatly increases rsyslog's ability to common malformed syslog message formats. Also, ommysql now allows a much more fine-grained control over MySQL settings. Then, a number of performance enhancements are made available: both imtcp and imptcp have an optimized code based and imptcp has been enhanced to be able to run multiple input threads concurrently (the same is planned for imtcp in the not so distant future). Also, imdup has now the capability to run at real-time priority, which greatly reduces the chance of message loss. Finally, there are a couple of bug fixes inside this release. We are especially proud that some of the important new features were contributed code: David Lang contributed the message parsers, Victor Pereira the experimental MongoDB module, Dra?en Ka?ar the imudp realtime-support and Ariel P. the enhancements for ommysql. Many thanks to all of you! Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-6-1-3-v6-devel/ Download: http://www.rsyslog.com/rsyslog-6-1-3-v6-devel/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From david at lang.hm Tue Feb 1 18:36:05 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 09:36:05 -0800 (PST) Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> Message-ID: also since 6.1.2 there has been a bunch of work on the imfile module, a number of bug fixes, but also it gained the ability to deal with some multi-line logs in files, combining the lines into one log entry before processing the log message. Rainer did the heavy debugging work and I contributed the multi-line capibility I think this was announced as part of a 5.x release, but in the 6.x tree this is the first release with these changes. David Lang On Tue, 1 Feb 2011, Tom Bergfeld wrote: > Hi all, > > We have just released rsyslog 6.1.3, the new v6-devel. > This release brings a number of enhancements. > First of all, it provides exciting new functionality in the form or an > experimental MongoDB output plugin and various message parsers. The latter > greatly increases rsyslog's ability to common malformed syslog message > formats. Also, ommysql now allows a much more fine-grained control over MySQL > settings. > Then, a number of performance enhancements are made available: both imtcp and > imptcp have an optimized code based and imptcp has been enhanced to be able > to run multiple input threads concurrently (the same is planned for imtcp in > the not so distant future). Also, imdup has now the capability to run at > real-time priority, which greatly reduces the chance of message loss. > Finally, there are a couple of bug fixes inside this release. > > We are especially proud that some of the important new features were > contributed code: David Lang contributed the message parsers, Victor Pereira > the experimental MongoDB module, Dra?en Ka?ar the imudp realtime-support and > Ariel P. the enhancements for ommysql. Many thanks to all of you! > > Please see the ChangeLog for more details. > > ChangeLog: > > http://www.rsyslog.com/changelog-for-6-1-3-v6-devel/ > > Download: > > http://www.rsyslog.com/rsyslog-6-1-3-v6-devel/ > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > > Support > > ======= > > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Feb 1 18:53:27 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 18:53:27 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> Hi David, thanks for putting this straight. It was actually my fault not to tell Tom this for v6. I think the release cycle was a bit too long, too many new features. I also think I need to concentrate on v6 now for new features -- going back and forth between v5 and v6 for development releases is sometimes quite confusing, at least to me ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, February 01, 2011 6:36 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > > also since 6.1.2 there has been a bunch of work on the imfile module, a > number of bug fixes, but also it gained the ability to deal with some > multi-line logs in files, combining the lines into one log entry before > processing the log message. > > Rainer did the heavy debugging work and I contributed the multi-line > capibility > > I think this was announced as part of a 5.x release, but in the 6.x > tree > this is the first release with these changes. > > David Lang > > > On Tue, 1 Feb 2011, Tom Bergfeld wrote: > > > Hi all, > > > > We have just released rsyslog 6.1.3, the new v6-devel. > > This release brings a number of enhancements. > > First of all, it provides exciting new functionality in the form or > an > > experimental MongoDB output plugin and various message parsers. The > latter > > greatly increases rsyslog's ability to common malformed syslog > message > > formats. Also, ommysql now allows a much more fine-grained control > over MySQL > > settings. > > Then, a number of performance enhancements are made available: both > imtcp and > > imptcp have an optimized code based and imptcp has been enhanced to > be able > > to run multiple input threads concurrently (the same is planned for > imtcp in > > the not so distant future). Also, imdup has now the capability to run > at > > real-time priority, which greatly reduces the chance of message loss. > > Finally, there are a couple of bug fixes inside this release. > > > > We are especially proud that some of the important new features were > > contributed code: David Lang contributed the message parsers, Victor > Pereira > > the experimental MongoDB module, Dra?en Ka?ar the imudp realtime- > support and > > Ariel P. the enhancements for ommysql. Many thanks to all of you! > > > > Please see the ChangeLog for more details. > > > > ChangeLog: > > > > http://www.rsyslog.com/changelog-for-6-1-3-v6-devel/ > > > > Download: > > > > http://www.rsyslog.com/rsyslog-6-1-3-v6-devel/ > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > > > Support > > > > ======= > > > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From madduck at madduck.net Tue Feb 1 19:40:48 2011 From: madduck at madduck.net (martin f krafft) Date: Tue, 1 Feb 2011 19:40:48 +0100 Subject: [rsyslog] Maintaining a symlink to a templated logfile Message-ID: <20110201184047.GA23189@albatross.oerlikon.madduck.net> Dear rsysloggers, I am experimenting with replacing the standard Unix logfile rotation scheme with something more useful: $template authlog,"/var/log/auth.log_%$NOW%" auth,authpriv.* ?authlog One problem with this approach is that standard tools now cannot find /var/log/auth.log anymore. The canonical way to solve this could be to maintain a symlink from /var/log/auth.log to the current /var/log/auth.log_%$NOW%. Can rsyslog maintain this symlink somehow? The only thing I found was the Shell Execute action, which would lead to a gross hack. Thanks, -- martin | http://madduck.net/ | http://two.sentenc.es/ http://www.transnationalrepublic.org/ spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: From madduck at madduck.net Tue Feb 1 19:33:23 2011 From: madduck at madduck.net (martin f krafft) Date: Tue, 1 Feb 2011 19:33:23 +0100 Subject: [rsyslog] Appending a timestamp to all log files Message-ID: <20110201183322.GA10663@albatross.oerlikon.madduck.net> Dear rsysloggers, I found that using a construct like $template authlog,"/var/log/auth.log_%$NOW%" auth,authpriv.* ?authlog I can write daily log files quite easily. Now I want to set this default on all my machines, but I would really love to be able to do this with a single file e.g. /etc/rsyslog.d/local-timestamp-suffix.conf, rather than changing the existing default rules in /etc/rsyslog.conf. Is this something that's already possible (and I just simply haven't found it)? I wrote down some ideas at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=536432 Is there anything in there that you would consider? Thanks! PS: please keep the bug report on CC. -- martin | http://madduck.net/ | http://two.sentenc.es/ "montag, why do you burn books?" "it's a job like any other, pay is good and there is a lot of variety." -- ray bradbury (f451) spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: From david at lang.hm Tue Feb 1 19:54:54 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 10:54:54 -0800 (PST) Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> Message-ID: I was going to ask about that. I didn't realize that v6 was really going in new features, I thought it was just a testbed for the new config parser system. so two questions 1. should we be basing all new work on v6? 2. what happened about the config stuff? David Lang On Tue, 1 Feb 2011, Rainer Gerhards wrote: > Hi David, > > thanks for putting this straight. It was actually my fault not to tell Tom > this for v6. I think the release cycle was a bit too long, too many new > features. I also think I need to concentrate on v6 now for new features -- > going back and forth between v5 and v6 for development releases is sometimes > quite confusing, at least to me ;) > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Tuesday, February 01, 2011 6:36 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released >> >> also since 6.1.2 there has been a bunch of work on the imfile module, a >> number of bug fixes, but also it gained the ability to deal with some >> multi-line logs in files, combining the lines into one log entry before >> processing the log message. >> >> Rainer did the heavy debugging work and I contributed the multi-line >> capibility >> >> I think this was announced as part of a 5.x release, but in the 6.x >> tree >> this is the first release with these changes. >> >> David Lang >> >> >> On Tue, 1 Feb 2011, Tom Bergfeld wrote: >> >>> Hi all, >>> >>> We have just released rsyslog 6.1.3, the new v6-devel. >>> This release brings a number of enhancements. >>> First of all, it provides exciting new functionality in the form or >> an >>> experimental MongoDB output plugin and various message parsers. The >> latter >>> greatly increases rsyslog's ability to common malformed syslog >> message >>> formats. Also, ommysql now allows a much more fine-grained control >> over MySQL >>> settings. >>> Then, a number of performance enhancements are made available: both >> imtcp and >>> imptcp have an optimized code based and imptcp has been enhanced to >> be able >>> to run multiple input threads concurrently (the same is planned for >> imtcp in >>> the not so distant future). Also, imdup has now the capability to run >> at >>> real-time priority, which greatly reduces the chance of message loss. >>> Finally, there are a couple of bug fixes inside this release. >>> >>> We are especially proud that some of the important new features were >>> contributed code: David Lang contributed the message parsers, Victor >> Pereira >>> the experimental MongoDB module, Dra?en Ka?ar the imudp realtime- >> support and >>> Ariel P. the enhancements for ommysql. Many thanks to all of you! >>> >>> Please see the ChangeLog for more details. >>> >>> ChangeLog: >>> >>> http://www.rsyslog.com/changelog-for-6-1-3-v6-devel/ >>> >>> Download: >>> >>> http://www.rsyslog.com/rsyslog-6-1-3-v6-devel/ >>> >>> As always, feedback is appreciated. >>> >>> Best regards, >>> Tom Bergfeld >>> -- >>> >>> Support >>> >>> ======= >>> >>> >>> Improving rsyslog is costly, but you can help! We are looking for >>> organizations that find rsyslog useful and wish to contribute back. >> You can >>> contribute by reporting bugs, improve the software, or donate money >> or >>> equipment. >>> >>> >>> Commercial support contracts for rsyslog are available, and they help >> finance >>> continued maintenance. Adiscon GmbH, a privately held German >> company, is >>> currently funding rsyslog development. We are always looking for >> interesting >>> development projects. For details on how to help, please see >>> http://www.rsyslog.com/doc-how2help.html. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Feb 1 19:57:52 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 10:57:52 -0800 (PST) Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: <20110201183322.GA10663@albatross.oerlikon.madduck.net> References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> Message-ID: rsyslog already has the include directive to include files (or all files in a directory) does this work for you? if not, what does it lack? David Lang -------------- next part -------------- Dear rsysloggers, I found that using a construct like $template authlog,"/var/log/auth.log_%$NOW%" auth,authpriv.* ?authlog I can write daily log files quite easily. Now I want to set this default on all my machines, but I would really love to be able to do this with a single file e.g. /etc/rsyslog.d/local-timestamp-suffix.conf, rather than changing the existing default rules in /etc/rsyslog.conf. Is this something that's already possible (and I just simply haven't found it)? I wrote down some ideas at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=536432 Is there anything in there that you would consider? Thanks! PS: please keep the bug report on CC. -- martin | http://madduck.net/ | http://two.sentenc.es/ "montag, why do you burn books?" "it's a job like any other, pay is good and there is a lot of variety." -- ray bradbury (f451) spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Feb 1 20:06:37 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 11:06:37 -0800 (PST) Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: <20110201190100.GA31236@albatross.oerlikon.madduck.net> References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201190100.GA31236@albatross.oerlikon.madduck.net> Message-ID: On Tue, 1 Feb 2011, martin f krafft wrote: > also sprach david at lang.hm [2011.02.01.1957 +0100]: >> rsyslog already has the include directive to include files (or all >> files in a directory) >> >> does this work for you? if not, what does it lack? > > I am aware of the include directive. It has nothing to do with what > I am trying to do. > > I am trying to tell rsyslog: "hey, all files that you write, append > the following suffix to their names!", or "hey, all files that you > write, run their names through this filter and use the output as > actual filename!". > > I want this because it's a cleaner way to replace all hard-coded > filenames with templated filenames, than to change every instance of > a hard-coded filename. > > Am I making myself clearer now? a little bit, but rsyslog does not have any hard-coded filenames. your distro may have something like this, but Ubuntu already does all output directives through include files, so why not just replace those include files? the problem with just running all filenames through a filter is that you also want to change normal, fixed filenames to dynamic filenames. this requires that the filename be evaluated for each log message as opposed to just at open time. There is a significant performance hit to doing this, one that people who don't use synamid filenames will not want to pay. David Lang From david at lang.hm Tue Feb 1 20:07:06 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 11:07:06 -0800 (PST) Subject: [rsyslog] Maintaining a symlink to a templated logfile In-Reply-To: <20110201184047.GA23189@albatross.oerlikon.madduck.net> References: <20110201184047.GA23189@albatross.oerlikon.madduck.net> Message-ID: I think the hack you are talking about is the only way to do this. David Lang -------------- next part -------------- Dear rsysloggers, I am experimenting with replacing the standard Unix logfile rotation scheme with something more useful: $template authlog,"/var/log/auth.log_%$NOW%" auth,authpriv.* ?authlog One problem with this approach is that standard tools now cannot find /var/log/auth.log anymore. The canonical way to solve this could be to maintain a symlink from /var/log/auth.log to the current /var/log/auth.log_%$NOW%. Can rsyslog maintain this symlink somehow? The only thing I found was the Shell Execute action, which would lead to a gross hack. Thanks, -- martin | http://madduck.net/ | http://two.sentenc.es/ http://www.transnationalrepublic.org/ spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From madduck at madduck.net Tue Feb 1 20:35:54 2011 From: madduck at madduck.net (martin f krafft) Date: Tue, 1 Feb 2011 20:35:54 +0100 Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> Message-ID: <20110201193554.GA8881@albatross.oerlikon.madduck.net> also sprach david at lang.hm [2011.02.01.1957 +0100]: > rsyslog already has the include directive to include files (or all > files in a directory) > > does this work for you? if not, what does it lack? Hello David, thanks for your time in writing back. Unfortunately, I think I was unclear about what I am trying to do. I am aware of the include directive. It has nothing to do with my goals. I am trying to tell rsyslog: "hey, all files that you write, append the following suffix to their names!", or "hey, all files that you write, run their names through this filter and use the output as actual filename!". I want this because it's a cleaner way to replace all hard-coded filenames with templated filenames, than to change every instance of a hard-coded filename. Am I making myself clearer now? -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net at madduck logik ist analsadismus: gedanken werden gewaltsam durch einen engen gang gepre?t. -- frei nach lacan spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: From madduck at madduck.net Tue Feb 1 20:53:06 2011 From: madduck at madduck.net (martin f krafft) Date: Tue, 1 Feb 2011 20:53:06 +0100 Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201190100.GA31236@albatross.oerlikon.madduck.net> Message-ID: <20110201195306.GA15260@albatross.oerlikon.madduck.net> also sprach david at lang.hm [2011.02.01.2006 +0100]: > your distro may have something like this, but Ubuntu already does all > output directives through include files, so why not just replace > those include files? Because it means I need to maintain them, rather than leave this job up to the package maintainer and use the defaults as much as possible. Debian (Ubuntu) has conffiles and all, making sure that my changes never get overwritten; but it's often worth to go a step further and make things even more parametrisable. The goal of every sysadmin should be to minimise the deviation from the distro default. > the problem with just running all filenames through a filter is that > you also want to change normal, fixed filenames to dynamic filenames. > this requires that the filename be evaluated for each log message as > opposed to just at open time. There is a significant performance hit > to doing this, one that people who don't use synamid filenames will > not want to pay. This is true. At the moment, you have a much lesser performance hit due to template evaluation. I would prefer an internal mangler to an external filter, so that the performance hit would be negligible over template evaluation. -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net at madduck when everything is coming your way, you're in the wrong lane. spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: From rgerhards at hq.adiscon.com Tue Feb 1 21:55:37 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 21:55:37 +0100 Subject: [rsyslog] Appending a timestamp to all log files References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201193554.GA8881@albatross.oerlikon.madduck.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB17@GRFEXC.intern.adiscon.com> This sounds useful, at least for some use cases. However, this most probably requires considerable changes to the config engine. And as this engine is scheduled to be replaced, I don't like to touch it too much. However, there may be one solution. We could declare something like "if the file name is defined as a template name, then make it a dynafile based on that template". Then, you would need to define templates e.g $template /var/log/messages,"/var/log/messages_%$NoW%" And those would automatically be converted. That seems to require relatively few changes. Would that help? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of martin f krafft > Sent: Tuesday, February 01, 2011 8:36 PM > To: david at lang.hm > Cc: rsyslog-users; 536432 at bugs.debian.org > Subject: Re: [rsyslog] Appending a timestamp to all log files > > also sprach david at lang.hm [2011.02.01.1957 +0100]: > > rsyslog already has the include directive to include files (or all > > files in a directory) > > > > does this work for you? if not, what does it lack? > > Hello David, > > thanks for your time in writing back. Unfortunately, I think I was > unclear about what I am trying to do. > > I am aware of the include directive. It has nothing to do with my > goals. > > I am trying to tell rsyslog: "hey, all files that you write, append the > following suffix to their names!", or "hey, all files that you write, > run their names through this filter and use the output as actual > filename!". > > I want this because it's a cleaner way to replace all hard-coded > filenames with templated filenames, than to change every instance of a > hard-coded filename. > > Am I making myself clearer now? > > -- > martin; (greetings from the heart of the sun.) > \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net at madduck > > logik ist analsadismus: gedanken werden gewaltsam durch einen engen > gang gepre?t. > -- frei nach lacan > > spamtraps: madduck.bogus at madduck.net From rgerhards at hq.adiscon.com Tue Feb 1 22:03:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Feb 2011 22:03:51 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB18@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, February 01, 2011 7:55 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > > I was going to ask about that. I didn't realize that v6 was really > going > in new features, I thought it was just a testbed for the new config > parser > system. It's both. I moved up to v6 because the config change is probably quite intrusive. But it *will* be the new main line, and so new features go into it (I usually branch within a release for purely experimental stuff). > > so two questions > > 1. should we be basing all new work on v6? > Ist probably a good idea, at least for major things. > 2. what happened about the config stuff? Well, our minimal consensus (action scoping) is in v6, but I did not get much feedback. I have begun to think that the minimal consensus is too minimal. I'd like to create at least an experimental completely new config parser with a cleaner and more powerful syntax. But there is quite some work in front of that. Unforutnately, the time window for changes to the config system closed last year when it became so unclear which direction to take. I hope it re-opens in spring, based on the other things that are going on. Maybe JSON would be an alternative -- liblognorm will probably need to support JSONish config for several reasons and if that works out, we could borrow there... Rainer > > David Lang > > On Tue, 1 Feb 2011, Rainer Gerhards wrote: > > > Hi David, > > > > thanks for putting this straight. It was actually my fault not to > tell Tom > > this for v6. I think the release cycle was a bit too long, too many > new > > features. I also think I need to concentrate on v6 now for new > features -- > > going back and forth between v5 and v6 for development releases is > sometimes > > quite confusing, at least to me ;) > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Tuesday, February 01, 2011 6:36 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > >> > >> also since 6.1.2 there has been a bunch of work on the imfile > module, a > >> number of bug fixes, but also it gained the ability to deal with > some > >> multi-line logs in files, combining the lines into one log entry > before > >> processing the log message. > >> > >> Rainer did the heavy debugging work and I contributed the multi-line > >> capibility > >> > >> I think this was announced as part of a 5.x release, but in the 6.x > >> tree > >> this is the first release with these changes. > >> > >> David Lang > >> > >> > >> On Tue, 1 Feb 2011, Tom Bergfeld wrote: > >> > >>> Hi all, > >>> > >>> We have just released rsyslog 6.1.3, the new v6-devel. > >>> This release brings a number of enhancements. > >>> First of all, it provides exciting new functionality in the form or > >> an > >>> experimental MongoDB output plugin and various message parsers. The > >> latter > >>> greatly increases rsyslog's ability to common malformed syslog > >> message > >>> formats. Also, ommysql now allows a much more fine-grained control > >> over MySQL > >>> settings. > >>> Then, a number of performance enhancements are made available: both > >> imtcp and > >>> imptcp have an optimized code based and imptcp has been enhanced to > >> be able > >>> to run multiple input threads concurrently (the same is planned for > >> imtcp in > >>> the not so distant future). Also, imdup has now the capability to > run > >> at > >>> real-time priority, which greatly reduces the chance of message > loss. > >>> Finally, there are a couple of bug fixes inside this release. > >>> > >>> We are especially proud that some of the important new features > were > >>> contributed code: David Lang contributed the message parsers, > Victor > >> Pereira > >>> the experimental MongoDB module, Dra?en Ka?ar the imudp realtime- > >> support and > >>> Ariel P. the enhancements for ommysql. Many thanks to all of you! > >>> > >>> Please see the ChangeLog for more details. > >>> > >>> ChangeLog: > >>> > >>> http://www.rsyslog.com/changelog-for-6-1-3-v6-devel/ > >>> > >>> Download: > >>> > >>> http://www.rsyslog.com/rsyslog-6-1-3-v6-devel/ > >>> > >>> As always, feedback is appreciated. > >>> > >>> Best regards, > >>> Tom Bergfeld > >>> -- > >>> > >>> Support > >>> > >>> ======= > >>> > >>> > >>> Improving rsyslog is costly, but you can help! We are looking for > >>> organizations that find rsyslog useful and wish to contribute back. > >> You can > >>> contribute by reporting bugs, improve the software, or donate money > >> or > >>> equipment. > >>> > >>> > >>> Commercial support contracts for rsyslog are available, and they > help > >> finance > >>> continued maintenance. Adiscon GmbH, a privately held German > >> company, is > >>> currently funding rsyslog development. We are always looking for > >> interesting > >>> development projects. For details on how to help, please see > >>> http://www.rsyslog.com/doc-how2help.html. > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Tue Feb 1 22:08:53 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 13:08:53 -0800 (PST) Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: <20110201194314.GB8881@albatross.oerlikon.madduck.net> References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201190100.GA31236@albatross.oerlikon.madduck.net> <20110201194314.GB8881@albatross.oerlikon.madduck.net> Message-ID: On Tue, 1 Feb 2011, martin f krafft wrote: > also sprach david at lang.hm [2011.02.01.2006 +0100]: >> your distro may have something like this, but Ubuntu already does all >> output directives through include files, so why not just replace >> those include files? > > Because it means I need to maintain them, rather than leave this job > up to the package maintainer and use the defaults as much as > possible. Debian (Ubuntu) has conffiles and all, making sure that my > changes never get overwritten; but it's often worth to go a step > further and make things even more parametrisable. The goal of every > sysadmin should be to minimise the deviation from the distro > default. > >> the problem with just running all filenames through a filter is that >> you also want to change normal, fixed filenames to dynamic filenames. >> this requires that the filename be evaluated for each log message as >> opposed to just at open time. There is a significant performance hit >> to doing this, one that people who don't use synamid filenames will >> not want to pay. > > This is true. At the moment, you have a much lesser performance hit > due to template evaluation. I would prefer an internal mangler to an > external filter, so that the performance hit would be negligible > over template evaluation. actually, for the most part you don't have template evaluation when writing logs either. the four most common log formats have hard-coded templates available (which resulted in a noticable speedup when they were implemented), and the file is only evaluated at startup/HUP time and from that point on it's just a write to an open file handle, it doesn't even look at the log file name. David Lang From david at lang.hm Tue Feb 1 22:15:43 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Feb 2011 13:15:43 -0800 (PST) Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB17@GRFEXC.intern.adiscon.com> References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201193554.GA8881@albatross.oerlikon.madduck.net> <9B6E2A8877C38245BFB15CC491A11DA71DDB17@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 1 Feb 2011, Rainer Gerhards wrote: > This sounds useful, at least for some use cases. However, this most probably > requires considerable changes to the config engine. And as this engine is > scheduled to be replaced, I don't like to touch it too much. > > However, there may be one solution. We could declare something like "if the > file name is defined as a template name, then make it a dynafile based on > that template". > > Then, you would need to define templates e.g > > $template /var/log/messages,"/var/log/messages_%$NoW%" this sounds like a significant simplification to dynafiles in general. > And those would automatically be converted. That seems to require relatively > few changes. > > Would that help? I don't think this will help his case. He is wanting to leave the existing config files untouched and have this change take place. if this capibility was in rsyslog, then he _may_ get away with it, but only if all writes to logs are done via includes and he can include his redefinition of things prior to the outputs being defined, _and_ he specifies all files this way. this seems like a pretty fragile way of doing things. I will admit that I don't understand his reluctance to modify the distro config files. especially if he is just modifying the includes, there is nothing in those files that should have to change from release to release (of either rsyslog or the distro). the main rsyslog.conf file may change, so I can understand wanting to avoid touching that (not neccessarily agree with, but understand ;-) but the snippits that are included really are the local configuration, and are expected to be tailored to your installation. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of martin f krafft >> Sent: Tuesday, February 01, 2011 8:36 PM >> To: david at lang.hm >> Cc: rsyslog-users; 536432 at bugs.debian.org >> Subject: Re: [rsyslog] Appending a timestamp to all log files >> >> also sprach david at lang.hm [2011.02.01.1957 +0100]: >>> rsyslog already has the include directive to include files (or all >>> files in a directory) >>> >>> does this work for you? if not, what does it lack? >> >> Hello David, >> >> thanks for your time in writing back. Unfortunately, I think I was >> unclear about what I am trying to do. >> >> I am aware of the include directive. It has nothing to do with my >> goals. >> >> I am trying to tell rsyslog: "hey, all files that you write, append the >> following suffix to their names!", or "hey, all files that you write, >> run their names through this filter and use the output as actual >> filename!". >> >> I want this because it's a cleaner way to replace all hard-coded >> filenames with templated filenames, than to change every instance of a >> hard-coded filename. >> >> Am I making myself clearer now? >> >> -- >> martin; (greetings from the heart of the sun.) >> \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net at madduck >> >> logik ist analsadismus: gedanken werden gewaltsam durch einen engen >> gang gepre?t. >> -- frei nach lacan >> >> spamtraps: madduck.bogus at madduck.net > From toddmichael at gmail.com Wed Feb 2 07:05:28 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Tue, 1 Feb 2011 22:05:28 -0800 Subject: [rsyslog] netcat (nc) utility to /dev/log not working with rsyslog Message-ID: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com> I'm trying to transition from sysklog and initially planned on using syslog-ng, but have since been turned on to rsyslog. I have a working copy of syslog-ng and I routinely test sends of larger log messages using netcat, rather than logger, because logger has a 1024-ish byte message size limit. I would instead put my large message in /tmp/file and do something like this: nc -w0 -U /dev/log < /tmp/file Pretty straightforward and works fine when testing syslog or syslog-ng. Unfortunately, it does not work with rsyslog, even though logger does. Even in debug mode, I see no response from rsyslog when I run this command. Is there something I'm missing here? Thanks todd From rgerhards at hq.adiscon.com Wed Feb 2 08:24:56 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Feb 2011 08:24:56 +0100 Subject: [rsyslog] netcat (nc) utility to /dev/log not working with rsyslog References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> Can you provide a debug log? This sounds a bit strange... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > Sent: Wednesday, February 02, 2011 7:05 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] netcat (nc) utility to /dev/log not working with > rsyslog > > I'm trying to transition from sysklog and initially planned on using > syslog-ng, but have since been turned on to rsyslog. I have a working > copy of syslog-ng and I routinely test sends of larger log messages > using netcat, rather than logger, because logger has a 1024-ish byte > message size limit. I would instead put my large message in /tmp/file > and do something like this: > > nc -w0 -U /dev/log < /tmp/file > > Pretty straightforward and works fine when testing syslog or syslog-ng. > Unfortunately, it does not work with rsyslog, even though logger does. > Even in debug mode, I see no response from rsyslog when I run this > command. Is there something I'm missing here? Thanks > > todd > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From madduck at madduck.net Wed Feb 2 08:46:57 2011 From: madduck at madduck.net (martin f krafft) Date: Wed, 2 Feb 2011 08:46:57 +0100 Subject: [rsyslog] Appending a timestamp to all log files In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB17@GRFEXC.intern.adiscon.com> References: <20110201183322.GA10663@albatross.oerlikon.madduck.net> <20110201193554.GA8881@albatross.oerlikon.madduck.net> <9B6E2A8877C38245BFB15CC491A11DA71DDB17@GRFEXC.intern.adiscon.com> Message-ID: <20110202074657.GD10155@albatross.oerlikon.madduck.net> also sprach Rainer Gerhards [2011.02.01.2155 +0100]: > Then, you would need to define templates e.g > > $template /var/log/messages,"/var/log/messages_%$NoW%" > > And those would automatically be converted. That seems to require relatively > few changes. > > Would that help? It would be a good step because it would allow me to convert existing rules into rules using templates without changing the rules ? just adding a file to /etc/rsyslog.d with the $template statements would do. Thanks for your consideration! -- martin | http://madduck.net/ | http://two.sentenc.es/ "i hate vulgar realism in literature. the man who could call a spade a spade should be compelled to use one. it is the only thing he is fit for." -- oscar wilde spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 1124 bytes Desc: Digital signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: From toddmichael at gmail.com Wed Feb 2 08:58:07 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Tue, 1 Feb 2011 23:58:07 -0800 Subject: [rsyslog] netcat (nc) utility to /dev/log not working with rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> Message-ID: <097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com> Rainer, Thanks for getting back to me so swiftly. Attached is a debug from a second test system - first system was CentOS 5 box with some minor config customization. This is a Fedora box with default RPM install. Same result: nothing to logs and nothing thrown in debug mode. I tried both "-c 3" as well as default backward compatibility mode. Have not yet tried throwing at network socket, but I guess I can give that a shot next. Appreciate your expertise on this. todd On Feb 1, 2011, at 11:24 PM, Rainer Gerhards wrote: > Can you provide a debug log? This sounds a bit strange... > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell >> Sent: Wednesday, February 02, 2011 7:05 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] netcat (nc) utility to /dev/log not working with >> rsyslog >> >> I'm trying to transition from sysklog and initially planned on using >> syslog-ng, but have since been turned on to rsyslog. I have a working >> copy of syslog-ng and I routinely test sends of larger log messages >> using netcat, rather than logger, because logger has a 1024-ish byte >> message size limit. I would instead put my large message in /tmp/file >> and do something like this: >> >> nc -w0 -U /dev/log < /tmp/file >> >> Pretty straightforward and works fine when testing syslog or syslog-ng. >> Unfortunately, it does not work with rsyslog, even though logger does. >> Even in debug mode, I see no response from rsyslog when I run this >> command. Is there something I'm missing here? Thanks >> >> todd >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog_debug_mama.txt URL: From toddmichael at gmail.com Wed Feb 2 09:09:42 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Wed, 2 Feb 2011 00:09:42 -0800 Subject: [rsyslog] netcat (nc) utility to /dev/log not working with rsyslog In-Reply-To: <097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com> References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> <097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com> Message-ID: <89429322-82A6-47AC-9495-71C9787CBAD3@gmail.com> FYI: enabled remote logging (-r514) and tried via network socket: "nc -w0 -u 127.0.0.1 514 < sample_applog_entry" That worked as expected. On Feb 1, 2011, at 11:58 PM, Todd Michael Bushnell wrote: > Rainer, > > Thanks for getting back to me so swiftly. Attached is a debug from a second test system - first system was CentOS 5 box with some minor config customization. This is a Fedora box with default RPM install. Same result: nothing to logs and nothing thrown in debug mode. I tried both "-c 3" as well as default backward compatibility mode. Have not yet tried throwing at network socket, but I guess I can give that a shot next. Appreciate your expertise on this. > > todd > > > On Feb 1, 2011, at 11:24 PM, Rainer Gerhards wrote: > >> Can you provide a debug log? This sounds a bit strange... >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell >>> Sent: Wednesday, February 02, 2011 7:05 AM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] netcat (nc) utility to /dev/log not working with >>> rsyslog >>> >>> I'm trying to transition from sysklog and initially planned on using >>> syslog-ng, but have since been turned on to rsyslog. I have a working >>> copy of syslog-ng and I routinely test sends of larger log messages >>> using netcat, rather than logger, because logger has a 1024-ish byte >>> message size limit. I would instead put my large message in /tmp/file >>> and do something like this: >>> >>> nc -w0 -U /dev/log < /tmp/file >>> >>> Pretty straightforward and works fine when testing syslog or syslog-ng. >>> Unfortunately, it does not work with rsyslog, even though logger does. >>> Even in debug mode, I see no response from rsyslog when I run this >>> command. Is there something I'm missing here? Thanks >>> >>> todd >>> >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Feb 2 09:55:26 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Feb 2011 09:55:26 +0100 Subject: [rsyslog] netcat (nc) utility to /dev/log not working withrsyslog References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> <097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB1C@GRFEXC.intern.adiscon.com> Mhhh... nothing exciting, so it should work. Rsyslogd uses datagram sockets on /dev/log. You use that with nc, too, right (I have no man available right now).? Also, sysklogd uses datagram sockets as well... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > Sent: Wednesday, February 02, 2011 8:58 AM > To: rsyslog-users > Subject: Re: [rsyslog] netcat (nc) utility to /dev/log not working > withrsyslog > > Rainer, > > Thanks for getting back to me so swiftly. Attached is a debug from a > second test system - first system was CentOS 5 box with some minor > config customization. This is a Fedora box with default RPM install. > Same result: nothing to logs and nothing thrown in debug mode. I tried > both "-c 3" as well as default backward compatibility mode. Have not > yet tried throwing at network socket, but I guess I can give that a > shot next. Appreciate your expertise on this. > > todd > > > On Feb 1, 2011, at 11:24 PM, Rainer Gerhards wrote: > > > Can you provide a debug log? This sounds a bit strange... > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > >> Sent: Wednesday, February 02, 2011 7:05 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] netcat (nc) utility to /dev/log not working with > >> rsyslog > >> > >> I'm trying to transition from sysklog and initially planned on using > >> syslog-ng, but have since been turned on to rsyslog. I have a > working > >> copy of syslog-ng and I routinely test sends of larger log messages > >> using netcat, rather than logger, because logger has a 1024-ish byte > >> message size limit. I would instead put my large message in > /tmp/file > >> and do something like this: > >> > >> nc -w0 -U /dev/log < /tmp/file > >> > >> Pretty straightforward and works fine when testing syslog or syslog- > ng. > >> Unfortunately, it does not work with rsyslog, even though logger > does. > >> Even in debug mode, I see no response from rsyslog when I run this > >> command. Is there something I'm missing here? Thanks > >> > >> todd > >> > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com From pgollucci at p6m7g8.com Thu Feb 3 07:56:11 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Thu, 3 Feb 2011 06:56:11 +0000 Subject: [rsyslog] rsyslog5 core dumps a-plenty Message-ID: <4D4A518B.8020202@p6m7g8.com> Rainer (All), At first I thought it was something in rsyslog 5.5.6+ that changed b/c I couldn't get the core dumps happen with 5.5.5; however, now I'm pretty sure it has nothing to do with that milestone. Unfortunately, from the looks of it, I'm betting its a FreeBSD specific threading issue. Below is the useful info, gdb back trace, rsyslog.conf, and the build transcript. I've also uploaded the files here for easier viewing: http://people.freebsd.org/~pgollucci/rsyslog-core http://people.apache.org/~pgollucci/rsyslog-core Move info is available on request. $ uname -a FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 23:12:18 UTC 2010 root at elektra :/usr/obj/usr/src/sys/GENERIC amd64 $ rsyslog --version rsyslogd 5.6.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: No FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): Yes Atomic operations supported: Yes Runtime Instrumentation (slow code): Yes $ sudo /usr/local/etc/rc.d/rsyslogd restart rsyslogd not running? (check /var/run/syslog.pid). Starting rsyslogd. $ ps -auxwww |grep rsyslogd root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 /usr/local/sbin/rsyslogd -c4 -i /var /run/syslog.pid -f /usr/local/etc/rsyslog.conf $ cd / ; ls core.rsyslogd.8* -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `rsyslogd'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libz.so.5...done. Loaded symbols for /lib/libz.so.5 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmnet.so Reading symbols from /usr/local/lib/rsyslog/immark.so...done. Loaded symbols for /usr/local/lib/rsyslog/immark.so Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imudp.so Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imtcp.so Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. Loaded symbols for /usr/local/lib/rsyslog/imklog.so Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imrelp.so Reading symbols from /usr/local/lib/librelp.so.0...done. Loaded symbols for /usr/local/lib/librelp.so.0 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 [New Thread 801a3ac80 (LWP 100568)] [New Thread 800c87fc0 (LWP 100567)] [New Thread 800c88180 (LWP 100541)] [New Thread 800c88340 (LWP 100540)] [New Thread 800c88500 (LWP 100539)] [New Thread 800c886c0 (LWP 100538)] [New Thread 800c88880 (LWP 100379)] [New Thread 800c041c0 (LWP 100548)] (gdb) bt full #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 No symbol table info available. #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) at msg.c:624 __func__ = "MsgDeleteMutexLockingCase" #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at msg.c:839 dbgCALLStaCK_POP_POINT = 7 iRet = RS_RET_OK iCancelStateSave = 1 pThis = (msg_t *) 0x801aaea00 currRefCount = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 __func__ = "msgDestruct" #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, pBatch=0x800c2fe70) at queue.c:1404 i = 0 pUsr = (void *) 0x801aaea00 nEnqueued = 0 localRet = 8 dbgCALLStaCK_POP_POINT = 6 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 __func__ = "DeleteProcessedBatch" #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 nDequeued = 8 nDiscarded = 4678208 nDeleted = 2 iQueueSize = 4673262 pUsr = (void *) 0x40042df73 localRet = RS_RET_OK dbgCALLStaCK_POP_POINT = 5 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 __func__ = "DequeueConsumableElements" #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1489 dbgCALLStaCK_POP_POINT = 4 iRet = RS_RET_OK iQueueSize = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 __func__ = "DequeueConsumable" #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1626 dbgCALLStaCK_POP_POINT = 3 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 __func__ = "DequeueForConsumer" #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1679 iCancelStateSave = 8 dbgCALLStaCK_POP_POINT = 2 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 __func__ = "ConsumerReg" #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} pWtp = (wtp_t *) 0x800c1c380 bInactivityTOOccured = 0 localRet = RS_RET_OK terminateRet = RS_RET_OK iCancelStateSave = 0 dbgCALLStaCK_POP_POINT = 1 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 __func__ = "wtiWorker" #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, 34372517440, 0, 0, 0, 0, 0}} pWti = (wti_t *) 0x800c2fe40 pThis = (wtp_t *) 0x800c1c380 sigSet = {__bits = {1048576, 0, 0, 0}} dbgCALLStaCK_POP_POINT = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 __func__ = "wtpWorker" #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 No symbol table info available. #11 0x0000000000000000 in ?? () No symbol table info available. Cannot access memory at address 0x7ffffeff9000 (gdb) frame 1 #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) at msg.c:624 624 msg.c: No such file or directory. in msg.c (gdb) info locals __func__ = "MsgDeleteMutexLockingCase" (gdb) p *pThis $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, iFacility = 22, of fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae b75 "phoenix", pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = 0x0, pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m onth = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = 776392}, szRawMsg = '\xff' , szHOSTNAME = "phoen ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { pszTAG = 0x722e313030707061
, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", '\xff' , pszTimestamp3339 = "2011-02-03T06:25:05.776392 +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} (gdb) quit rsyslog.conf: ------------- #### Modules ## Input $ModLoad immark $ModLoad imudp $ModLoad imtcp $ModLoad imuxsock $ModLoad imklog $ModLoad imrelp ## Output #$ModLoad omsnmp.so # Send SNMP traps #### Globals $umask 0000 $DirCreateMode 0640 $FileCreateMode 0640 $RepeatedMsgReduction on #$RepeatedMsgContainsOrigionalMsg on $DefaultNetstreamDriver ptcp # location for work (spool) file $WorkDirectory /var/spool/rsyslog # Queue Types $MainMsgQueueType LinkedList $ActionQueueType LinkedList # Become DA Queues $MainMsgQueueFileName mainq $ActionQueueFileNAme actionq # Disable Rate Throttling $MainMsgQueueTimeoutEnqueue 0 $ActionQueueTimeoutEnqueue 0 # Discard any message at all if there double the queue we allow in memory # 10000 in ram + 10000 on disk $MainMsgQueueDiscardMark 20000 $ActionQueueDiscardMark 20000 0 Emergency: system is unusable # 1 Alert: action must be taken immediately # 2 Critical: critical conditions # 3 Error: error conditions # 4 Warning: warning conditions # 5 Notice: normal but significant condition # 6 Informational: informational messages # 7 Debug: debug-level messages # When discarding kicks in (above) any msg # with priority >= this is out the door $MainMsgQueueDiscardSeverity 0 $ActionQueueDiscardSeverity 0 # default templates $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $UDPServerRun 514 $InputRELPServerRun 2514 # Templates $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" #### Logging Facilities *.* @localhost:18712;logsaggd +hulk *.err;kern.warning;auth.notice;mail.crit /var/log/console.log *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug /var/log/debug.log *.emerg * -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From toddmichael at gmail.com Wed Feb 2 18:39:49 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Wed, 2 Feb 2011 09:39:49 -0800 Subject: [rsyslog] netcat (nc) utility to /dev/log not working withrsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB1C@GRFEXC.intern.adiscon.com> References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com> <097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB1C@GRFEXC.intern.adiscon.com> Message-ID: <96F288C1-D35B-4B19-95CE-BB61F98435AA@gmail.com> Rainer, I stand corrected. It turns out this does not work for sysklog or rsyslog, but does work with syslog-ng. My apologies - I must have only used logger with sysklog which makes sense given that I didn't need to test large messages with sysklog; I already know it doesn't work :-) I know from your documentation that there are some standards with which rsyslog complies that syslog-ng does not necessarily. My need was to test large messages against rsyslog which I now can do via network socket so my personal emergency is over. Still, I am curious why I can do this with syslog-ng, but not rsyslog and sysklog so the curious cat in me will probably geek out on that for awhile to figure out why. I'm certain I'll end up learning something along the way and will share whatever I do learn with this email list. I searched netcat manpage for "datagram" and got nothing. The option I use is explained like so: -U Specifies to use Unix Domain Sockets. That's not the same thing as a datagram socket so I guess syslog-ng "takes over" /dev/log a little bit differently to do its thing than does rsyslog and sysklog. Sorry for the misinformation and my appreciation for your prompt and diligent assistance. Keep up the great work. Todd On Feb 2, 2011, at 12:55 AM, Rainer Gerhards wrote: > Mhhh... nothing exciting, so it should work. Rsyslogd uses datagram sockets > on /dev/log. You use that with nc, too, right (I have no man available right > now).? Also, sysklogd uses datagram sockets as well... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell >> Sent: Wednesday, February 02, 2011 8:58 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] netcat (nc) utility to /dev/log not working >> withrsyslog >> >> Rainer, >> >> Thanks for getting back to me so swiftly. Attached is a debug from a >> second test system - first system was CentOS 5 box with some minor >> config customization. This is a Fedora box with default RPM install. >> Same result: nothing to logs and nothing thrown in debug mode. I tried >> both "-c 3" as well as default backward compatibility mode. Have not >> yet tried throwing at network socket, but I guess I can give that a >> shot next. Appreciate your expertise on this. >> >> todd >> >> >> On Feb 1, 2011, at 11:24 PM, Rainer Gerhards wrote: >> >>> Can you provide a debug log? This sounds a bit strange... >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell >>>> Sent: Wednesday, February 02, 2011 7:05 AM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] netcat (nc) utility to /dev/log not working with >>>> rsyslog >>>> >>>> I'm trying to transition from sysklog and initially planned on using >>>> syslog-ng, but have since been turned on to rsyslog. I have a >> working >>>> copy of syslog-ng and I routinely test sends of larger log messages >>>> using netcat, rather than logger, because logger has a 1024-ish byte >>>> message size limit. I would instead put my large message in >> /tmp/file >>>> and do something like this: >>>> >>>> nc -w0 -U /dev/log < /tmp/file >>>> >>>> Pretty straightforward and works fine when testing syslog or syslog- >> ng. >>>> Unfortunately, it does not work with rsyslog, even though logger >> does. >>>> Even in debug mode, I see no response from rsyslog when I run this >>>> command. Is there something I'm missing here? Thanks >>>> >>>> todd >>>> >>>> >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 3 01:18:11 2011 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 3 Feb 2011 01:18:11 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> Message-ID: 2011/2/1 Rainer Gerhards : > Hi David, > > thanks for putting this straight. It was actually my fault not to tell Tom > this for v6. I think the release cycle was a bit too long, too many new > features. I also think I need to concentrate on v6 now for new features -- > going back and forth between v5 and v6 for development releases is sometimes > quite confusing, at least to me ;) A stable release with the systemd bits included would be nice. If you concentrate on v6, does that mean that getting 5.7.x stable and ready for 5.8 is less on your focus? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 3 11:07:24 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Feb 2011 11:07:24 +0100 Subject: [rsyslog] netcat (nc) utility to /dev/log not workingwithrsyslog References: <531FEBB2-2301-43BA-A595-2CAA209872BA@gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DDB19@GRFEXC.intern.adiscon.com><097BF0AD-E8CF-4301-BADE-389C52C9BA6E@gmail.com><9B6E2A8877C38245BFB15CC491A11DA71DDB1C@GRFEXC.intern.adiscon.com> <96F288C1-D35B-4B19-95CE-BB61F98435AA@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB34@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > Sent: Wednesday, February 02, 2011 6:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] netcat (nc) utility to /dev/log not > workingwithrsyslog > > Rainer, > > I stand corrected. It turns out this does not work for sysklog or > rsyslog, but does work with syslog-ng. My apologies - I must have only > used logger with sysklog which makes sense given that I didn't need to > test large messages with sysklog; I already know it doesn't work :-) I > know from your documentation that there are some standards with which > rsyslog complies that syslog-ng does not necessarily. My need was to > test large messages against rsyslog which I now can do via network > socket so my personal emergency is over. Still, I am curious why I can > do this with syslog-ng, but not rsyslog and sysklog so the curious cat > in me will probably geek out on that for awhile to figure out why. I'm > certain I'll end up learning something along the way and will share > whatever I do learn with this email list. I searched netcat manpage > for "datagram" and got nothing. The option I use is explained like so: > > -U Specifies to use Unix Domain Sockets. > > That's not the same thing as a datagram socket so I guess syslog-ng > "takes over" /dev/log a little bit differently to do its thing than > does rsyslog and sysklog. Sorry for the misinformation and my > appreciation for your prompt and diligent assistance. Keep up the > great work. Ah. That explains it. The story is rather simple: there are datagram and stream sockets for unix domain sockets as well. Syslog-ng, as far as I know, implements stream sockets, whereas rsyslog sticks with the standard datagram socket way of doing things. Both have pro's and con's. One result seems to be that netcat seems not to support datagram sockets (which I find surprising, but...). HTH Rainer > > Todd > > > > > On Feb 2, 2011, at 12:55 AM, Rainer Gerhards wrote: > > > Mhhh... nothing exciting, so it should work. Rsyslogd uses datagram > sockets > > on /dev/log. You use that with nc, too, right (I have no man > available right > > now).? Also, sysklogd uses datagram sockets as well... > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > >> Sent: Wednesday, February 02, 2011 8:58 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] netcat (nc) utility to /dev/log not working > >> withrsyslog > >> > >> Rainer, > >> > >> Thanks for getting back to me so swiftly. Attached is a debug from > a > >> second test system - first system was CentOS 5 box with some minor > >> config customization. This is a Fedora box with default RPM > install. > >> Same result: nothing to logs and nothing thrown in debug mode. I > tried > >> both "-c 3" as well as default backward compatibility mode. Have > not > >> yet tried throwing at network socket, but I guess I can give that a > >> shot next. Appreciate your expertise on this. > >> > >> todd > >> > >> > >> On Feb 1, 2011, at 11:24 PM, Rainer Gerhards wrote: > >> > >>> Can you provide a debug log? This sounds a bit strange... > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Todd Michael Bushnell > >>>> Sent: Wednesday, February 02, 2011 7:05 AM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: [rsyslog] netcat (nc) utility to /dev/log not working > with > >>>> rsyslog > >>>> > >>>> I'm trying to transition from sysklog and initially planned on > using > >>>> syslog-ng, but have since been turned on to rsyslog. I have a > >> working > >>>> copy of syslog-ng and I routinely test sends of larger log > messages > >>>> using netcat, rather than logger, because logger has a 1024-ish > byte > >>>> message size limit. I would instead put my large message in > >> /tmp/file > >>>> and do something like this: > >>>> > >>>> nc -w0 -U /dev/log < /tmp/file > >>>> > >>>> Pretty straightforward and works fine when testing syslog or > syslog- > >> ng. > >>>> Unfortunately, it does not work with rsyslog, even though logger > >> does. > >>>> Even in debug mode, I see no response from rsyslog when I run this > >>>> command. Is there something I'm missing here? Thanks > >>>> > >>>> todd > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 3 11:13:53 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Feb 2011 11:13:53 +0100 Subject: [rsyslog] rsyslog5 core dumps a-plenty References: <4D4A518B.8020202@p6m7g8.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB37@GRFEXC.intern.adiscon.com> Thanks! I am after one bug in v5, which looks like a race. So far, it has been very hard for me to reproduce it. Maybe it works better on BSD! I am not sure if I can look into it in depth this week, as I am working on some rather complex threadin improvement, but will check ASAP. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > Sent: Thursday, February 03, 2011 7:56 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog5 core dumps a-plenty > > Rainer (All), > > At first I thought it was something in rsyslog 5.5.6+ that changed b/c > I > couldn't get the core dumps happen with 5.5.5; however, now I'm pretty > sure it has nothing to do with that milestone. Unfortunately, from the > looks of it, I'm betting its a FreeBSD specific threading issue. Below > is the useful info, gdb back trace, rsyslog.conf, and the build > transcript. > > I've also uploaded the files here for easier viewing: > http://people.freebsd.org/~pgollucci/rsyslog-core > http://people.apache.org/~pgollucci/rsyslog-core > > Move info is available on request. > > > > $ uname -a > FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 > 23:12:18 UTC 2010 root at elektra > :/usr/obj/usr/src/sys/GENERIC amd64 > > $ rsyslog --version > rsyslogd 5.6.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): Yes > Atomic operations supported: Yes > Runtime Instrumentation (slow code): Yes > > $ sudo /usr/local/etc/rc.d/rsyslogd restart > rsyslogd not running? (check /var/run/syslog.pid). > Starting rsyslogd. > > $ ps -auxwww |grep rsyslogd > root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 > /usr/local/sbin/rsyslogd -c4 -i /var > /run/syslog.pid -f /usr/local/etc/rsyslog.conf > > $ cd / ; ls core.rsyslogd.8* > -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 > -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 > -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 > -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 > -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 > -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 > > $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `rsyslogd'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libz.so.5...done. > Loaded symbols for /lib/libz.so.5 > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnet.so > Reading symbols from /usr/local/lib/rsyslog/immark.so...done. > Loaded symbols for /usr/local/lib/rsyslog/immark.so > Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imudp.so > Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imtcp.so > Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so > Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so > Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so > Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imklog.so > Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imrelp.so > Reading symbols from /usr/local/lib/librelp.so.0...done. > Loaded symbols for /usr/local/lib/librelp.so.0 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > [New Thread 801a3ac80 (LWP 100568)] > [New Thread 800c87fc0 (LWP 100567)] > [New Thread 800c88180 (LWP 100541)] > [New Thread 800c88340 (LWP 100540)] > [New Thread 800c88500 (LWP 100539)] > [New Thread 800c886c0 (LWP 100538)] > [New Thread 800c88880 (LWP 100379)] > [New Thread 800c041c0 (LWP 100548)] > (gdb) bt full > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > No symbol table info available. > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > __func__ = "MsgDeleteMutexLockingCase" > #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at > msg.c:839 > dbgCALLStaCK_POP_POINT = 7 > iRet = RS_RET_OK > iCancelStateSave = 1 > pThis = (msg_t *) 0x801aaea00 > currRefCount = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 > __func__ = "msgDestruct" > #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, > pBatch=0x800c2fe70) at queue.c:1404 > i = 0 > pUsr = (void *) 0x801aaea00 > nEnqueued = 0 > localRet = 8 > dbgCALLStaCK_POP_POINT = 6 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 > __func__ = "DeleteProcessedBatch" > #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, > pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 > nDequeued = 8 > nDiscarded = 4678208 > nDeleted = 2 > iQueueSize = 4673262 > pUsr = (void *) 0x40042df73 > localRet = RS_RET_OK > dbgCALLStaCK_POP_POINT = 5 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 > __func__ = "DequeueConsumableElements" > #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1489 > dbgCALLStaCK_POP_POINT = 4 > iRet = RS_RET_OK > iQueueSize = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 > __func__ = "DequeueConsumable" > #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1626 > dbgCALLStaCK_POP_POINT = 3 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 > __func__ = "DequeueForConsumer" > #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1679 > iCancelStateSave = 8 > dbgCALLStaCK_POP_POINT = 2 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 > __func__ = "ConsumerReg" > #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 > __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, > 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} > pWtp = (wtp_t *) 0x800c1c380 > bInactivityTOOccured = 0 > localRet = RS_RET_OK > terminateRet = RS_RET_OK > iCancelStateSave = 0 > dbgCALLStaCK_POP_POINT = 1 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 > __func__ = "wtiWorker" > #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 > __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, > 34372517440, 0, 0, 0, 0, 0}} > pWti = (wti_t *) 0x800c2fe40 > pThis = (wtp_t *) 0x800c1c380 > sigSet = {__bits = {1048576, 0, 0, 0}} > dbgCALLStaCK_POP_POINT = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 > __func__ = "wtpWorker" > #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 > No symbol table info available. > #11 0x0000000000000000 in ?? () > No symbol table info available. > Cannot access memory at address 0x7ffffeff9000 > (gdb) frame 1 > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > 624 msg.c: No such file or directory. > in msg.c > (gdb) info locals > __func__ = "MsgDeleteMutexLockingCase" > (gdb) p *pThis > $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, > pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = > 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, > iFacility = 22, of > fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, > iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, > pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix > app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae > b75 "phoenix", > pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, > pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = > 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = > 0x0, > pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, > pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, > pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset > = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m > onth = 2 '\002', day = 3 '\003', hour = 6 '\006', > minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', > OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = > 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon > th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', > second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 > '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = > 776392}, szRawMsg = '\xff' , szHOSTNAME = "phoen > ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { > pszTAG = 0x722e313030707061
bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", > '\xff' , pszTimestamp3339 = "2011-02- > 03T06:25:05.776392 > +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", > pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} > (gdb) quit > > rsyslog.conf: > ------------- > #### Modules > ## Input > $ModLoad immark > $ModLoad imudp > $ModLoad imtcp > $ModLoad imuxsock > $ModLoad imklog > $ModLoad imrelp > > ## Output > #$ModLoad omsnmp.so # Send SNMP traps > > #### Globals > $umask 0000 > $DirCreateMode 0640 > $FileCreateMode 0640 > > $RepeatedMsgReduction on > #$RepeatedMsgContainsOrigionalMsg on > > $DefaultNetstreamDriver ptcp > > # location for work (spool) file > $WorkDirectory /var/spool/rsyslog > > # Queue Types > $MainMsgQueueType LinkedList > $ActionQueueType LinkedList > > # Become DA Queues > $MainMsgQueueFileName mainq > $ActionQueueFileNAme actionq > > # Disable Rate Throttling > $MainMsgQueueTimeoutEnqueue 0 > $ActionQueueTimeoutEnqueue 0 > > # Discard any message at all if there double the queue we allow in > memory > # 10000 in ram + 10000 on disk > $MainMsgQueueDiscardMark 20000 > $ActionQueueDiscardMark 20000 > > 0 Emergency: system is unusable > # 1 Alert: action must be taken immediately > # 2 Critical: critical conditions > # 3 Error: error conditions > # 4 Warning: warning conditions > # 5 Notice: normal but significant condition > # 6 Informational: informational messages > # 7 Debug: debug-level messages > > # When discarding kicks in (above) any msg > # with priority >= this is out the door > $MainMsgQueueDiscardSeverity 0 > $ActionQueueDiscardSeverity 0 > > # default templates > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $UDPServerRun 514 > $InputRELPServerRun 2514 > > # Templates > $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% > %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" > > #### Logging Facilities > *.* @localhost:18712;logsaggd > > +hulk > *.err;kern.warning;auth.notice;mail.crit > /var/log/console.log > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug /var/log/debug.log > *.emerg * > > -- > ----------------------------------------------------------------------- > - > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 > VP Apache Infrastructure; Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 3 11:17:10 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Feb 2011 11:17:10 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB38@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 03, 2011 1:18 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > > 2011/2/1 Rainer Gerhards : > > Hi David, > > > > thanks for putting this straight. It was actually my fault not to > tell Tom > > this for v6. I think the release cycle was a bit too long, too many > new > > features. I also think I need to concentrate on v6 now for new > features -- > > going back and forth between v5 and v6 for development releases is > sometimes > > quite confusing, at least to me ;) > > A stable release with the systemd bits included would be nice. Yeah, that's right. Let me check what the current betas say... > > If you concentrate on v6, does that mean that getting 5.7.x stable and > ready for 5.8 is less on your focus? No! The "for new features" part of my reply is important. In fact, if I concentrate on v6 for new features, this helps with getting the stable versions out sooner. Actually, having "two and a half" devel builds (some v4, full v5 and v6) takes up quite some time, as I regularly need to change code that I just developed. If I develop new things on v6 only, I save that time (and that's quite a bit). This also means debugging only on v6. Plus it is easier to track mentally which codebase I work with. Rainer From rgerhards at hq.adiscon.com Thu Feb 3 11:45:19 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Feb 2011 11:45:19 +0100 Subject: [rsyslog] rsyslog5 core dumps a-plenty References: <4D4A518B.8020202@p6m7g8.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB3C@GRFEXC.intern.adiscon.com> Philip, the config looks relatively simple, that's good (so far I had more complex configs to work with). Can you tell me a bit about the workload you process. Just the regular stuff? Do you have the impression that this bug is triggered by some specific messages? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > Sent: Thursday, February 03, 2011 7:56 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog5 core dumps a-plenty > > Rainer (All), > > At first I thought it was something in rsyslog 5.5.6+ that changed b/c > I > couldn't get the core dumps happen with 5.5.5; however, now I'm pretty > sure it has nothing to do with that milestone. Unfortunately, from the > looks of it, I'm betting its a FreeBSD specific threading issue. Below > is the useful info, gdb back trace, rsyslog.conf, and the build > transcript. > > I've also uploaded the files here for easier viewing: > http://people.freebsd.org/~pgollucci/rsyslog-core > http://people.apache.org/~pgollucci/rsyslog-core > > Move info is available on request. > > > > $ uname -a > FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 > 23:12:18 UTC 2010 root at elektra > :/usr/obj/usr/src/sys/GENERIC amd64 > > $ rsyslog --version > rsyslogd 5.6.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): Yes > Atomic operations supported: Yes > Runtime Instrumentation (slow code): Yes > > $ sudo /usr/local/etc/rc.d/rsyslogd restart > rsyslogd not running? (check /var/run/syslog.pid). > Starting rsyslogd. > > $ ps -auxwww |grep rsyslogd > root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 > /usr/local/sbin/rsyslogd -c4 -i /var > /run/syslog.pid -f /usr/local/etc/rsyslog.conf > > $ cd / ; ls core.rsyslogd.8* > -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 > -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 > -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 > -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 > -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 > -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 > > $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `rsyslogd'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libz.so.5...done. > Loaded symbols for /lib/libz.so.5 > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnet.so > Reading symbols from /usr/local/lib/rsyslog/immark.so...done. > Loaded symbols for /usr/local/lib/rsyslog/immark.so > Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imudp.so > Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imtcp.so > Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so > Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so > Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so > Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imklog.so > Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imrelp.so > Reading symbols from /usr/local/lib/librelp.so.0...done. > Loaded symbols for /usr/local/lib/librelp.so.0 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > [New Thread 801a3ac80 (LWP 100568)] > [New Thread 800c87fc0 (LWP 100567)] > [New Thread 800c88180 (LWP 100541)] > [New Thread 800c88340 (LWP 100540)] > [New Thread 800c88500 (LWP 100539)] > [New Thread 800c886c0 (LWP 100538)] > [New Thread 800c88880 (LWP 100379)] > [New Thread 800c041c0 (LWP 100548)] > (gdb) bt full > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > No symbol table info available. > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > __func__ = "MsgDeleteMutexLockingCase" > #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at > msg.c:839 > dbgCALLStaCK_POP_POINT = 7 > iRet = RS_RET_OK > iCancelStateSave = 1 > pThis = (msg_t *) 0x801aaea00 > currRefCount = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 > __func__ = "msgDestruct" > #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, > pBatch=0x800c2fe70) at queue.c:1404 > i = 0 > pUsr = (void *) 0x801aaea00 > nEnqueued = 0 > localRet = 8 > dbgCALLStaCK_POP_POINT = 6 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 > __func__ = "DeleteProcessedBatch" > #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, > pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 > nDequeued = 8 > nDiscarded = 4678208 > nDeleted = 2 > iQueueSize = 4673262 > pUsr = (void *) 0x40042df73 > localRet = RS_RET_OK > dbgCALLStaCK_POP_POINT = 5 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 > __func__ = "DequeueConsumableElements" > #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1489 > dbgCALLStaCK_POP_POINT = 4 > iRet = RS_RET_OK > iQueueSize = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 > __func__ = "DequeueConsumable" > #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1626 > dbgCALLStaCK_POP_POINT = 3 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 > __func__ = "DequeueForConsumer" > #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1679 > iCancelStateSave = 8 > dbgCALLStaCK_POP_POINT = 2 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 > __func__ = "ConsumerReg" > #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 > __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, > 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} > pWtp = (wtp_t *) 0x800c1c380 > bInactivityTOOccured = 0 > localRet = RS_RET_OK > terminateRet = RS_RET_OK > iCancelStateSave = 0 > dbgCALLStaCK_POP_POINT = 1 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 > __func__ = "wtiWorker" > #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 > __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, > 34372517440, 0, 0, 0, 0, 0}} > pWti = (wti_t *) 0x800c2fe40 > pThis = (wtp_t *) 0x800c1c380 > sigSet = {__bits = {1048576, 0, 0, 0}} > dbgCALLStaCK_POP_POINT = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 > __func__ = "wtpWorker" > #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 > No symbol table info available. > #11 0x0000000000000000 in ?? () > No symbol table info available. > Cannot access memory at address 0x7ffffeff9000 > (gdb) frame 1 > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > 624 msg.c: No such file or directory. > in msg.c > (gdb) info locals > __func__ = "MsgDeleteMutexLockingCase" > (gdb) p *pThis > $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, > pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = > 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, > iFacility = 22, of > fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, > iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, > pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix > app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae > b75 "phoenix", > pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, > pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = > 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = > 0x0, > pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, > pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, > pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset > = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m > onth = 2 '\002', day = 3 '\003', hour = 6 '\006', > minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', > OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = > 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon > th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', > second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 > '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = > 776392}, szRawMsg = '\xff' , szHOSTNAME = "phoen > ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { > pszTAG = 0x722e313030707061
bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", > '\xff' , pszTimestamp3339 = "2011-02- > 03T06:25:05.776392 > +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", > pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} > (gdb) quit > > rsyslog.conf: > ------------- > #### Modules > ## Input > $ModLoad immark > $ModLoad imudp > $ModLoad imtcp > $ModLoad imuxsock > $ModLoad imklog > $ModLoad imrelp > > ## Output > #$ModLoad omsnmp.so # Send SNMP traps > > #### Globals > $umask 0000 > $DirCreateMode 0640 > $FileCreateMode 0640 > > $RepeatedMsgReduction on > #$RepeatedMsgContainsOrigionalMsg on > > $DefaultNetstreamDriver ptcp > > # location for work (spool) file > $WorkDirectory /var/spool/rsyslog > > # Queue Types > $MainMsgQueueType LinkedList > $ActionQueueType LinkedList > > # Become DA Queues > $MainMsgQueueFileName mainq > $ActionQueueFileNAme actionq > > # Disable Rate Throttling > $MainMsgQueueTimeoutEnqueue 0 > $ActionQueueTimeoutEnqueue 0 > > # Discard any message at all if there double the queue we allow in > memory > # 10000 in ram + 10000 on disk > $MainMsgQueueDiscardMark 20000 > $ActionQueueDiscardMark 20000 > > 0 Emergency: system is unusable > # 1 Alert: action must be taken immediately > # 2 Critical: critical conditions > # 3 Error: error conditions > # 4 Warning: warning conditions > # 5 Notice: normal but significant condition > # 6 Informational: informational messages > # 7 Debug: debug-level messages > > # When discarding kicks in (above) any msg > # with priority >= this is out the door > $MainMsgQueueDiscardSeverity 0 > $ActionQueueDiscardSeverity 0 > > # default templates > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $UDPServerRun 514 > $InputRELPServerRun 2514 > > # Templates > $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% > %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" > > #### Logging Facilities > *.* @localhost:18712;logsaggd > > +hulk > *.err;kern.warning;auth.notice;mail.crit > /var/log/console.log > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug /var/log/debug.log > *.emerg * > > -- > ----------------------------------------------------------------------- > - > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 > VP Apache Infrastructure; Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From pgollucci at p6m7g8.com Fri Feb 4 00:47:08 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Thu, 3 Feb 2011 23:47:08 +0000 Subject: [rsyslog] rsyslog5 core dumps a-plenty In-Reply-To: <4D4B0446.8010503@p6m7g8.com> References: <4D4A518B.8020202@p6m7g8.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB3C@GRFEXC.intern.adiscon.com> <4D4B0446.8010503@p6m7g8.com> Message-ID: <4D4B3E7C.1090407@p6m7g8.com> FWIW, 5.6.3 is no better or worse; however, 5.7.2 seems to be holding its own for all of about 15 minutes. I'm going to watch it closely on the central box before I push that out to all the clients. 5.7.1 does not. On 02/03/11 19:38, Philip M. Gollucci wrote: > On 02/03/11 10:45, Rainer Gerhards wrote: >> Philip, >> >> the config looks relatively simple, that's good (so far I had more complex >> configs to work with). Can you tell me a bit about the workload you process. >> Just the regular stuff? Do you have the impression that this bug is triggered >> by some specific messages? > > While I don't think the issues is 5.5.[56]+ specific, Its definitely > progressively less stable as you get newer versions. I might give 5.6.3 > a spin later today after I update the FreeBSD port. > > What kind of work load stats do you want ? I can get pretty detailed, > here's my guess at whats useful: > > ~35 machines + networking hardware, 1 central one all use relp to > communicate except the networking hardware which is using UDP. > > The master passes to a perl daemon to insert into a local mysql and > aggregate real-time. (ommysql was just too tedious to do all the sql I > could do in perl). > > We essentially log everything both system and app (PCI-level 1) > > We're doing between 600-1500 messages/second depending on time of day > and how many of the rsyslogds happen to be in the 1 minute of downtime > before my cron restarts them. > > Also attached now is a ktrace -i / kdump -sE > > +---------------------------------------------------------------+ > | table_schema | data_size | index_size | total_size | > +---------------------------------------------------------------+ > | logs | 133.23G | 48.58G | 181.82G | > > > Db Sizes by table (in RAM) CL.logs: > +----------------------------------------------------------------+ > | table_name | data_size | index_size | total_size | > +----------------------------------------------------------------+ > | web_logs_all | 53.17G | -- | 53.17G | > | mk_logs | 12.14G | 4.07G | 16.22G | > | web_logs | 6.91G | 4.5G | 11.42G | > | system_logs | 7.52G | 3.27G | 10.79G | > | ripsd_logs | 424.49M | 483.16M | 907.65M | > | windows_logs | 704.87M | 118.94M | 823.81M | > | daemon_logs | 435.61M | 210.74M | 646.36M | > | web_error_logs | 357.83M | 187.23M | 545.06M | > | mail_logs | 236.16M | 66.05M | 302.22M | > | cron_logs | 137.31M | 112.35M | 249.66M | > > > I'm absolutely sure *some* messages crash it, but thats not whats > happening here. (I.E. messages generated by the OTRS software have a > high percentage of doing it). > > One thing I forgot, I sent you the rsyslog.conf from the Central one. > Attached now is the 'client side' version. > -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From pgollucci at p6m7g8.com Fri Feb 4 00:49:54 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Thu, 3 Feb 2011 23:49:54 +0000 Subject: [rsyslog] Fwd: Re: rsyslog5 core dumps a-plenty Message-ID: <4D4B3F22.5030300@p6m7g8.com> This message below never made it the list before b/c the attachments were too large. On 02/03/11 19:38, Philip M. Gollucci wrote: > On 02/03/11 10:45, Rainer Gerhards wrote: >> Philip, >> >> the config looks relatively simple, that's good (so far I had more complex >> configs to work with). Can you tell me a bit about the workload you process. >> Just the regular stuff? Do you have the impression that this bug is triggered >> by some specific messages? > > While I don't think the issues is 5.5.[56]+ specific, Its definitely > progressively less stable as you get newer versions. I might give 5.6.3 > a spin later today after I update the FreeBSD port. > > What kind of work load stats do you want ? I can get pretty detailed, > here's my guess at whats useful: > > ~35 machines + networking hardware, 1 central one all use relp to > communicate except the networking hardware which is using UDP. > > The master passes to a perl daemon to insert into a local mysql and > aggregate real-time. (ommysql was just too tedious to do all the sql I > could do in perl). > > We essentially log everything both system and app (PCI-level 1) > > We're doing between 600-1500 messages/second depending on time of day > and how many of the rsyslogds happen to be in the 1 minute of downtime > before my cron restarts them. > > Also attached now is a ktrace -i / kdump -sE > > +---------------------------------------------------------------+ > | table_schema | data_size | index_size | total_size | > +---------------------------------------------------------------+ > | logs | 133.23G | 48.58G | 181.82G | > > > Db Sizes by table (in RAM) CL.logs: > +----------------------------------------------------------------+ > | table_name | data_size | index_size | total_size | > +----------------------------------------------------------------+ > | web_logs_all | 53.17G | -- | 53.17G | > | mk_logs | 12.14G | 4.07G | 16.22G | > | web_logs | 6.91G | 4.5G | 11.42G | > | system_logs | 7.52G | 3.27G | 10.79G | > | ripsd_logs | 424.49M | 483.16M | 907.65M | > | windows_logs | 704.87M | 118.94M | 823.81M | > | daemon_logs | 435.61M | 210.74M | 646.36M | > | web_error_logs | 357.83M | 187.23M | 545.06M | > | mail_logs | 236.16M | 66.05M | 302.22M | > | cron_logs | 137.31M | 112.35M | 249.66M | > > > I'm absolutely sure *some* messages crash it, but thats not whats > happening here. (I.E. messages generated by the OTRS software have a > high percentage of doing it). > > One thing I forgot, I sent you the rsyslog.conf from the Central one. > Attached now is the 'client side' version. > -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From pgollucci at p6m7g8.com Fri Feb 4 03:09:07 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Fri, 4 Feb 2011 02:09:07 +0000 Subject: [rsyslog] rsyslog5 core dumps a-plenty In-Reply-To: <4D4B3E7C.1090407@p6m7g8.com> References: <4D4A518B.8020202@p6m7g8.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB3C@GRFEXC.intern.adiscon.com> <4D4B0446.8010503@p6m7g8.com> <4D4B3E7C.1090407@p6m7g8.com> Message-ID: <4D4B5FC3.90805@p6m7g8.com> On 02/03/11 23:47, Philip M. Gollucci wrote: > FWIW, > > 5.6.3 is no better or worse; however, > > 5.7.2 seems to be holding its own for all of about 15 minutes. I'm > going to watch it closely on the central box before I push that out to > all the clients. ls core.rsyslogd.2* -rw------- 1 root wheel - 12M Feb 4 01:27:28 2011 core.rsyslogd.27112 -rw------- 1 root wheel - 10M Feb 4 01:50:35 2011 core.rsyslogd.28340 -rw------- 1 root wheel - 10M Feb 4 01:57:08 2011 core.rsyslogd.28561 -rw------- 1 root wheel - 10M Feb 4 02:07:10 2011 core.rsyslogd.28634 D'oh, thats far better then 1/minute though. unfortunately, 5.7.2 has decided to over rule me on my debugging flags and thus I have no symbols in cores at the moment. Another day, another try. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From rgerhards at hq.adiscon.com Fri Feb 4 08:59:08 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Feb 2011 08:59:08 +0100 Subject: [rsyslog] Fwd: Re: rsyslog5 core dumps a-plenty References: <4D4B3F22.5030300@p6m7g8.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB4F@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > Sent: Friday, February 04, 2011 12:50 AM > To: rsyslog-users > Subject: [rsyslog] Fwd: Re: rsyslog5 core dumps a-plenty > > This message below never made it the list before b/c the attachments > were too large. > > > On 02/03/11 19:38, Philip M. Gollucci wrote: > > On 02/03/11 10:45, Rainer Gerhards wrote: > >> Philip, > >> > >> the config looks relatively simple, that's good (so far I had more > complex > >> configs to work with). Can you tell me a bit about the workload you > process. > >> Just the regular stuff? Do you have the impression that this bug is > triggered > >> by some specific messages? > > > > While I don't think the issues is 5.5.[56]+ specific, Its definitely > > progressively less stable as you get newer versions. I might give > 5.6.3 > > a spin later today after I update the FreeBSD port. > > > > What kind of work load stats do you want ? I can get pretty detailed, > > here's my guess at whats useful: > > > > ~35 machines + networking hardware, 1 central one all use relp to > > communicate except the networking hardware which is using UDP. > > > > The master passes to a perl daemon to insert into a local mysql and > > aggregate real-time. (ommysql was just too tedious to do all the sql > I > > could do in perl). > > > > We essentially log everything both system and app (PCI-level 1) > > > > We're doing between 600-1500 messages/second depending on time of day > > and how many of the rsyslogds happen to be in the 1 minute of > downtime > > before my cron restarts them. > > > > Also attached now is a ktrace -i / kdump -sE > > > > +---------------------------------------------------------------+ > > | table_schema | data_size | index_size | total_size | > > +---------------------------------------------------------------+ > > | logs | 133.23G | 48.58G | 181.82G | > > > > > > Db Sizes by table (in RAM) CL.logs: > > +----------------------------------------------------------------+ > > | table_name | data_size | index_size | total_size | > > +----------------------------------------------------------------+ > > | web_logs_all | 53.17G | -- | 53.17G | > > | mk_logs | 12.14G | 4.07G | 16.22G | > > | web_logs | 6.91G | 4.5G | 11.42G | > > | system_logs | 7.52G | 3.27G | 10.79G | > > | ripsd_logs | 424.49M | 483.16M | 907.65M | > > | windows_logs | 704.87M | 118.94M | 823.81M | > > | daemon_logs | 435.61M | 210.74M | 646.36M | > > | web_error_logs | 357.83M | 187.23M | 545.06M | > > | mail_logs | 236.16M | 66.05M | 302.22M | > > | cron_logs | 137.31M | 112.35M | 249.66M | > > > > > > I'm absolutely sure *some* messages crash it, but thats not whats > > happening here. (I.E. messages generated by the OTRS software have a > > high percentage of doing it). Could you provide me a set of sample messages so that I can check? If you like, you can mail me privately. I think the root problem is somewhere message-induced and, if so, it would be very useful to see how it could be triggered. I already use a large set of "malformed" messages (including a few gig of truely random data), but none of them trigger the issue :( Rainer From mbiebl at gmail.com Sun Feb 6 14:43:16 2011 From: mbiebl at gmail.com (Michael Biebl) Date: Sun, 6 Feb 2011 14:43:16 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB38@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB38@GRFEXC.intern.adiscon.com> Message-ID: Hi Rainer! 2011/2/3 Rainer Gerhards : >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, February 03, 2011 1:18 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released >> >> 2011/2/1 Rainer Gerhards : >> > Hi David, >> > >> > thanks for putting this straight. It was actually my fault not to >> tell Tom >> > this for v6. I think the release cycle was a bit too long, too many >> new >> > features. I also think I need to concentrate on v6 now for new >> features -- >> > going back and forth between v5 and v6 for development releases is >> sometimes >> > quite confusing, at least to me ;) >> >> A stable release with the systemd bits included would be nice. > > Yeah, that's right. Let me check what the current betas say... Have you come to a conclusion yet? Is there an ETA for 5.8? Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Feb 7 07:26:12 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Feb 2011 07:26:12 +0100 Subject: [rsyslog] rsyslog 6.1.3 (v6-devel) released References: <9B6E2A8877C38245BFB15CC491A11DA71DDB0B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDB15@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDB38@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB7B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Sunday, February 06, 2011 2:43 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > > Hi Rainer! > > 2011/2/3 Rainer Gerhards : > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl > >> Sent: Thursday, February 03, 2011 1:18 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog 6.1.3 (v6-devel) released > >> > >> 2011/2/1 Rainer Gerhards : > >> > Hi David, > >> > > >> > thanks for putting this straight. It was actually my fault not to > >> tell Tom > >> > this for v6. I think the release cycle was a bit too long, too > many > >> new > >> > features. I also think I need to concentrate on v6 now for new > >> features -- > >> > going back and forth between v5 and v6 for development releases is > >> sometimes > >> > quite confusing, at least to me ;) > >> > >> A stable release with the systemd bits included would be nice. > > > > Yeah, that's right. Let me check what the current betas say... > > Have you come to a conclusion yet? Is there an ETA for 5.8? I need to open up a new beta, but it probably does not need the full three months it usually needs. The reason is that v5-devel had not much real development but mere bugfixing the past weeks. Also, there are only very few bug reports filed against it, which is also a good sign. One thing that concerns me is that there must be an abort condition in the current code, which is occasionally triggered. All bug reports I have filed against it describe this condition. Yet it is hard to reproduce in lab. Anyhow, I am right now working on fixing that issues (actually starting last week I asked Florian if he could help me try to reproduce it), so I am a bit positive we can tackle that beast in the forseable future. Rainer From tbergfeld at hq.adiscon.com Mon Feb 7 11:16:24 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 7 Feb 2011 11:16:24 +0100 Subject: [rsyslog] rsyslog 5.7.3 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB84@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.7.3, the new v5-beta. This release both offers a set of new features and, at the same time, turns 5.7.3 into beta state. At first, this sounds a bit contradictory, but we do this for two reasons: a) the new features introduced are non-intrusive in regard to the existing feature set, so no "bad surprises" are expected b) other than that, primarily bugfixing went into this release, with only few remaining issues being open In order to move towards a new v5-stable, we consider it useful to begin with a new v5-beta stage. Note that this time the beta phase may be rather quick, because of the argument b) above. Note that we did not have any serious bug reports (except for one open issue) since December. Once the remaining issue is solved, we plan to do a short "proof in practice" and then move on to a new stable version. In regard to new features, this release offers imfile multi-line capability, realtime UDP reception capability and better configurability for ommysql as recently announced for 6.1.3. Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-5-7-3-v5-beta/ Download: http://www.rsyslog.com/rsyslog-5-7-3-v5-beta/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From rgerhards at hq.adiscon.com Mon Feb 7 11:16:31 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Feb 2011 11:16:31 +0100 Subject: [rsyslog] New beta, version focus Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDB85@GRFEXC.intern.adiscon.com> Hi all, you'll see an announcement for a new v5-beta here on this list soon. I have written up some background about its meaning, including v6, here: http://blog.gerhards.net/2011/02/new-rsyslog-v5-beta-and-focus-on-v6.html Rainer From david at lang.hm Mon Feb 7 18:38:38 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 7 Feb 2011 09:38:38 -0800 (PST) Subject: [rsyslog] make install issue Message-ID: it appears that make install doesn't always install everything. specifically, I find that if I don't rm /usr/lib/rsyslog/* before doing a make install, the make install will not always overwrite the existing files. sometimes it does and sometimes it doesn't. I haven't figured out the exact logic yet, but I've been bit by this several times now. David Lang From gregory.fuller at oswego.edu Mon Feb 7 18:57:07 2011 From: gregory.fuller at oswego.edu (Gregory Fuller) Date: Mon, 7 Feb 2011 12:57:07 -0500 Subject: [rsyslog] ActionFileDefaultTemplate in 6.1.3 not working Message-ID: I'm trying to apply the RSYSLOG_TraditionalFileFormat as the default template using the $ActionFileDefaultTemplate directive in v6.1.3. It doesn't appear to work. I get RSYSLOG_FileFormat style logs that show up in all of my log files like this:fwp 2011-02-07T12:08:05-05:00 fwp %ASA-6-302013: Built outbound TCP connection 1285581751 for outside:x.x.x.x/80 (24.143.204.155/80) to res:x.x.x/58084 (x.x.x.x/58084) I have the default template and file where this is being logged configured like this: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat if ($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log & ~ Starting rsyslog with the -d debug option I can see it looks like looks like it isn't using the correct default format that I set in the config file: 8982.054735362:b7f886c0: cfline: '$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat' . . 8982.055999738:b7f886c0: module is incompatible with RepeatedMsgReduction - turned off 8982.056005382:b7f886c0: cfline: 'if ($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log' 8982.056009113:b7f886c0: selector line successfully processed 8982.056012836:b7f886c0: - general expression-based filter 8982.056017374:b7f886c0: skipped whitespace, stream now '($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log' 8982.056024231:b7f886c0: ctok_token 0x9e94eb0: token: 10 8982.056133884:b7f886c0: expr 0x9e94030: successfully parsed/created expression 8982.056148358:b7f886c0: file stream firewall-fwp.log params: flush interval 0, async write 0 8982.056153077:b7f886c0: tried selector action for builtin-file: 0 8982.056156714:b7f886c0: Module builtin-file processed this config line. 8982.056161564:b7f886c0: template: 'RSYSLOG_FileFormat' assigned But if I change the individual rule and specifically tell it to use the RSYSLOG_TraditionalFileFormat it works fine: if ($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log;RSYSLOG_TraditionalFileFormat & ~ 0992.733655388:b7f046c0: module is incompatible with RepeatedMsgReduction - turned off 0992.733660566:b7f046c0: cfline: 'if ($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log;RSYSLOG_TraditionalFileFormat' 0992.733664380:b7f046c0: selector line successfully processed 0992.733667985:b7f046c0: - general expression-based filter 0992.733672523:b7f046c0: skipped whitespace, stream now '($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log;RSYSLOG_TraditionalFileFormat' 0992.733676941:b7f046c0: ctok_token 0x8e24ed8: token: 10 0992.733799831:b7f046c0: tried selector action for builtin-file: 0 0992.733803366:b7f046c0: Module builtin-file processed this config line. 0992.733808082:b7f046c0: template: 'RSYSLOG_TraditionalFileFormat' assigned I stripped my config file so basically all it is doing is logging the host to the /var/log/firewall/firewall-fwp.log file for testing and I still get the same results. Any ideas? $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat if ($fromhost == 'fwp') then /var/log/firewall/firewall-fwp.log;RSYSLOG_Traditional &~ --greg From david at lang.hm Wed Feb 9 03:19:37 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Feb 2011 18:19:37 -0800 (PST) Subject: [rsyslog] Losing UDP packages In-Reply-To: <4D2AFEA0.3060603@hq.adiscon.com> References: <20101228145725.GA6780@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com> <20101229170805.GB14863@fly.srk.fer.hr> <4D2AFEA0.3060603@hq.adiscon.com> Message-ID: is there any documentation on these parameters? especially what the priority means (what is probably safe and what probably isn't type of things) David Lang On Mon, 10 Jan 2011, Rainer Gerhards wrote: > Date: Mon, 10 Jan 2011 13:42:08 +0100 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Losing UDP packages > > Hi, > > thanks for all your hard work! I have replaced my initial implementation > with your patch. It is available starting with 5.7.3 (as it changes > quite a bit, it can not go immediately into the stable version). I have > removed support from v4, because the changes required are too large to > justify supporting it in v4. > > So far, I have just done code review and very rough testing. I will let > the new code run in my lab within the next couple of days, but I thought > it is such a good addition that I merged it ASAP. > > Thanks again for your help! > > Rainer > > > On 12/29/2010 06:08 PM, Dra?en Ka?ar wrote: >> Rainer Gerhards wrote: >>> I am on vacation right now. But I think what happens is that the worker >>> threads inherit the priority setting from the UDP listener thread. You >>> probably need to change thread creation in ./runtime/wtp.c. >> >> I hoped there would be a better method. Anyway, I've added thread >> attributes in every pthread_create call, since changing just the one in >> wtp.c wasn't enough. >> >> There is one pthread_create() in plugins/imsolaris/sun_cddl.c which I >> didn't touch because it seems buggy. It's using create_door_thr as >> pthread_attr_t, but create_door_thr is never initialized, as far as I can >> see. >> >> The updated patch against rsyslog 5.6.2 is attached. I have only UDP >> thread in real-time mode now. >> >> I'm not sure if the code which gets the default thread properties should >> go in rsyslog.c or somewhere else. It can be safely moved anywhere in the >> initialization sequence, before the first pthread_create is called. >> >> About configure check: the proper way to check for the functionality would >> be to check for _XOPEN_REALTIME_THREADS preprocessor macro. That's what's >> supposed to be defined if real-time thread functionality is available. >> However, there is no that symbol anywhere in /usr/include on Solaris 10 >> (update 6 is what I checked). >> >> The equivalent run-time check sysconf(_SC_XOPEN_REALTIME_THREADS) is >> returning 1, though, so the lack of _XOPEN_REALTIME_THREADS macro is a >> Solaris bug, as far as I can tell. >> >> Therefore I'm checking for the availability of pthread_setschedparam() and >> then have all real-time thread code in #ifdef HAVE_PTHREAD_SETSCHEDPARAM >> blocks. It's not ideal, but I hope it works. >> >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Feb 9 07:28:37 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Feb 2011 07:28:37 +0100 Subject: [rsyslog] Losing UDP packages References: <20101228145725.GA6780@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com><20101229170805.GB14863@fly.srk.fer.hr><4D2AFEA0.3060603@hq.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBCA@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, February 09, 2011 3:20 AM > To: rsyslog-users > Subject: Re: [rsyslog] Losing UDP packages > > is there any documentation on these parameters? especially what the > priority means (what is probably safe and what probably isn't type of > things) Some doc was added. But the meaning is system-specific, there is no standard on that. You need to lookup the scheduling params of your os (I think there are hints inside the pthreads man pages). Rainer > David Lang > > On Mon, 10 Jan 2011, Rainer Gerhards wrote: > > > Date: Mon, 10 Jan 2011 13:42:08 +0100 > > From: Rainer Gerhards > > Reply-To: rsyslog-users > > To: rsyslog-users > > Subject: Re: [rsyslog] Losing UDP packages > > > > Hi, > > > > thanks for all your hard work! I have replaced my initial > implementation > > with your patch. It is available starting with 5.7.3 (as it changes > > quite a bit, it can not go immediately into the stable version). I > have > > removed support from v4, because the changes required are too large > to > > justify supporting it in v4. > > > > So far, I have just done code review and very rough testing. I will > let > > the new code run in my lab within the next couple of days, but I > thought > > it is such a good addition that I merged it ASAP. > > > > Thanks again for your help! > > > > Rainer > > > > > > On 12/29/2010 06:08 PM, Dra?en Ka?ar wrote: > >> Rainer Gerhards wrote: > >>> I am on vacation right now. But I think what happens is that the > worker > >>> threads inherit the priority setting from the UDP listener thread. > You > >>> probably need to change thread creation in ./runtime/wtp.c. > >> > >> I hoped there would be a better method. Anyway, I've added thread > >> attributes in every pthread_create call, since changing just the one > in > >> wtp.c wasn't enough. > >> > >> There is one pthread_create() in plugins/imsolaris/sun_cddl.c which > I > >> didn't touch because it seems buggy. It's using create_door_thr as > >> pthread_attr_t, but create_door_thr is never initialized, as far as > I can > >> see. > >> > >> The updated patch against rsyslog 5.6.2 is attached. I have only UDP > >> thread in real-time mode now. > >> > >> I'm not sure if the code which gets the default thread properties > should > >> go in rsyslog.c or somewhere else. It can be safely moved anywhere > in the > >> initialization sequence, before the first pthread_create is called. > >> > >> About configure check: the proper way to check for the functionality > would > >> be to check for _XOPEN_REALTIME_THREADS preprocessor macro. That's > what's > >> supposed to be defined if real-time thread functionality is > available. > >> However, there is no that symbol anywhere in /usr/include on Solaris > 10 > >> (update 6 is what I checked). > >> > >> The equivalent run-time check sysconf(_SC_XOPEN_REALTIME_THREADS) is > >> returning 1, though, so the lack of _XOPEN_REALTIME_THREADS macro is > a > >> Solaris bug, as far as I can tell. > >> > >> Therefore I'm checking for the availability of > pthread_setschedparam() and > >> then have all real-time thread code in #ifdef > HAVE_PTHREAD_SETSCHEDPARAM > >> blocks. It's not ideal, but I hope it works. > >> > >> > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dave at fly.srk.fer.hr Wed Feb 9 16:02:37 2011 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 9 Feb 2011 16:02:37 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBCA@GRFEXC.intern.adiscon.com> References: <20101228145725.GA6780@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com> <20101229170805.GB14863@fly.srk.fer.hr> <4D2AFEA0.3060603@hq.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBCA@GRFEXC.intern.adiscon.com> Message-ID: <20110209150237.GA31760@fly.srk.fer.hr> Rainer Gerhards wrote: > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Wednesday, February 09, 2011 3:20 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Losing UDP packages > > > > is there any documentation on these parameters? especially what the > > priority means (what is probably safe and what probably isn't type of > > things) > > Some doc was added. But the meaning is system-specific, there is no standard > on that. You need to lookup the scheduling params of your os (I think there > are hints inside the pthreads man pages). You can try putting a crazy number (like 10000) and you'll get an error message from rsyslog stating the allowed range. However, you shouldn't use this option if you don't know what you're doing. If rsyslog is the only program in the FIFO class, then the priority you put doesn't matter. It only comes into play if there are several processes (or threads) in FIFO class. But then you need to know what they all are and what priority they all have, in order to prevent possible starvation problems. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From mark at thermeon.com Wed Feb 9 17:37:03 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 9 Feb 2011 16:37:03 +0000 Subject: [rsyslog] External Script Message-ID: Hi, I have the following section in my conf file, where it should log all file error to a file, then email me once every 60 seconds if more are ariving. I also want it to SMS me every 600 seconds for the SMS to work i need to give it a few arguments, how can i do that using rsyslog? So far my config looks like this: if $programname == 'FILLER' then /var/log/application/filler.log $ModLoad ommail $ActionMailSMTPServer localhost $ActionMailFrom rsyslog at example.com $ActionMailTo mpo at example.com $template fillerSubject,"FILE ERROR %HOSTNAME% %timestamp%" $template mailBody,"%msg%" $ActionMailSubject fillerSubject $ActionExecOnlyOnceEveryInterval 10 if $programname == 'FILLER' then :ommail:;mailBody if $programname == 'FILLER' /usr/bin/alert & ~ For the SMS to work /usr/bin/alert needs the following arguments STATUS Message Phone_Number Ideally Message will be %msg%, phone number will be fixed and STATUS would be the priority level. Any help will be gratefully received. Thanks Mark From mark at thermeon.com Wed Feb 9 18:36:50 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 9 Feb 2011 17:36:50 +0000 Subject: [rsyslog] OMPROG Doccumentation Message-ID: Hi, Is there any documentation on omprog as i can not find anything other than a few lines in the mailing list and my C source reading is not great. So far the following works to trigger my script but is there a way to send the contents of the msg and or any custom arguments that my script might want? $ModLoad omprog $actionomprogbinary /usr/bin/alert1 if $programname == 'FILLER' then :omprog: Thanks Matk From david at lang.hm Wed Feb 9 19:59:17 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 9 Feb 2011 10:59:17 -0800 (PST) Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: Message-ID: On Wed, 9 Feb 2011, Mark Olliver wrote: > Hi, > > Is there any documentation on omprog as i can not find anything other > than a few lines in the mailing list and my C source reading is not > great. > > So far the following works to trigger my script but is there a way to > send the contents of the msg and or any custom arguments that my > script might want? > $ModLoad omprog > $actionomprogbinary /usr/bin/alert1 > if $programname == 'FILLER' then :omprog: does your script check both it's stdin and command-line parameters? I would expect that the log line would be passed through one of those (and could then be modified with a ;format_name added to the line that calls the script) I haven't worked with omfile yet myself. a second option that you have (especially if you want the script to send one mail for several log messages), is to have rsyslog write to a named pipe, and then have your alert script read from that pipe (so it can see multiple messages) and send out the mail from within the script instead of sending out one mail per script invocation. David Lang From norman.zhang at gmail.com Thu Feb 10 01:44:17 2011 From: norman.zhang at gmail.com (Norman Zhang) Date: Wed, 09 Feb 2011 17:44:17 -0700 Subject: [rsyslog] Log Verbosity Message-ID: I have applied the following template $template precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n" if ($syslogseverity <= '6') then ?precise However my log still shows the default format. 2011-02-09T19:38:10.279495-05:00 10.131.47.4 : 2011 Feb 9 19:38:10 EST: %RADIUS-3-RADIUS_ERROR_MESSAGE: All RADIUS servers failed to respond after retries. 2011-02-09T19:38:15.249808-05:00 10.131.47.4 : 2011 Feb 9 19:38:15 EST: %TACACS-3-TACACS_ERROR_MESSAGE: All servers failed to respond Can someone please show me how I can achieve "precise"? Norman From david at lang.hm Thu Feb 10 02:24:19 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 9 Feb 2011 17:24:19 -0800 (PST) Subject: [rsyslog] Log Verbosity In-Reply-To: References: Message-ID: On Wed, 9 Feb 2011, Norman Zhang wrote: > I have applied the following template > > $template > precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n" > > if ($syslogseverity <= '6') then ?precise > > However my log still shows the default format. > > 2011-02-09T19:38:10.279495-05:00 10.131.47.4 : 2011 Feb 9 19:38:10 EST: > %RADIUS-3-RADIUS_ERROR_MESSAGE: All RADIUS servers failed to respond after > retries. > 2011-02-09T19:38:15.249808-05:00 10.131.47.4 : 2011 Feb 9 19:38:15 EST: > %TACACS-3-TACACS_ERROR_MESSAGE: All servers failed to respond > > Can someone please show me how I can achieve "precise"? *.* /var/log/messages;precise If I understand what you have defined, you've defined it to log to a file named (result of template evaluation of precise), not logging to a file using precise as the format for the log lines. David Lang From norman.zhang at gmail.com Thu Feb 10 05:59:24 2011 From: norman.zhang at gmail.com (Norman Zhang) Date: Wed, 09 Feb 2011 21:59:24 -0700 Subject: [rsyslog] Invalid Dynamic Filename Message-ID: I configured Dynamic file $template DynFile,"/test/log/%HOSTNAME%-%YEAR%-%MONTH%-%DAY%.log" :source , !isequal , "localhost" ?DynFile However, I get invalid property in the filename. [root at box ~]# ls -l /test/log/ total 4 -rw------- 1 root root 146 Feb 9 23:51 10.131.47.4-**INVALID PROPERTY NAME**-**INVALID PROPERTY NAME**-**INVALID PROPERTY NAME**.log 1. Am I missing something? 2. Can I compare two different strings with !isequal? Norman From timo.veith at gmail.com Thu Feb 10 10:29:41 2011 From: timo.veith at gmail.com (Timo Veith) Date: Thu, 10 Feb 2011 10:29:41 +0100 Subject: [rsyslog] filtering programname with slashes problem? Message-ID: Hello rsyslog list readers, I have found that I can not use a filter with the programname property when the string contains a slash. Has somebody made the same experience? My setup is a debian squeeze box, with almost default settings and I want to have a separate log for policyd-weight messages. Here is a example line out of mail.log: Feb 10 09:21:32 mx1 postfix/policyd-weight[3161]: weighted check: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 HELO_IP_IN_CL_SUBNET=-1.2 (check from: .rediffmail. - helo: .rediffmail. - helo-domain: .rediffmail.) FROM/MX_MATCHES_HELO(DOMAIN)=-2 IN_ABUSE_RFCI=0.1; ; rate: -7.6 I tried these filter statements: if $programname contains 'postfix/policyd-weight' and \ $msg contains 'decided action=550' then -/var/log/spam/policyd-weight.log if $programname contains 'policyd-weight' and \ $msg contains 'decided action=550' then -/var/log/spam/policyd-weight.log if $programname contains 'policyd' then -/var/log/spam/policyd-weight_debug.log :programname, regex, ".*policyd.*" -/var/log/spam/policyd-weight_debug.log None of them work. This one works: if $msg contains 'decided action=' or \ $msg contains 'weighted check' then -/var/log/spam/policyd-weight_debug.log But it doesn't catch all policyd-weight messages. Am I going wrong somewhere? Any ideas? Kind regards, Timo From rgerhards at hq.adiscon.com Thu Feb 10 10:31:47 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 10:31:47 +0100 Subject: [rsyslog] filtering programname with slashes problem? In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com> I think that RFC3164 tells that a slash terminates the program name (or something along these lines). You may want to check not for theprogram name, but for the actual tag value (which starts with the program name). I guess that will solve the issue. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Timo Veith > Sent: Thursday, February 10, 2011 10:30 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] filtering programname with slashes problem? > > Hello rsyslog list readers, > > I have found that I can not use a filter with the programname property > when the string contains a slash. Has somebody made the same > experience? > > My setup is a debian squeeze box, with almost default settings and I > want to have a separate log for policyd-weight messages. > Here is a example line out of mail.log: > > Feb 10 09:21:32 mx1 postfix/policyd-weight[3161]: weighted check: > NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 > HELO_IP_IN_CL_SUBNET=-1.2 (check from: .rediffmail. - helo: > .rediffmail. - helo-domain: .rediffmail.) > FROM/MX_MATCHES_HELO(DOMAIN)=-2 IN_ABUSE_RFCI=0.1; > > ; rate: > -7.6 > > I tried these filter statements: > > if $programname contains 'postfix/policyd-weight' and \ > $msg contains 'decided action=550' then -/var/log/spam/policyd- > weight.log > > if $programname contains 'policyd-weight' and \ > $msg contains 'decided action=550' then -/var/log/spam/policyd- > weight.log > > if $programname contains 'policyd' then -/var/log/spam/policyd- > weight_debug.log > > :programname, regex, ".*policyd.*" -/var/log/spam/policyd- > weight_debug.log > > None of them work. > > This one works: > > if $msg contains 'decided action=' or \ > $msg contains 'weighted check' then -/var/log/spam/policyd- > weight_debug.log > > But it doesn't catch all policyd-weight messages. > > Am I going wrong somewhere? Any ideas? > > Kind regards, > Timo > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 10 10:32:59 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 10:32:59 +0100 Subject: [rsyslog] Invalid Dynamic Filename In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBDA@GRFEXC.intern.adiscon.com> You need to check the exact field syntax. At least a $-Sign is missing in front of YEAR and those (but I am not 100% sure if they are fully correct as given). To compare two strings, you need to use script-based filters. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Norman Zhang > Sent: Thursday, February 10, 2011 5:59 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Invalid Dynamic Filename > > I configured Dynamic file > > $template DynFile,"/test/log/%HOSTNAME%-%YEAR%-%MONTH%-%DAY%.log" > :source , !isequal , "localhost" ?DynFile > > However, I get invalid property in the filename. > > [root at box ~]# ls -l /test/log/ > total 4 > -rw------- 1 root root 146 Feb 9 23:51 10.131.47.4-**INVALID PROPERTY > NAME**-**INVALID PROPERTY NAME**-**INVALID PROPERTY NAME**.log > > 1. Am I missing something? > 2. Can I compare two different strings with !isequal? > > Norman > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From timo.veith at gmail.com Thu Feb 10 12:26:25 2011 From: timo.veith at gmail.com (Timo Veith) Date: Thu, 10 Feb 2011 12:26:25 +0100 Subject: [rsyslog] filtering programname with slashes problem? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com> Message-ID: Hello Rainer, thank you for your super fast reply! Your suggestion brought me a little further. If I search within $msg for "policyd-weight" then it works. It also works, if I use $rawmsg. You stated in your reply that the slash character separates the TAG from the CONTENT at the wrong position, right? I.e. "postfix" is the TAG and the rest "/policy-weightd[...." becomes the CONTENT ? I tried to test that: The filters if $msg startswith '/policyd-weight' then -/var/log/spam/policyd-weight_debug.log if $msg startswith 'policyd-weight' then -/var/log/spam/policyd-weight_debug.log don't work. I hope I didn't miss a service restart command, I am testing a lot... Maybe the $msg also contains $programname? I tried: if $msg startswith 'postfix/policyd-weight' then -/var/log/spam/policyd-weight_debug.log if $msg contains 'postfix/policyd-weight' then -/var/log/spam/policyd-weight_debug.log Both don't work either. I am little confused now. I could just use the one filter from above which is working, but I would like to understand why and how it is working or why not respectivley. Thank you and kind regards, Timo From rgerhards at hq.adiscon.com Thu Feb 10 12:29:43 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 12:29:43 +0100 Subject: [rsyslog] filtering programname with slashes problem? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBE7@GRFEXC.intern.adiscon.com> timo, more in depth later, but maybe this helps: fields stem back to RFC3164/5424: so the tag is NOT part of msg! tag is the full beast "blub[4711]" whereas programname is just the process name "blub". I think if you use syslogtag (check property replacer doc if that is the right name!) instead of programname should work. The appendix in RFC5424 (I think A) has information on the mapping of legacy tags to programname. HTH rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Timo Veith > Sent: Thursday, February 10, 2011 12:26 PM > To: rsyslog-users > Subject: Re: [rsyslog] filtering programname with slashes problem? > > Hello Rainer, > > thank you for your super fast reply! > > Your suggestion brought me a little further. If I search within $msg > for "policyd-weight" then it works. It also works, if I use $rawmsg. > > You stated in your reply that the slash character separates the TAG > from the CONTENT at the wrong position, right? > > I.e. "postfix" is the TAG and the rest "/policy-weightd[...." becomes > the CONTENT ? > > I tried to test that: > > The filters > > if $msg startswith '/policyd-weight' then > -/var/log/spam/policyd-weight_debug.log > if $msg startswith 'policyd-weight' then -/var/log/spam/policyd- > weight_debug.log > > don't work. I hope I didn't miss a service restart command, I am > testing a lot... > > Maybe the $msg also contains $programname? > I tried: > > if $msg startswith 'postfix/policyd-weight' then > -/var/log/spam/policyd-weight_debug.log > if $msg contains 'postfix/policyd-weight' then > -/var/log/spam/policyd-weight_debug.log > > Both don't work either. > > I am little confused now. I could just use the one filter from above > which is working, but I would like to understand why and how it is > working or why not respectivley. > > Thank you and kind regards, > Timo > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Thu Feb 10 12:37:59 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 11:37:59 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: Message-ID: Hi, I have been playing more with OMPROG and have it working nearly the way i want now in that it it now passes the correct message to my external script by the pipe however, OMPROG seems to only flush the data out to the pipe when rsyslog is shutdown, until that point it just keeps the pipe/script open. Is there a way to force OMPROG to flush each line to the pipe and then close the pipe or is there a better way to do this? So far i have: $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" $ModLoad omprog $actionomprogbinary /usr/bin/alert1 if $programname == 'FILLER' then :omprog:;smsBody & ~ Thanks Mark From rgerhards at hq.adiscon.com Thu Feb 10 12:40:19 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 12:40:19 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 12:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > Hi, > > I have been playing more with OMPROG and have it working nearly the > way i want now in that it it now passes the correct message to my > external script by the pipe however, OMPROG seems to only flush the > data out to the pipe when rsyslog is shutdown, until that point it > just keeps the pipe/script open. You mean no data is sent until shutdown? I'll try to have a look this afternoon... Rainer Is there a way to force OMPROG to > flush each line to the pipe and then close the pipe or is there a > better way to do this? > > So far i have: > $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" > $ModLoad omprog > $actionomprogbinary /usr/bin/alert1 > if $programname == 'FILLER' then :omprog:;smsBody > & ~ > > Thanks > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Thu Feb 10 12:45:32 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 11:45:32 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> Message-ID: Hi Rainer Yep that is what it appears to be, I have a Ruby script listening which is executed by omprog, which is set to send an sms with the error message however, as the moment i have the script set to dump to a file what it reads from STDIN. Only when rsyslog is shutdown does the file get populated with data but as soon as the log message is sent to the log the script is hit and the file opened. So it would appear the omprog is not closing its handle or not flushing the data until rsyslog closes. I am currently using rsyslog 4.6.4 (backported with omprog manually builtin) on ubuntu 10.04 Thanks for you help Mark On 10 February 2011 11:40, Rainer Gerhards wrote: > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> Sent: Thursday, February 10, 2011 12:38 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> Hi, >> >> I have been playing more with OMPROG and have it working nearly the >> way i want now in that it it now passes the correct message to my >> external script by the pipe however, OMPROG seems to only flush the >> data out to the pipe when rsyslog is shutdown, until that point it >> just keeps the pipe/script open. > > You mean no data is sent until shutdown? I'll try to have a look this > afternoon... > > Rainer > ?Is there a way to force OMPROG to >> flush each line to the pipe and then close the pipe or is there a >> better way to do this? > >> >> So far i have: >> $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" >> $ModLoad omprog >> $actionomprogbinary /usr/bin/alert1 >> if $programname == 'FILLER' then :omprog:;smsBody >> & ~ >> >> Thanks >> >> Mark >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From timo.veith at gmail.com Thu Feb 10 13:14:16 2011 From: timo.veith at gmail.com (Timo Veith) Date: Thu, 10 Feb 2011 13:14:16 +0100 Subject: [rsyslog] filtering programname with slashes problem? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBE7@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBE7@GRFEXC.intern.adiscon.com> Message-ID: Rainer, again thank you for the fast reply! Great support ! :) I am using $syslogtag now and it works as expected. Let me explain why I came to the thinking from above: I 've read a little of the rfc which you mentioned in your first reply, And from the section "4.1.3 MSG Part of a syslog Packet", second paragraph I thought that the slash separates at the wrong postition. Here is a cite from the section: "The MSG part has two fields known as the TAG field and the CONTENT field. The value in the TAG field will be the name of the program or process that generated the message. The CONTENT contains the details of the message. .... Any non-alphanumeric character will terminate the TAG field and will be assumed to be the starting character of the CONTENT field." So I thought the slash terminated the string name too early. However, as I already said, with $syslogtag it works now. Thanks again, Timo From rgerhards at hq.adiscon.com Thu Feb 10 13:32:32 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 13:32:32 +0100 Subject: [rsyslog] filtering programname with slashes problem? In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBD9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBE7@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBE9@GRFEXC.intern.adiscon.com> yeah, it's a bit complicated, unfortunately. RFC3164 does not cover everything, it was merely an effort to "document what was usually seen". We tried to make things much more clear in 5424, though (but I fear it is still not perfect in regard to existing implementations). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Timo Veith > Sent: Thursday, February 10, 2011 1:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] filtering programname with slashes problem? > > Rainer, > > again thank you for the fast reply! Great support ! :) > > I am using $syslogtag now and it works as expected. > > Let me explain why I came to the thinking from above: > I 've read a little of the rfc which you mentioned in your first > reply, And from the section "4.1.3 MSG Part of a syslog Packet", > second paragraph I thought that the slash separates at the wrong > postition. Here is a cite from the section: > > "The MSG part has two fields known as the TAG field and the CONTENT > field. The value in the TAG field will be the name of the program or > process that generated the message. The CONTENT contains the details > of the message. .... Any non-alphanumeric character will terminate > the TAG > field and will be assumed to be the starting character of the CONTENT > field." > > So I thought the slash terminated the string name too early. > > However, as I already said, with $syslogtag it works now. > > Thanks again, > Timo > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 10 13:44:00 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 13:44:00 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> I have now looked at the code. I can clearly see that a write to the pipe fd is done for every message. Could you provide a debug log? That would be useful. Google for "rsyslog debug", that should bring up the relevant page (usually at position 1). Raienr > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 12:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > Hi Rainer > > Yep that is what it appears to be, I have a Ruby script listening > which is executed by omprog, which is set to send an sms with the > error message however, as the moment i have the script set to dump to > a file what it reads from STDIN. Only when rsyslog is shutdown does > the file get populated with data but as soon as the log message is > sent to the log the script is hit and the file opened. So it would > appear the omprog is not closing its handle or not flushing the data > until rsyslog closes. > > I am currently using rsyslog 4.6.4 (backported with omprog manually > builtin) on ubuntu 10.04 > > Thanks for you help > > Mark > > > On 10 February 2011 11:40, Rainer Gerhards > wrote: > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> Sent: Thursday, February 10, 2011 12:38 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> > >> Hi, > >> > >> I have been playing more with OMPROG and have it working nearly the > >> way i want now in that it it now passes the correct message to my > >> external script by the pipe however, OMPROG seems to only flush the > >> data out to the pipe when rsyslog is shutdown, until that point it > >> just keeps the pipe/script open. > > > > You mean no data is sent until shutdown? I'll try to have a look this > > afternoon... > > > > Rainer > > ?Is there a way to force OMPROG to > >> flush each line to the pipe and then close the pipe or is there a > >> better way to do this? > > > >> > >> So far i have: > >> $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" > >> $ModLoad omprog > >> $actionomprogbinary /usr/bin/alert1 > >> if $programname == 'FILLER' then :omprog:;smsBody > >> & ~ > >> > >> Thanks > >> > >> Mark > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 10 14:06:01 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 14:06:01 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> mhhh... I can see that omprog is called, and everything looks like it should be. But that could be absence of debug information. I could probably add some more instrumentation so that we get more insight. Could you build it from a git tree? > -----Original Message----- > From: Mark Olliver [mailto:mark at thermeon.com] > Sent: Thursday, February 10, 2011 1:53 PM > To: Rainer Gerhards > Subject: Re: [rsyslog] OMPROG Doccumentation > > Hi Rainer, > > Here is the logfile as requested. > > Thanks > > Mark > > On 10 February 2011 12:44, Rainer Gerhards > wrote: > > I have now looked at the code. I can clearly see that a write to the > pipe fd > > is done for every message. Could you provide a debug log? That would > be > > useful. Google for "rsyslog debug", that should bring up the relevant > page > > (usually at position 1). > > > > Raienr > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> Sent: Thursday, February 10, 2011 12:46 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> > >> Hi Rainer > >> > >> Yep that is what it appears to be, I have a Ruby script listening > >> which is executed by omprog, which is set to send an sms with the > >> error message however, as the moment i have the script set to dump > to > >> a file what it reads from STDIN. Only when rsyslog is shutdown does > >> the file get populated with data but as soon as the log message is > >> sent to the log the script is hit and the file opened. So it would > >> appear the omprog is not closing its handle or not flushing the data > >> until rsyslog closes. > >> > >> I am currently using rsyslog 4.6.4 (backported with omprog manually > >> builtin) on ubuntu 10.04 > >> > >> Thanks for you help > >> > >> Mark > >> > >> > >> On 10 February 2011 11:40, Rainer Gerhards > > >> wrote: > >> > > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> >> Sent: Thursday, February 10, 2011 12:38 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> >> > >> >> Hi, > >> >> > >> >> I have been playing more with OMPROG and have it working nearly > the > >> >> way i want now in that it it now passes the correct message to my > >> >> external script by the pipe however, OMPROG seems to only flush > the > >> >> data out to the pipe when rsyslog is shutdown, until that point > it > >> >> just keeps the pipe/script open. > >> > > >> > You mean no data is sent until shutdown? I'll try to have a look > this > >> > afternoon... > >> > > >> > Rainer > >> > ?Is there a way to force OMPROG to > >> >> flush each line to the pipe and then close the pipe or is there a > >> >> better way to do this? > >> > > >> >> > >> >> So far i have: > >> >> $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" > >> >> $ModLoad omprog > >> >> $actionomprogbinary /usr/bin/alert1 > >> >> if $programname == 'FILLER' then :omprog:;smsBody > >> >> & ~ > >> >> > >> >> Thanks > >> >> > >> >> Mark > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From mark at thermeon.com Thu Feb 10 14:13:16 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 13:13:16 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> Message-ID: Hi, Yes I should be able to build from a GIT tree ok. Regards Mark On 10 February 2011 13:06, Rainer Gerhards wrote: > mhhh... I can see that omprog is called, and everything looks like it should > be. But that could be absence of debug information. I could probably add some > more instrumentation so that we get more insight. Could you build it from a > git tree? > >> -----Original Message----- >> From: Mark Olliver [mailto:mark at thermeon.com] >> Sent: Thursday, February 10, 2011 1:53 PM >> To: Rainer Gerhards >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> Hi Rainer, >> >> Here is the logfile as requested. >> >> Thanks >> >> Mark >> >> On 10 February 2011 12:44, Rainer Gerhards >> wrote: >> > I have now looked at the code. I can clearly see that a write to the >> pipe fd >> > is done for every message. Could you provide a debug log? That would >> be >> > useful. Google for "rsyslog debug", that should bring up the relevant >> page >> > (usually at position 1). >> > >> > Raienr >> > >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> >> Sent: Thursday, February 10, 2011 12:46 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> >> >> Hi Rainer >> >> >> >> Yep that is what it appears to be, I have a Ruby script listening >> >> which is executed by omprog, which is set to send an sms with the >> >> error message however, as the moment i have the script set to dump >> to >> >> a file what it reads from STDIN. Only when rsyslog is shutdown does >> >> the file get populated with data but as soon as the log message is >> >> sent to the log the script is hit and the file opened. So it would >> >> appear the omprog is not closing its handle or not flushing the data >> >> until rsyslog closes. >> >> >> >> I am currently using rsyslog 4.6.4 (backported with omprog manually >> >> builtin) on ubuntu 10.04 >> >> >> >> Thanks for you help >> >> >> >> Mark >> >> >> >> >> >> On 10 February 2011 11:40, Rainer Gerhards >> >> >> wrote: >> >> > >> >> >> -----Original Message----- >> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> >> >> Sent: Thursday, February 10, 2011 12:38 PM >> >> >> To: rsyslog-users >> >> >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> >> >> >> >> Hi, >> >> >> >> >> >> I have been playing more with OMPROG and have it working nearly >> the >> >> >> way i want now in that it it now passes the correct message to my >> >> >> external script by the pipe however, OMPROG seems to only flush >> the >> >> >> data out to the pipe when rsyslog is shutdown, until that point >> it >> >> >> just keeps the pipe/script open. >> >> > >> >> > You mean no data is sent until shutdown? I'll try to have a look >> this >> >> > afternoon... >> >> > >> >> > Rainer >> >> > ?Is there a way to force OMPROG to >> >> >> flush each line to the pipe and then close the pipe or is there a >> >> >> better way to do this? >> >> > >> >> >> >> >> >> So far i have: >> >> >> $template smsBody,"%programname% 44788900000 %hostname% %rawmsg%" >> >> >> $ModLoad omprog >> >> >> $actionomprogbinary /usr/bin/alert1 >> >> >> if $programname == 'FILLER' then :omprog:;smsBody >> >> >> & ~ >> >> >> >> >> >> Thanks >> >> >> >> >> >> Mark >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com >> >> > >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > > From rgerhards at hq.adiscon.com Thu Feb 10 14:57:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 14:57:51 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> I just pushed a new branch v4.6.4-omprogTest to rsyslog's public git. It has the additional instrumentation and should provide more insight. Rainer > -----Original Message----- > From: Mark Olliver [mailto:mark at thermeon.com] > Sent: Thursday, February 10, 2011 2:13 PM > To: Rainer Gerhards > Cc: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > Hi, > > Yes I should be able to build from a GIT tree ok. > > Regards > > Mark > > On 10 February 2011 13:06, Rainer Gerhards > wrote: > > mhhh... I can see that omprog is called, and everything looks like it > should > > be. But that could be absence of debug information. I could probably > add some > > more instrumentation so that we get more insight. Could you build it > from a > > git tree? > > > >> -----Original Message----- > >> From: Mark Olliver [mailto:mark at thermeon.com] > >> Sent: Thursday, February 10, 2011 1:53 PM > >> To: Rainer Gerhards > >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> > >> Hi Rainer, > >> > >> Here is the logfile as requested. > >> > >> Thanks > >> > >> Mark > >> > >> On 10 February 2011 12:44, Rainer Gerhards > > >> wrote: > >> > I have now looked at the code. I can clearly see that a write to > the > >> pipe fd > >> > is done for every message. Could you provide a debug log? That > would > >> be > >> > useful. Google for "rsyslog debug", that should bring up the > relevant > >> page > >> > (usually at position 1). > >> > > >> > Raienr > >> > > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> >> Sent: Thursday, February 10, 2011 12:46 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> >> > >> >> Hi Rainer > >> >> > >> >> Yep that is what it appears to be, I have a Ruby script listening > >> >> which is executed by omprog, which is set to send an sms with the > >> >> error message however, as the moment i have the script set to > dump > >> to > >> >> a file what it reads from STDIN. Only when rsyslog is shutdown > does > >> >> the file get populated with data but as soon as the log message > is > >> >> sent to the log the script is hit and the file opened. So it > would > >> >> appear the omprog is not closing its handle or not flushing the > data > >> >> until rsyslog closes. > >> >> > >> >> I am currently using rsyslog 4.6.4 (backported with omprog > manually > >> >> builtin) on ubuntu 10.04 > >> >> > >> >> Thanks for you help > >> >> > >> >> Mark > >> >> > >> >> > >> >> On 10 February 2011 11:40, Rainer Gerhards > >> > >> >> wrote: > >> >> > > >> >> >> -----Original Message----- > >> >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> >> >> Sent: Thursday, February 10, 2011 12:38 PM > >> >> >> To: rsyslog-users > >> >> >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> I have been playing more with OMPROG and have it working > nearly > >> the > >> >> >> way i want now in that it it now passes the correct message to > my > >> >> >> external script by the pipe however, OMPROG seems to only > flush > >> the > >> >> >> data out to the pipe when rsyslog is shutdown, until that > point > >> it > >> >> >> just keeps the pipe/script open. > >> >> > > >> >> > You mean no data is sent until shutdown? I'll try to have a > look > >> this > >> >> > afternoon... > >> >> > > >> >> > Rainer > >> >> > ?Is there a way to force OMPROG to > >> >> >> flush each line to the pipe and then close the pipe or is > there a > >> >> >> better way to do this? > >> >> > > >> >> >> > >> >> >> So far i have: > >> >> >> $template smsBody,"%programname% 44788900000 %hostname% > %rawmsg%" > >> >> >> $ModLoad omprog > >> >> >> $actionomprogbinary /usr/bin/alert1 > >> >> >> if $programname == 'FILLER' then :omprog:;smsBody > >> >> >> & ~ > >> >> >> > >> >> >> Thanks > >> >> >> > >> >> >> Mark > >> >> >> _______________________________________________ > >> >> >> rsyslog mailing list > >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> http://www.rsyslog.com > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com > >> >> > > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > > From mark at thermeon.com Thu Feb 10 15:08:28 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 14:08:28 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> Message-ID: Hi, I am not sure on how to build that as there is no configure script, how should i generate this from the git checkout? Thanks Mark From rgerhards at hq.adiscon.com Thu Feb 10 15:09:23 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 15:09:23 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> http://www.rsyslog.com/doc/build_from_repo.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 3:08 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > Hi, > > I am not sure on how to build that as there is no configure script, > how should i generate this from the git checkout? > > Thanks > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Thu Feb 10 15:14:45 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 14:14:45 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> Message-ID: mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi Can't exec "libtoolize": No such file or directory at /usr/bin/autoreconf line 189. Use of uninitialized value in pattern match (m//) at /usr/bin/autoreconf line 189. autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal --force -I m4 autoreconf: configure.ac: tracing autoreconf: configure.ac: not using Libtool autoreconf: running: /usr/bin/autoconf --force configure.ac:25: error: possibly undefined macro: AC_DISABLE_STATIC If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf: /usr/bin/autoconf failed with exit status: 1 On 10 February 2011 14:09, Rainer Gerhards wrote: > http://www.rsyslog.com/doc/build_from_repo.html > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> Sent: Thursday, February 10, 2011 3:08 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> Hi, >> >> I am not sure on how to build that as there is no configure script, >> how should i generate this from the git checkout? >> >> Thanks >> >> Mark >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Feb 10 15:16:25 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 15:16:25 +0100 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> libtool is missing, probably more. let me see that I upload a tarball (but that will take a short while, I am absent from the system now). > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 3:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi > Can't exec "libtoolize": No such file or directory at > /usr/bin/autoreconf line 189. > Use of uninitialized value in pattern match (m//) at > /usr/bin/autoreconf line 189. > autoreconf: Entering directory `.' > autoreconf: configure.ac: not using Gettext > autoreconf: running: aclocal --force -I m4 > autoreconf: configure.ac: tracing > autoreconf: configure.ac: not using Libtool > autoreconf: running: /usr/bin/autoconf --force > configure.ac:25: error: possibly undefined macro: AC_DISABLE_STATIC > If this token and others are legitimate, please use > m4_pattern_allow. > See the Autoconf documentation. > configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL > autoreconf: /usr/bin/autoconf failed with exit status: 1 > > > On 10 February 2011 14:09, Rainer Gerhards > wrote: > > http://www.rsyslog.com/doc/build_from_repo.html > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> Sent: Thursday, February 10, 2011 3:08 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> > >> Hi, > >> > >> I am not sure on how to build that as there is no configure script, > >> how should i generate this from the git checkout? > >> > >> Thanks > >> > >> Mark > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Thu Feb 10 15:20:37 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 14:20:37 +0000 Subject: [rsyslog] OMPROG Doccumentation In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> Message-ID: Thanks Libtool fixed the build i will send the new log shortly. Regards On 10 February 2011 14:16, Rainer Gerhards wrote: > libtool is missing, probably more. let me see that I upload a tarball (but > that will take a short while, I am absent from the system now). > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> Sent: Thursday, February 10, 2011 3:15 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi >> Can't exec "libtoolize": No such file or directory at >> /usr/bin/autoreconf line 189. >> Use of uninitialized value in pattern match (m//) at >> /usr/bin/autoreconf line 189. >> autoreconf: Entering directory `.' >> autoreconf: configure.ac: not using Gettext >> autoreconf: running: aclocal --force -I m4 >> autoreconf: configure.ac: tracing >> autoreconf: configure.ac: not using Libtool >> autoreconf: running: /usr/bin/autoconf --force >> configure.ac:25: error: possibly undefined macro: AC_DISABLE_STATIC >> ? ? ? If this token and others are legitimate, please use >> m4_pattern_allow. >> ? ? ? See the Autoconf documentation. >> configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL >> autoreconf: /usr/bin/autoconf failed with exit status: 1 >> >> >> On 10 February 2011 14:09, Rainer Gerhards >> wrote: >> > http://www.rsyslog.com/doc/build_from_repo.html >> > >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> >> Sent: Thursday, February 10, 2011 3:08 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] OMPROG Doccumentation >> >> >> >> Hi, >> >> >> >> I am not sure on how to build that as there is no configure script, >> >> how should i generate this from the git checkout? >> >> >> >> Thanks >> >> >> >> Mark >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From laurentducos at gmail.com Thu Feb 10 15:46:52 2011 From: laurentducos at gmail.com (laurent ducos) Date: Thu, 10 Feb 2011 15:46:52 +0100 Subject: [rsyslog] memory leak on rsyslog 5.6.3 with SSL Message-ID: Hello I'm a french user of Rsyslog sorry for my bad english. I use rsyslog 5.6.3 with SSL on a network of 10 machines. On client machines rsyslogd process behaves normally. On the server (Gentoo, 2GB RAM) I noticed a memory leak. In two hours, the process rsyslog increases from 0 to 20% memory usage. I tried to pass the STACK_SIZE of 8192 to 4096 and the phenomenon of memory leak is even more important. My config : $ModLoad imuxsock $ModLoad imklog $ModLoad immark $KLogPath /proc/kmsg $ModLoad imtcp $InputTCPMaxSessions 4000 $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/xxx.crt $DefaultNetstreamDriverCertFile /etc/ssl/rsyslog/xxx.crt $DefaultNetstreamDriverKeyFile /etc/ssl/rsyslog/xxx.pem $InputTCPServerStreamDriverMode 1 $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.xxx.xxx $InputTCPServerRun 10514 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $template TraditionalFileFormatWithFacility,"%TIMESTAMP% %syslogfacility-text% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog $ModLoad ommail $ActionMailSMTPServer xxx.xxx.com $ActionMailFrom xxxx at xxxx $ActionMailTo xxxx at xxxx.xxx $template TraditionalFormatWithPRI,"%PRI-text%: %syslogseverity-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template TraditionalFormatForTomcat,"%msg:::drop-last-lf\n%" $IncludeConfig /etc/rsyslog.d/*.conf From rgerhards at hq.adiscon.com Thu Feb 10 15:48:40 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 15:48:40 +0100 Subject: [rsyslog] OMPROG data not being received In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBF8@GRFEXC.intern.adiscon.com> thanks for the log. But please have a look at it: search for omprog. You'll see that the data is written to the pipe. I have no idea why the OS doesn't deliver it to your program. At this point, rsyslog has called the write() OS API, and I do not see what else should be required. Anyone an idea? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 3:21 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG Doccumentation > > Thanks > > Libtool fixed the build i will send the new log shortly. > > Regards > > On 10 February 2011 14:16, Rainer Gerhards > wrote: > > libtool is missing, probably more. let me see that I upload a tarball > (but > > that will take a short while, I am absent from the system now). > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> Sent: Thursday, February 10, 2011 3:15 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> > >> mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi > >> Can't exec "libtoolize": No such file or directory at > >> /usr/bin/autoreconf line 189. > >> Use of uninitialized value in pattern match (m//) at > >> /usr/bin/autoreconf line 189. > >> autoreconf: Entering directory `.' > >> autoreconf: configure.ac: not using Gettext > >> autoreconf: running: aclocal --force -I m4 > >> autoreconf: configure.ac: tracing > >> autoreconf: configure.ac: not using Libtool > >> autoreconf: running: /usr/bin/autoconf --force > >> configure.ac:25: error: possibly undefined macro: AC_DISABLE_STATIC > >> ? ? ? If this token and others are legitimate, please use > >> m4_pattern_allow. > >> ? ? ? See the Autoconf documentation. > >> configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL > >> autoreconf: /usr/bin/autoconf failed with exit status: 1 > >> > >> > >> On 10 February 2011 14:09, Rainer Gerhards > > >> wrote: > >> > http://www.rsyslog.com/doc/build_from_repo.html > >> > > >> >> -----Original Message----- > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> >> Sent: Thursday, February 10, 2011 3:08 PM > >> >> To: rsyslog-users > >> >> Subject: Re: [rsyslog] OMPROG Doccumentation > >> >> > >> >> Hi, > >> >> > >> >> I am not sure on how to build that as there is no configure > script, > >> >> how should i generate this from the git checkout? > >> >> > >> >> Thanks > >> >> > >> >> Mark > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 10 15:49:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 15:49:51 +0100 Subject: [rsyslog] OMPROG data not being received In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBF8@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71 DDBF8@GR FEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> ahhh... I see the message. Does your application wait for a NL character? I am asking, because you have written none! > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, February 10, 2011 3:49 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG data not being received > > thanks for the log. But please have a look at it: search for omprog. > You'll > see that the data is written to the pipe. I have no idea why the OS > doesn't > deliver it to your program. At this point, rsyslog has called the > write() OS > API, and I do not see what else should be required. Anyone an idea? > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > > Sent: Thursday, February 10, 2011 3:21 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] OMPROG Doccumentation > > > > Thanks > > > > Libtool fixed the build i will send the new log shortly. > > > > Regards > > > > On 10 February 2011 14:16, Rainer Gerhards > > wrote: > > > libtool is missing, probably more. let me see that I upload a > tarball > > (but > > > that will take a short while, I am absent from the system now). > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > > >> Sent: Thursday, February 10, 2011 3:15 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] OMPROG Doccumentation > > >> > > >> mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi > > >> Can't exec "libtoolize": No such file or directory at > > >> /usr/bin/autoreconf line 189. > > >> Use of uninitialized value in pattern match (m//) at > > >> /usr/bin/autoreconf line 189. > > >> autoreconf: Entering directory `.' > > >> autoreconf: configure.ac: not using Gettext > > >> autoreconf: running: aclocal --force -I m4 > > >> autoreconf: configure.ac: tracing > > >> autoreconf: configure.ac: not using Libtool > > >> autoreconf: running: /usr/bin/autoconf --force > > >> configure.ac:25: error: possibly undefined macro: > AC_DISABLE_STATIC > > >> ? ? ? If this token and others are legitimate, please use > > >> m4_pattern_allow. > > >> ? ? ? See the Autoconf documentation. > > >> configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL > > >> autoreconf: /usr/bin/autoconf failed with exit status: 1 > > >> > > >> > > >> On 10 February 2011 14:09, Rainer Gerhards > > > > >> wrote: > > >> > http://www.rsyslog.com/doc/build_from_repo.html > > >> > > > >> >> -----Original Message----- > > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > > >> >> Sent: Thursday, February 10, 2011 3:08 PM > > >> >> To: rsyslog-users > > >> >> Subject: Re: [rsyslog] OMPROG Doccumentation > > >> >> > > >> >> Hi, > > >> >> > > >> >> I am not sure on how to build that as there is no configure > > script, > > >> >> how should i generate this from the git checkout? > > >> >> > > >> >> Thanks > > >> >> > > >> >> Mark > > >> >> _______________________________________________ > > >> >> rsyslog mailing list > > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> >> http://www.rsyslog.com > > >> > _______________________________________________ > > >> > rsyslog mailing list > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> > http://www.rsyslog.com > > >> > > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Thu Feb 10 16:24:59 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 15:24:59 +0000 Subject: [rsyslog] OMPROG data not being received In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> Message-ID: Hi All, How do i send a NL ? in the template ? If it helps anyone below is a copy of my alert script. ------------------------------------------- #!/usr/bin/ruby1.9.1 require 'net/http' require 'uri' require 'cgi' #BudgetSMS Account Details $budgetsms_username = "user" $budgetsms_userid = "uid" $budgetsms_handle = "handle" $budgetsms_password = "password" $budgetsms_server = "http://www.budgetsms.net/api/sendsms" $budgetsms_from = "Company Name" argv = ARGV.join(", ") ARGV.clear def split_args( jmsg ) args = jmsg.split(',') level = args.shift phone = args.shift.lstrip message = args.join("").lstrip action( level, phone, message) end def join_message( argv, argf) if argv.size == 0 split_args(argf) else split_args("#{argv}, #{argf}") end end def write_file( level, phone, message ) dump_file = File.open("/tmp/dump", 'w' ) dump_file.puts "Level: #{level}; Phone: #{phone}; Message: #{message}" dump_file.close end def smsalert( encoded_message, to ) puts to url="#{$budgetsms_server}?username=#{$budgetsms_username}&handle=#{$budgetsms_handle}&userid=#{$budgetsms_userid}&to=#{to}&msg=#{encoded_message}&from=#{$budgetsms_from}" puts url answer=Net::HTTP.get_print URI.parse(url) puts answer end def smscheck( ) url="#{$budgetsms_server}?username=#{$budgetsms_username}&handle=#{$budgetsms_handle}&userid=#{$budgetsms_userid}" puts url answer=Net::HTTP.get_print URI.parse(url) puts answer end def alert( level, phone, message ) if ( level =~ /FILLER/ ) message = "FILE ERROR Alert: #{message}" else message = "System Alert: LEVEL: #{level} #{message}" end encoded_message = CGI.escape( message ).gsub("+", "%20") #smsalert( encoded_message, phone ) write_file( level, phone, message ) end def action( status, phone, message) if ( status =~ /not exist/ ) level = "CRITICAL" alert( level, phone, message ) elsif ( status =~ /Timeout/ ) level = "CRITICAL" alert( level, phone, message ) elsif ( status =~ /failed/ ) level = "CRITICAL" alert( level, phone, message ) elsif ( status =~ /Resource limit matched/ ) level = "CRITICAL" alert( level, phone, message ) elsif ( status =~ /Invalid type/ ) level = "CRITICAL" alert( level, phone, message ) elsif ( status =~ /changed/ ) level = "WARNING" alert( level, phone, message ) elsif ( status =~ /done/ ) level = "WARNING" alert( level, phone, message ) elsif ( status =~ /FILLER/ ) level = "FILLER" alert( level, phone, message ) else level = "OK" alert( level, phone, message ) end end while !ARGF.eof? join_message( argv, ARGF.readline) end On 10 February 2011 14:49, Rainer Gerhards wrote: > ahhh... I see the message. Does your application wait for a NL character? I > am asking, because you have written none! > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Thursday, February 10, 2011 3:49 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] OMPROG data not being received >> >> thanks for the log. But please have a look at it: search for omprog. >> You'll >> see that the data is written to the pipe. I have no idea why the OS >> doesn't >> deliver it to your program. At this point, rsyslog has called the >> write() OS >> API, and I do not see what else should be required. Anyone an idea? >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> > Sent: Thursday, February 10, 2011 3:21 PM >> > To: rsyslog-users >> > Subject: Re: [rsyslog] OMPROG Doccumentation >> > >> > Thanks >> > >> > Libtool fixed the build i will send the new log shortly. >> > >> > Regards >> > >> > On 10 February 2011 14:16, Rainer Gerhards >> > wrote: >> > > libtool is missing, probably more. let me see that I upload a >> tarball >> > (but >> > > that will take a short while, I am absent from the system now). >> > > >> > >> -----Original Message----- >> > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> > >> Sent: Thursday, February 10, 2011 3:15 PM >> > >> To: rsyslog-users >> > >> Subject: Re: [rsyslog] OMPROG Doccumentation >> > >> >> > >> mpo at mordac:rsyslog(v4.6.4-omprogTest)$ autoreconf -fvi >> > >> Can't exec "libtoolize": No such file or directory at >> > >> /usr/bin/autoreconf line 189. >> > >> Use of uninitialized value in pattern match (m//) at >> > >> /usr/bin/autoreconf line 189. >> > >> autoreconf: Entering directory `.' >> > >> autoreconf: configure.ac: not using Gettext >> > >> autoreconf: running: aclocal --force -I m4 >> > >> autoreconf: configure.ac: tracing >> > >> autoreconf: configure.ac: not using Libtool >> > >> autoreconf: running: /usr/bin/autoconf --force >> > >> configure.ac:25: error: possibly undefined macro: >> AC_DISABLE_STATIC >> > >> ? ? ? If this token and others are legitimate, please use >> > >> m4_pattern_allow. >> > >> ? ? ? See the Autoconf documentation. >> > >> configure.ac:26: error: possibly undefined macro: AC_PROG_LIBTOOL >> > >> autoreconf: /usr/bin/autoconf failed with exit status: 1 >> > >> >> > >> >> > >> On 10 February 2011 14:09, Rainer Gerhards >> > >> > >> wrote: >> > >> > http://www.rsyslog.com/doc/build_from_repo.html >> > >> > >> > >> >> -----Original Message----- >> > >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > >> >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> > >> >> Sent: Thursday, February 10, 2011 3:08 PM >> > >> >> To: rsyslog-users >> > >> >> Subject: Re: [rsyslog] OMPROG Doccumentation >> > >> >> >> > >> >> Hi, >> > >> >> >> > >> >> I am not sure on how to build that as there is no configure >> > script, >> > >> >> how should i generate this from the git checkout? >> > >> >> >> > >> >> Thanks >> > >> >> >> > >> >> Mark >> > >> >> _______________________________________________ >> > >> >> rsyslog mailing list >> > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > >> >> http://www.rsyslog.com >> > >> > _______________________________________________ >> > >> > rsyslog mailing list >> > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > >> > http://www.rsyslog.com >> > >> > >> > >> _______________________________________________ >> > >> rsyslog mailing list >> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > >> http://www.rsyslog.com >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com >> > > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mark at thermeon.com Thu Feb 10 16:27:46 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 15:27:46 +0000 Subject: [rsyslog] OMPROG data not being received In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> Message-ID: I think my script is actually waiting on an end of file rather than end of line terminator, If that makes a difference ? Mark From mark at thermeon.com Thu Feb 10 16:51:14 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 10 Feb 2011 15:51:14 +0000 Subject: [rsyslog] OMPROG data not being received In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF8@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> Message-ID: Hi, Thanks that is fixed, by adding "\n" to the end of my template it now flushes to the script straight away. Thanks Mark On 10 February 2011 15:27, Mark Olliver wrote: > I think my script is actually waiting on an end of file rather than > end of line terminator, > > If that makes a difference ? > > Mark > From rgerhards at hq.adiscon.com Thu Feb 10 16:55:41 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 16:55:41 +0100 Subject: [rsyslog] OMPROG data not being received In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBE8@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEA@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBEB@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF1@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF6@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF7@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71D DBF8@GRF EXC.intern.a discon.com><9B6E2A8877C38245BFB15CC491A11DA71DDBF9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBFC@GRFEXC.intern.adiscon.com> Great. It is important to know that omprog only writes what you tell it. So if the program waits for a NL, this must also be written. As a side-note, waiting for end of file would not make much sense at all: opening and closing the pipe for each message would be enormous overhead. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Thursday, February 10, 2011 4:51 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG data not being received > > Hi, > > Thanks that is fixed, by adding "\n" to the end of my template it now > flushes to the script straight away. > > Thanks > > Mark > > On 10 February 2011 15:27, Mark Olliver wrote: > > I think my script is actually waiting on an end of file rather than > > end of line terminator, > > > > If that makes a difference ? > > > > Mark > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From eschoeller at users.sourceforge.net Thu Feb 10 16:50:55 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Thu, 10 Feb 2011 08:50:55 -0700 Subject: [rsyslog] Queuing subsystem and message filtering Message-ID: <4D54095F.6050906@users.sourceforge.net> Hello list, I recently tried out message queuing using rsyslog-5.6.3. Take the following config snippet: ... # Throw out all messages with debug severity before we log to the network if $syslogseverity-text == 'debug' then ~ $WorkDirectory /var/spool/rsyslog # location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName logserver-queue # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down # Log anything that hasn't been specifically filtered out with '~' to logserver *.* @logserver.colorado.edu ... In this scenario I would assume that all messages with the severity of debug don't get logged over the network to 'logserver.colorado.edu'. But they do. If I comment all 5 of the directives that activate queuing, the debug messages are no longer logged over the network. Is this by design, and if so could someone explain why? Is this a bug? Do I have a configuration problem? Thanks! Eric Schoeller University of Colorado, Boulder Information Technology Services From rgerhards at hq.adiscon.com Thu Feb 10 17:03:10 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 17:03:10 +0100 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <4D54095F.6050906@users.sourceforge.net> References: <4D54095F.6050906@users.sourceforge.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> HAve you checked for any error messages from rsyslogd inside your logs (do you even log them -- many distros don't do by default :-(( ) I am asking because the snippet looks syntactically wrong (then on a separate line). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > Sent: Thursday, February 10, 2011 4:51 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Queuing subsystem and message filtering > > Hello list, > > I recently tried out message queuing using rsyslog-5.6.3. > > Take the following config snippet: > > ... > # Throw out all messages with debug severity before we log to the > network > if $syslogseverity-text == 'debug' > then ~ > > $WorkDirectory /var/spool/rsyslog # location for work (spool) > files > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName logserver-queue # set file name, also enables > disk mode > $ActionResumeRetryCount -1 # infinite retries on insert > failure > $ActionQueueSaveOnShutdown on # save in-memory data if > rsyslog > shuts down > > # Log anything that hasn't been specifically filtered out with '~' to > logserver > *.* @logserver.colorado.edu > ... > > In this scenario I would assume that all messages with the severity of > debug don't get logged over the network to 'logserver.colorado.edu'. > But > they do. If I comment all 5 of the directives that activate queuing, > the > debug messages are no longer logged over the network. > > Is this by design, and if so could someone explain why? Is this a bug? > Do I have a configuration problem? > > Thanks! > > Eric Schoeller > University of Colorado, Boulder > Information Technology Services > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Feb 10 18:13:50 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Feb 2011 09:13:50 -0800 (PST) Subject: [rsyslog] OMPROG data not being received In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBFC@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDBFC@GRFEXC.intern.adiscon.com> Message-ID: now I'm a little puzzled. how is omprog better than the old sysklog approach of |/path/to/script? David Lang On Thu, 10 Feb 2011, Rainer Gerhards wrote: > Great. It is important to know that omprog only writes what you tell it. So > if the program waits for a NL, this must also be written. As a side-note, > waiting for end of file would not make much sense at all: opening and closing > the pipe for each message would be enormous overhead. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver >> Sent: Thursday, February 10, 2011 4:51 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] OMPROG data not being received >> >> Hi, >> >> Thanks that is fixed, by adding "\n" to the end of my template it now >> flushes to the script straight away. >> >> Thanks >> >> Mark >> >> On 10 February 2011 15:27, Mark Olliver wrote: >>> I think my script is actually waiting on an end of file rather than >>> end of line terminator, >>> >>> If that makes a difference ? >>> >>> Mark >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Feb 10 18:15:55 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 18:15:55 +0100 Subject: [rsyslog] OMPROG data not being received In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDBFC@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC00@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, February 10, 2011 6:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] OMPROG data not being received > > now I'm a little puzzled. > > how is omprog better than the old sysklog approach of |/path/to/script? It's not a script, it's a pipe. omprog actually starts up the script. In theory, it also restarts it if it dies (I say in theory because after I had written the module, those that had interest in it never again showed up -- so I did not put much more effort into it at that time). Rainer > David Lang > > On Thu, 10 Feb 2011, Rainer Gerhards wrote: > > > Great. It is important to know that omprog only writes what you tell > it. So > > if the program waits for a NL, this must also be written. As a side- > note, > > waiting for end of file would not make much sense at all: opening and > closing > > the pipe for each message would be enormous overhead. > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mark Olliver > >> Sent: Thursday, February 10, 2011 4:51 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] OMPROG data not being received > >> > >> Hi, > >> > >> Thanks that is fixed, by adding "\n" to the end of my template it > now > >> flushes to the script straight away. > >> > >> Thanks > >> > >> Mark > >> > >> On 10 February 2011 15:27, Mark Olliver wrote: > >>> I think my script is actually waiting on an end of file rather than > >>> end of line terminator, > >>> > >>> If that makes a difference ? > >>> > >>> Mark > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From eschoeller at users.sourceforge.net Thu Feb 10 19:10:45 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Thu, 10 Feb 2011 11:10:45 -0700 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> References: <4D54095F.6050906@users.sourceforge.net> <9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> Message-ID: <4D542A25.3050006@users.sourceforge.net> Oh, sorry my mail client must have munged those lines. I am getting a clean start-up and shutdown: Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] exiting on signal 15. Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] start Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg started. Here is a better representation of it: http://pastebin.com/M6Lwseqr the two '/var/log/everything_to_send' where being used for debugging. I actually stacked both of those lines directly above the @logserver.colorado.edu line, and that's when I noticed a discrepancy between what was getting logged to everything_to_send and everything_to_send2. Then I bumped those lines back up above the queuing directives and observed that debug (and a bunch of other filtered messages) were making it to logserver.colorado.edu. -- Eric Rainer Gerhards wrote: > HAve you checked for any error messages from rsyslogd inside your logs (do > you even log them -- many distros don't do by default :-(( ) > > I am asking because the snippet looks syntactically wrong (then on a separate > line). > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >> Sent: Thursday, February 10, 2011 4:51 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Queuing subsystem and message filtering >> >> Hello list, >> >> I recently tried out message queuing using rsyslog-5.6.3. >> >> Take the following config snippet: >> >> ... >> # Throw out all messages with debug severity before we log to the >> network >> if $syslogseverity-text == 'debug' >> then ~ >> >> $WorkDirectory /var/spool/rsyslog # location for work (spool) >> files >> $ActionQueueType LinkedList # use asynchronous processing >> $ActionQueueFileName logserver-queue # set file name, also enables >> disk mode >> $ActionResumeRetryCount -1 # infinite retries on insert >> failure >> $ActionQueueSaveOnShutdown on # save in-memory data if >> rsyslog >> shuts down >> >> # Log anything that hasn't been specifically filtered out with '~' to >> logserver >> *.* @logserver.colorado.edu >> ... >> >> In this scenario I would assume that all messages with the severity of >> debug don't get logged over the network to 'logserver.colorado.edu'. >> But >> they do. If I comment all 5 of the directives that activate queuing, >> the >> debug messages are no longer logged over the network. >> >> Is this by design, and if so could someone explain why? Is this a bug? >> Do I have a configuration problem? >> >> Thanks! >> >> Eric Schoeller >> University of Colorado, Boulder >> Information Technology Services >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> From rgerhards at hq.adiscon.com Thu Feb 10 21:10:35 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Feb 2011 21:10:35 +0100 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <4D542A25.3050006@users.sourceforge.net> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> <4D542A25.3050006@users.sourceforge.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> Could you create a debug log with such a message? It should tell us what is going on. Instructions: http://www.rsyslog.com/doc/troubleshoot.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > Sent: Thursday, February 10, 2011 7:11 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing subsystem and message filtering > > Oh, sorry my mail client must have munged those lines. I am getting a > clean start-up and shutdown: > > Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] > exiting > on signal 15. > Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. > Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] start > Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg > started. > > Here is a better representation of it: > > http://pastebin.com/M6Lwseqr > > the two '/var/log/everything_to_send' where being used for debugging. I > actually stacked both of those lines directly above the > @logserver.colorado.edu line, and that's when I noticed a discrepancy > between what was getting logged to everything_to_send and > everything_to_send2. Then I bumped those lines back up above the > queuing > directives and observed that debug (and a bunch of other filtered > messages) were making it to logserver.colorado.edu. > > -- Eric > > Rainer Gerhards wrote: > > HAve you checked for any error messages from rsyslogd inside your > logs (do > > you even log them -- many distros don't do by default :-(( ) > > > > I am asking because the snippet looks syntactically wrong (then on a > separate > > line). > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >> Sent: Thursday, February 10, 2011 4:51 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] Queuing subsystem and message filtering > >> > >> Hello list, > >> > >> I recently tried out message queuing using rsyslog-5.6.3. > >> > >> Take the following config snippet: > >> > >> ... > >> # Throw out all messages with debug severity before we log to the > >> network > >> if $syslogseverity-text == 'debug' > >> then ~ > >> > >> $WorkDirectory /var/spool/rsyslog # location for work (spool) > >> files > >> $ActionQueueType LinkedList # use asynchronous > processing > >> $ActionQueueFileName logserver-queue # set file name, also > enables > >> disk mode > >> $ActionResumeRetryCount -1 # infinite retries on insert > >> failure > >> $ActionQueueSaveOnShutdown on # save in-memory data if > >> rsyslog > >> shuts down > >> > >> # Log anything that hasn't been specifically filtered out with '~' > to > >> logserver > >> *.* @logserver.colorado.edu > >> ... > >> > >> In this scenario I would assume that all messages with the severity > of > >> debug don't get logged over the network to 'logserver.colorado.edu'. > >> But > >> they do. If I comment all 5 of the directives that activate queuing, > >> the > >> debug messages are no longer logged over the network. > >> > >> Is this by design, and if so could someone explain why? Is this a > bug? > >> Do I have a configuration problem? > >> > >> Thanks! > >> > >> Eric Schoeller > >> University of Colorado, Boulder > >> Information Technology Services > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From eschoeller at users.sourceforge.net Fri Feb 11 01:47:48 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Thu, 10 Feb 2011 17:47:48 -0700 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> <4D542A25.3050006@users.sourceforge.net> <9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> Message-ID: <4D548734.5090902@users.sourceforge.net> Okay, I started up rsyslog with debugging twice. Once with queuing turned on, once without. I immediately ran the test script that sends one message to each facility at each severity and then killed the rsyslog daemon ... so the results between the two *should* be fairly identical, there is minimal additional syslog traffic on the machine at this time. Immediately, I noticed: host: /tmp> grep -i logserver rsyslog.queue.debug | wc -l 173 host: /tmp> grep -i logserver rsyslog.noqueue.debug | wc -l 142 As I suspected, more logs were being sent to the logserver with queuing turned on (ie. they weren't getting filtered). I've posted the debug logs here: http://spark.colorado.edu/rsyslog/ I am somewhat new to rsyslog, and I've certainly never trolled through a debug file. I may have a chance to take a look tonight, but given my familiarity with the software it may be difficult to zero in on anything. Thanks! -- Eric Rainer Gerhards wrote: > Could you create a debug log with such a message? It should tell us what is > going on. Instructions: > > http://www.rsyslog.com/doc/troubleshoot.html > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >> Sent: Thursday, February 10, 2011 7:11 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing subsystem and message filtering >> >> Oh, sorry my mail client must have munged those lines. I am getting a >> clean start-up and shutdown: >> >> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >> swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] >> exiting >> on signal 15. >> Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. >> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >> swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] start >> Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg >> started. >> >> Here is a better representation of it: >> >> http://pastebin.com/M6Lwseqr >> >> the two '/var/log/everything_to_send' where being used for debugging. I >> actually stacked both of those lines directly above the >> @logserver.colorado.edu line, and that's when I noticed a discrepancy >> between what was getting logged to everything_to_send and >> everything_to_send2. Then I bumped those lines back up above the >> queuing >> directives and observed that debug (and a bunch of other filtered >> messages) were making it to logserver.colorado.edu. >> >> -- Eric >> >> Rainer Gerhards wrote: >> >>> HAve you checked for any error messages from rsyslogd inside your >>> >> logs (do >> >>> you even log them -- many distros don't do by default :-(( ) >>> >>> I am asking because the snippet looks syntactically wrong (then on a >>> >> separate >> >>> line). >>> >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >>>> Sent: Thursday, February 10, 2011 4:51 PM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] Queuing subsystem and message filtering >>>> >>>> Hello list, >>>> >>>> I recently tried out message queuing using rsyslog-5.6.3. >>>> >>>> Take the following config snippet: >>>> >>>> ... >>>> # Throw out all messages with debug severity before we log to the >>>> network >>>> if $syslogseverity-text == 'debug' >>>> then ~ >>>> >>>> $WorkDirectory /var/spool/rsyslog # location for work (spool) >>>> files >>>> $ActionQueueType LinkedList # use asynchronous >>>> >> processing >> >>>> $ActionQueueFileName logserver-queue # set file name, also >>>> >> enables >> >>>> disk mode >>>> $ActionResumeRetryCount -1 # infinite retries on insert >>>> failure >>>> $ActionQueueSaveOnShutdown on # save in-memory data if >>>> rsyslog >>>> shuts down >>>> >>>> # Log anything that hasn't been specifically filtered out with '~' >>>> >> to >> >>>> logserver >>>> *.* @logserver.colorado.edu >>>> ... >>>> >>>> In this scenario I would assume that all messages with the severity >>>> >> of >> >>>> debug don't get logged over the network to 'logserver.colorado.edu'. >>>> But >>>> they do. If I comment all 5 of the directives that activate queuing, >>>> the >>>> debug messages are no longer logged over the network. >>>> >>>> Is this by design, and if so could someone explain why? Is this a >>>> >> bug? >> >>>> Do I have a configuration problem? >>>> >>>> Thanks! >>>> >>>> Eric Schoeller >>>> University of Colorado, Boulder >>>> Information Technology Services >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> From david at lang.hm Fri Feb 11 02:32:28 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Feb 2011 17:32:28 -0800 (PST) Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <4D548734.5090902@users.sourceforge.net> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com> <4D542A25.3050006@users.sourceforge.net> <9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> <4D548734.5090902@users.sourceforge.net> Message-ID: start by searching for your test message, you will probably see it a few times in the log. in that area you will see what tests it is doing, and what it decides the result of each test is. this will probably make it obvious what test isn't acting as expected. David Lang On Thu, 10 Feb 2011, Eric Schoeller wrote: > Date: Thu, 10 Feb 2011 17:47:48 -0700 > From: Eric Schoeller > To: rsyslog-users > Subject: Re: [rsyslog] Queuing subsystem and message filtering > > Okay, > > I started up rsyslog with debugging twice. Once with queuing turned on, once > without. I immediately ran the test script that sends one message to each > facility at each severity and then killed the rsyslog daemon ... so the > results between the two *should* be fairly identical, there is minimal > additional syslog traffic on the machine at this time. > > Immediately, I noticed: > > host: /tmp> grep -i logserver rsyslog.queue.debug | wc -l > 173 > host: /tmp> grep -i logserver rsyslog.noqueue.debug | wc -l > 142 > > As I suspected, more logs were being sent to the logserver with queuing > turned on (ie. they weren't getting filtered). I've posted the debug logs > here: > http://spark.colorado.edu/rsyslog/ > > I am somewhat new to rsyslog, and I've certainly never trolled through a > debug file. I may have a chance to take a look tonight, but given my > familiarity with the software it may be difficult to zero in on anything. > > Thanks! > > -- Eric > > Rainer Gerhards wrote: >> Could you create a debug log with such a message? It should tell us what is >> going on. Instructions: >> >> http://www.rsyslog.com/doc/troubleshoot.html >> >> Rainer >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >>> Sent: Thursday, February 10, 2011 7:11 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Queuing subsystem and message filtering >>> >>> Oh, sorry my mail client must have munged those lines. I am getting a >>> clean start-up and shutdown: >>> >>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >>> swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] >>> exiting >>> on signal 15. >>> Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. >>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >>> swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] start >>> Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg >>> started. >>> >>> Here is a better representation of it: >>> >>> http://pastebin.com/M6Lwseqr >>> >>> the two '/var/log/everything_to_send' where being used for debugging. I >>> actually stacked both of those lines directly above the >>> @logserver.colorado.edu line, and that's when I noticed a discrepancy >>> between what was getting logged to everything_to_send and >>> everything_to_send2. Then I bumped those lines back up above the >>> queuing >>> directives and observed that debug (and a bunch of other filtered >>> messages) were making it to logserver.colorado.edu. >>> >>> -- Eric >>> >>> Rainer Gerhards wrote: >>> >>>> HAve you checked for any error messages from rsyslogd inside your >>>> >>> logs (do >>> >>>> you even log them -- many distros don't do by default :-(( ) >>>> >>>> I am asking because the snippet looks syntactically wrong (then on a >>>> >>> separate >>> >>>> line). >>>> >>>> Rainer >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >>>>> Sent: Thursday, February 10, 2011 4:51 PM >>>>> To: rsyslog at lists.adiscon.com >>>>> Subject: [rsyslog] Queuing subsystem and message filtering >>>>> >>>>> Hello list, >>>>> >>>>> I recently tried out message queuing using rsyslog-5.6.3. >>>>> >>>>> Take the following config snippet: >>>>> >>>>> ... >>>>> # Throw out all messages with debug severity before we log to the >>>>> network >>>>> if $syslogseverity-text == 'debug' >>>>> then ~ >>>>> >>>>> $WorkDirectory /var/spool/rsyslog # location for work (spool) >>>>> files >>>>> $ActionQueueType LinkedList # use asynchronous >>>>> >>> processing >>> >>>>> $ActionQueueFileName logserver-queue # set file name, also >>>>> >>> enables >>> >>>>> disk mode >>>>> $ActionResumeRetryCount -1 # infinite retries on insert >>>>> failure >>>>> $ActionQueueSaveOnShutdown on # save in-memory data if >>>>> rsyslog >>>>> shuts down >>>>> >>>>> # Log anything that hasn't been specifically filtered out with '~' >>>>> >>> to >>> >>>>> logserver >>>>> *.* @logserver.colorado.edu >>>>> ... >>>>> >>>>> In this scenario I would assume that all messages with the severity >>>>> >>> of >>> >>>>> debug don't get logged over the network to 'logserver.colorado.edu'. >>>>> But >>>>> they do. If I comment all 5 of the directives that activate queuing, >>>>> the >>>>> debug messages are no longer logged over the network. >>>>> >>>>> Is this by design, and if so could someone explain why? Is this a >>>>> >>> bug? >>> >>>>> Do I have a configuration problem? >>>>> >>>>> Thanks! >>>>> >>>>> Eric Schoeller >>>>> University of Colorado, Boulder >>>>> Information Technology Services >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Feb 11 05:38:13 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Feb 2011 20:38:13 -0800 (PST) Subject: [rsyslog] how can a parser insert data into a message Message-ID: the various parser modules that I've submitted are all removing data from the log message or overwriting the data in place. But I've now run across a situation where I need to insert information into the message. I know that this can be done because the sanitizing call does exactly this. I am assuming that this is doing something like allocating a new string and copying the data into the new string. the concern is how to do this in a way that will survive the exit from the module, not confuse any of the many pointers or sizes that are involved, and make sure everything is properly freed afterwords. should I just search for the sanitizing routine and copy what it does (and can you point me at it?), or do you want me to wait until you have time to write something up on this? David Lang From rgerhards at hq.adiscon.com Fri Feb 11 07:29:10 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Feb 2011 07:29:10 +0100 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <4D548734.5090902@users.sourceforge.net> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com><4D542A25.3050006@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> <4D548734.5090902@users.sourceforge.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC0C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > Sent: Friday, February 11, 2011 1:48 AM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing subsystem and message filtering > > Okay, > > I started up rsyslog with debugging twice. Once with queuing turned on, > once without. What exactly do you mean with "queing turned on"? You mean disk queue enabled, right? I am asking because rsyslog always queues, but via different methods. I'd like to be sure to look at the right thing (and, yes, I could extract that from the debug log as well but I'd like me to save that time...). Thanks, Rainer >I immediately ran the test script that sends one message > to each facility at each severity and then killed the rsyslog daemon > ... > so the results between the two *should* be fairly identical, there is > minimal additional syslog traffic on the machine at this time. > > Immediately, I noticed: > > host: /tmp> grep -i logserver rsyslog.queue.debug | wc -l > 173 > host: /tmp> grep -i logserver rsyslog.noqueue.debug | wc -l > 142 > > As I suspected, more logs were being sent to the logserver with queuing > turned on (ie. they weren't getting filtered). I've posted the debug > logs here: > http://spark.colorado.edu/rsyslog/ > > I am somewhat new to rsyslog, and I've certainly never trolled through > a > debug file. I may have a chance to take a look tonight, but given my > familiarity with the software it may be difficult to zero in on > anything. > > Thanks! > > -- Eric > > Rainer Gerhards wrote: > > Could you create a debug log with such a message? It should tell us > what is > > going on. Instructions: > > > > http://www.rsyslog.com/doc/troubleshoot.html > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >> Sent: Thursday, February 10, 2011 7:11 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing subsystem and message filtering > >> > >> Oh, sorry my mail client must have munged those lines. I am getting > a > >> clean start-up and shutdown: > >> > >> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > >> swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] > >> exiting > >> on signal 15. > >> Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. > >> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > >> swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] > start > >> Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg > >> started. > >> > >> Here is a better representation of it: > >> > >> http://pastebin.com/M6Lwseqr > >> > >> the two '/var/log/everything_to_send' where being used for > debugging. I > >> actually stacked both of those lines directly above the > >> @logserver.colorado.edu line, and that's when I noticed a > discrepancy > >> between what was getting logged to everything_to_send and > >> everything_to_send2. Then I bumped those lines back up above the > >> queuing > >> directives and observed that debug (and a bunch of other filtered > >> messages) were making it to logserver.colorado.edu. > >> > >> -- Eric > >> > >> Rainer Gerhards wrote: > >> > >>> HAve you checked for any error messages from rsyslogd inside your > >>> > >> logs (do > >> > >>> you even log them -- many distros don't do by default :-(( ) > >>> > >>> I am asking because the snippet looks syntactically wrong (then on > a > >>> > >> separate > >> > >>> line). > >>> > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >>>> Sent: Thursday, February 10, 2011 4:51 PM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: [rsyslog] Queuing subsystem and message filtering > >>>> > >>>> Hello list, > >>>> > >>>> I recently tried out message queuing using rsyslog-5.6.3. > >>>> > >>>> Take the following config snippet: > >>>> > >>>> ... > >>>> # Throw out all messages with debug severity before we log to the > >>>> network > >>>> if $syslogseverity-text == 'debug' > >>>> then ~ > >>>> > >>>> $WorkDirectory /var/spool/rsyslog # location for work > (spool) > >>>> files > >>>> $ActionQueueType LinkedList # use asynchronous > >>>> > >> processing > >> > >>>> $ActionQueueFileName logserver-queue # set file name, also > >>>> > >> enables > >> > >>>> disk mode > >>>> $ActionResumeRetryCount -1 # infinite retries on > insert > >>>> failure > >>>> $ActionQueueSaveOnShutdown on # save in-memory data if > >>>> rsyslog > >>>> shuts down > >>>> > >>>> # Log anything that hasn't been specifically filtered out with '~' > >>>> > >> to > >> > >>>> logserver > >>>> *.* @logserver.colorado.edu > >>>> ... > >>>> > >>>> In this scenario I would assume that all messages with the > severity > >>>> > >> of > >> > >>>> debug don't get logged over the network to > 'logserver.colorado.edu'. > >>>> But > >>>> they do. If I comment all 5 of the directives that activate > queuing, > >>>> the > >>>> debug messages are no longer logged over the network. > >>>> > >>>> Is this by design, and if so could someone explain why? Is this a > >>>> > >> bug? > >> > >>>> Do I have a configuration problem? > >>>> > >>>> Thanks! > >>>> > >>>> Eric Schoeller > >>>> University of Colorado, Boulder > >>>> Information Technology Services > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Fri Feb 11 07:41:44 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Feb 2011 07:41:44 +0100 Subject: [rsyslog] how can a parser insert data into a message In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com> Have a look at ./runtime/parser.c, function SanitizeMsg. It builds a new buffer and uses MsgSetRawMsg to set the new buffer. MsgSetRawMsg handles the "dirty" internals of message object buffer manipulation. Note that it may be quicker to manipulate the buffer pointers yourself. But then you must be very careful. MsgSetRawMsg should provide the necessary hints. The thing to keep on your mind is that up to a certain message length, a buffer is used from the msg object itself (thus saving one malloc/free call) whereas for larger size messages, memory is allocated. You need to keep that straight during manipulation. As a side-note, it would probably be useful if you could take some bullet points on how to modify things, so that others can find that information in the case they want to do that themselves. Could go to the wiki or I could include it in the doc set. Just a suggestion, though... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 11, 2011 5:38 AM > To: rsyslog-users > Subject: [rsyslog] how can a parser insert data into a message > > the various parser modules that I've submitted are all removing data > from > the log message or overwriting the data in place. > > But I've now run across a situation where I need to insert information > into the message. I know that this can be done because the sanitizing > call > does exactly this. I am assuming that this is doing something like > allocating a new string and copying the data into the new string. > > the concern is how to do this in a way that will survive the exit from > the > module, not confuse any of the many pointers or sizes that are > involved, > and make sure everything is properly freed afterwords. > > should I just search for the sanitizing routine and copy what it does > (and > can you point me at it?), or do you want me to wait until you have time > to > write something up on this? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Feb 11 08:02:06 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Feb 2011 23:02:06 -0800 (PST) Subject: [rsyslog] how can a parser insert data into a message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Feb 2011, Rainer Gerhards wrote: > Have a look at ./runtime/parser.c, function SanitizeMsg. It builds a new > buffer and uses MsgSetRawMsg to set the new buffer. MsgSetRawMsg handles the > "dirty" internals of message object buffer manipulation. > > Note that it may be quicker to manipulate the buffer pointers yourself. But > then you must be very careful. MsgSetRawMsg should provide the necessary > hints. The thing to keep on your mind is that up to a certain message length, > a buffer is used from the msg object itself (thus saving one malloc/free > call) whereas for larger size messages, memory is allocated. You need to keep > that straight during manipulation. I'll look at it and see how hard it is to separate these two cases. thanks for the pointer here. > As a side-note, it would probably be useful if you could take some bullet > points on how to modify things, so that others can find that information in > the case they want to do that themselves. Could go to the wiki or I could > include it in the doc set. Just a suggestion, though... I'll see what I can do. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, February 11, 2011 5:38 AM >> To: rsyslog-users >> Subject: [rsyslog] how can a parser insert data into a message >> >> the various parser modules that I've submitted are all removing data >> from >> the log message or overwriting the data in place. >> >> But I've now run across a situation where I need to insert information >> into the message. I know that this can be done because the sanitizing >> call >> does exactly this. I am assuming that this is doing something like >> allocating a new string and copying the data into the new string. >> >> the concern is how to do this in a way that will survive the exit from >> the >> module, not confuse any of the many pointers or sizes that are >> involved, >> and make sure everything is properly freed afterwords. >> >> should I just search for the sanitizing routine and copy what it does >> (and >> can you point me at it?), or do you want me to wait until you have time >> to >> write something up on this? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Feb 11 08:05:35 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Feb 2011 08:05:35 +0100 Subject: [rsyslog] how can a parser insert data into a message In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC11@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 11, 2011 8:02 AM > To: rsyslog-users > Subject: Re: [rsyslog] how can a parser insert data into a message > > On Fri, 11 Feb 2011, Rainer Gerhards wrote: > > > Have a look at ./runtime/parser.c, function SanitizeMsg. It builds a > new > > buffer and uses MsgSetRawMsg to set the new buffer. MsgSetRawMsg > handles the > > "dirty" internals of message object buffer manipulation. > > > > Note that it may be quicker to manipulate the buffer pointers > yourself. But > > then you must be very careful. MsgSetRawMsg should provide the > necessary > > hints. The thing to keep on your mind is that up to a certain message > length, > > a buffer is used from the msg object itself (thus saving one > malloc/free > > call) whereas for larger size messages, memory is allocated. You need > to keep > > that straight during manipulation. > > I'll look at it and see how hard it is to separate these two cases. > thanks > for the pointer here. Just let me add that I did find it of questionable value to try avoid the malloc here. At least in the sanitization problem, this would have resulted in very complex code. And while saving memory writes and calls to the malloc subsystem is useful, I thought that it would not have brought much benefit in that case. Depending on what you intend to do (well-defined insert at late point) things may be different, though. Rainer > > > As a side-note, it would probably be useful if you could take some > bullet > > points on how to modify things, so that others can find that > information in > > the case they want to do that themselves. Could go to the wiki or I > could > > include it in the doc set. Just a suggestion, though... > > I'll see what I can do. > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Friday, February 11, 2011 5:38 AM > >> To: rsyslog-users > >> Subject: [rsyslog] how can a parser insert data into a message > >> > >> the various parser modules that I've submitted are all removing data > >> from > >> the log message or overwriting the data in place. > >> > >> But I've now run across a situation where I need to insert > information > >> into the message. I know that this can be done because the > sanitizing > >> call > >> does exactly this. I am assuming that this is doing something like > >> allocating a new string and copying the data into the new string. > >> > >> the concern is how to do this in a way that will survive the exit > from > >> the > >> module, not confuse any of the many pointers or sizes that are > >> involved, > >> and make sure everything is properly freed afterwords. > >> > >> should I just search for the sanitizing routine and copy what it > does > >> (and > >> can you point me at it?), or do you want me to wait until you have > time > >> to > >> write something up on this? > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Feb 11 08:41:58 2011 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Feb 2011 23:41:58 -0800 (PST) Subject: [rsyslog] how can a parser insert data into a message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC11@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDC11@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Feb 2011, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> >> On Fri, 11 Feb 2011, Rainer Gerhards wrote: >> >>> Have a look at ./runtime/parser.c, function SanitizeMsg. It builds a >>> new buffer and uses MsgSetRawMsg to set the new buffer. MsgSetRawMsg >>> handles the "dirty" internals of message object buffer manipulation. >>> >>> Note that it may be quicker to manipulate the buffer pointers >>> yourself. But then you must be very careful. MsgSetRawMsg should >>> provide the necessary hints. The thing to keep on your mind is that up >>> to a certain message length, a buffer is used from the msg object >>> itself (thus saving one malloc/free call) whereas for larger size >>> messages, memory is allocated. You need to keep that straight during >>> manipulation. >> >> I'll look at it and see how hard it is to separate these two cases. >> thanks >> for the pointer here. > > Just let me add that I did find it of questionable value to try avoid the > malloc here. At least in the sanitization problem, this would have resulted > in very complex code. And while saving memory writes and calls to the malloc > subsystem is useful, I thought that it would not have brought much benefit in > that case. Depending on what you intend to do (well-defined insert at late > point) things may be different, though. My initial thought is something along the following 1. find out how much space is available in whatever buffer the message is in (potentially 0 if the buffer is exactly the right size) document what needs to happen to adjust how much of the buffer is used (I've already figured out some of this with the existing parser modules) 2. if there is not enough space, document what the process is to allocate a new buffer and make the system use it. at this point it should be fairly straightforward to write a routine to do something along the lines of 'make sure I have enough space in the buffer to add X characters' and have it either return immediatly if there's enough space or allocate the larger buffer if needed and return after doing that. there will be some things that will need to be documented as side effects (pointers into the existing message may be invalid at that point, including values in the msg structure) this could be mis-used (running this routine for every control character found could result in many malloc/free pairs for example), and so examples will need to be given of doing a 2-pass routine, pass 1 to figure out what you want to do, and then make sure there's enough space and do pass 2 to modify the buffer as needed. Using this for sanitizing would still be slightly less efficient than the approach you probably use now (allocate a new buffer, copy things into it as you go to construct a new message, then set the message into the structure), but probably not by more than two copies of the text. As a result, it may be that the result will be enough cleaner to be worth the cost. I'm thinking that the new routine would be to copy the text from the old buffer to the new one, then copy everything after your first insert to the end of the buffer. after that you are copying data from late in the buffer to earlier in the buffer, which may even be faster than copying small amounts of data from one buffer to another as it may result in better cache behavior. in fact, this pattern is probably common enough to make it a routine itself something like int InsertIntoRawMsg(int offset, int count) inserts at least count spaces into the message at position offset from the beginning of the message, returns the number of spaces actually inserted (may be more than the number requested) or would it be better to return the number of extra characters available in the buffer after the end of the string? I figure error checking on the return is not needed because if it can't allocate the space we need to bail out (with whatever rsyslog does when it runs out of memory, probably aborting the message entirely) David Lang > Rainer > >> >>> As a side-note, it would probably be useful if you could take some >>> bullet points on how to modify things, so that others can find that >>> information in the case they want to do that themselves. Could go to >>> the wiki or I could include it in the doc set. Just a suggestion, >>> though... >> >> I'll see what I can do. >> >> David Lang >> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >>>> Sent: Friday, February 11, 2011 5:38 AM >>>> To: rsyslog-users >>>> Subject: [rsyslog] how can a parser insert data into a message >>>> >>>> the various parser modules that I've submitted are all removing data >>>> from >>>> the log message or overwriting the data in place. >>>> >>>> But I've now run across a situation where I need to insert >> information >>>> into the message. I know that this can be done because the >> sanitizing >>>> call >>>> does exactly this. I am assuming that this is doing something like >>>> allocating a new string and copying the data into the new string. >>>> >>>> the concern is how to do this in a way that will survive the exit >> from >>>> the >>>> module, not confuse any of the many pointers or sizes that are >>>> involved, >>>> and make sure everything is properly freed afterwords. >>>> >>>> should I just search for the sanitizing routine and copy what it >> does >>>> (and >>>> can you point me at it?), or do you want me to wait until you have >> time >>>> to >>>> write something up on this? >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From eschoeller at users.sourceforge.net Fri Feb 11 16:10:53 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Fri, 11 Feb 2011 08:10:53 -0700 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC0C@GRFEXC.intern.adiscon.com> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com><4D542A25.3050006@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com> <4D548734.5090902@users.sourceforge.net> <9B6E2A8877C38245BFB15CC491A11DA71DDC0C@GRFEXC.intern.adiscon.com> Message-ID: <4D55517D.90506@users.sourceforge.net> Yes you are right, I meant to say disk queuing. -- Eric On 02/10/11 23:29, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >> Sent: Friday, February 11, 2011 1:48 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Queuing subsystem and message filtering >> >> Okay, >> >> I started up rsyslog with debugging twice. Once with queuing turned on, >> once without. > What exactly do you mean with "queing turned on"? You mean disk queue > enabled, right? I am asking because rsyslog always queues, but via different > methods. I'd like to be sure to look at the right thing (and, yes, I could > extract that from the debug log as well but I'd like me to save that > time...). > > Thanks, > Rainer > >> I immediately ran the test script that sends one message >> to each facility at each severity and then killed the rsyslog daemon >> ... >> so the results between the two *should* be fairly identical, there is >> minimal additional syslog traffic on the machine at this time. >> >> Immediately, I noticed: >> >> host: /tmp> grep -i logserver rsyslog.queue.debug | wc -l >> 173 >> host: /tmp> grep -i logserver rsyslog.noqueue.debug | wc -l >> 142 >> >> As I suspected, more logs were being sent to the logserver with queuing >> turned on (ie. they weren't getting filtered). I've posted the debug >> logs here: >> http://spark.colorado.edu/rsyslog/ >> >> I am somewhat new to rsyslog, and I've certainly never trolled through >> a >> debug file. I may have a chance to take a look tonight, but given my >> familiarity with the software it may be difficult to zero in on >> anything. >> >> Thanks! >> >> -- Eric >> >> Rainer Gerhards wrote: >>> Could you create a debug log with such a message? It should tell us >> what is >>> going on. Instructions: >>> >>> http://www.rsyslog.com/doc/troubleshoot.html >>> >>> Rainer >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >>>> Sent: Thursday, February 10, 2011 7:11 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Queuing subsystem and message filtering >>>> >>>> Oh, sorry my mail client must have munged those lines. I am getting >> a >>>> clean start-up and shutdown: >>>> >>>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >>>> swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] >>>> exiting >>>> on signal 15. >>>> Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. >>>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" >>>> swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] >> start >>>> Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg >>>> started. >>>> >>>> Here is a better representation of it: >>>> >>>> http://pastebin.com/M6Lwseqr >>>> >>>> the two '/var/log/everything_to_send' where being used for >> debugging. I >>>> actually stacked both of those lines directly above the >>>> @logserver.colorado.edu line, and that's when I noticed a >> discrepancy >>>> between what was getting logged to everything_to_send and >>>> everything_to_send2. Then I bumped those lines back up above the >>>> queuing >>>> directives and observed that debug (and a bunch of other filtered >>>> messages) were making it to logserver.colorado.edu. >>>> >>>> -- Eric >>>> >>>> Rainer Gerhards wrote: >>>> >>>>> HAve you checked for any error messages from rsyslogd inside your >>>>> >>>> logs (do >>>> >>>>> you even log them -- many distros don't do by default :-(( ) >>>>> >>>>> I am asking because the snippet looks syntactically wrong (then on >> a >>>> separate >>>> >>>>> line). >>>>> >>>>> Rainer >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller >>>>>> Sent: Thursday, February 10, 2011 4:51 PM >>>>>> To: rsyslog at lists.adiscon.com >>>>>> Subject: [rsyslog] Queuing subsystem and message filtering >>>>>> >>>>>> Hello list, >>>>>> >>>>>> I recently tried out message queuing using rsyslog-5.6.3. >>>>>> >>>>>> Take the following config snippet: >>>>>> >>>>>> ... >>>>>> # Throw out all messages with debug severity before we log to the >>>>>> network >>>>>> if $syslogseverity-text == 'debug' >>>>>> then ~ >>>>>> >>>>>> $WorkDirectory /var/spool/rsyslog # location for work >> (spool) >>>>>> files >>>>>> $ActionQueueType LinkedList # use asynchronous >>>>>> >>>> processing >>>> >>>>>> $ActionQueueFileName logserver-queue # set file name, also >>>>>> >>>> enables >>>> >>>>>> disk mode >>>>>> $ActionResumeRetryCount -1 # infinite retries on >> insert >>>>>> failure >>>>>> $ActionQueueSaveOnShutdown on # save in-memory data if >>>>>> rsyslog >>>>>> shuts down >>>>>> >>>>>> # Log anything that hasn't been specifically filtered out with '~' >>>>>> >>>> to >>>> >>>>>> logserver >>>>>> *.* @logserver.colorado.edu >>>>>> ... >>>>>> >>>>>> In this scenario I would assume that all messages with the >> severity >>>> of >>>> >>>>>> debug don't get logged over the network to >> 'logserver.colorado.edu'. >>>>>> But >>>>>> they do. If I comment all 5 of the directives that activate >> queuing, >>>>>> the >>>>>> debug messages are no longer logged over the network. >>>>>> >>>>>> Is this by design, and if so could someone explain why? Is this a >>>>>> >>>> bug? >>>> >>>>>> Do I have a configuration problem? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Eric Schoeller >>>>>> University of Colorado, Boulder >>>>>> Information Technology Services >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com From rgerhards at hq.adiscon.com Fri Feb 11 16:15:47 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Feb 2011 16:15:47 +0100 Subject: [rsyslog] Queuing subsystem and message filtering In-Reply-To: <4D55517D.90506@users.sourceforge.net> References: <4D54095F.6050906@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDBFD@GRFEXC.intern.adiscon.com><4D542A25.3050006@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC07@GRFEXC.intern.adiscon.com><4D548734.5090902@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC0C@GRFEXC.intern.adiscon.com> <4D55517D.90506@users.sourceforge.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC31@GRFEXC.intern.adiscon.com> ok, thanks will dig into it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > Sent: Friday, February 11, 2011 4:11 PM > To: rsyslog-users > Subject: Re: [rsyslog] Queuing subsystem and message filtering > > Yes you are right, I meant to say disk queuing. > > -- Eric > > On 02/10/11 23:29, Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >> Sent: Friday, February 11, 2011 1:48 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Queuing subsystem and message filtering > >> > >> Okay, > >> > >> I started up rsyslog with debugging twice. Once with queuing turned > on, > >> once without. > > What exactly do you mean with "queing turned on"? You mean disk queue > > enabled, right? I am asking because rsyslog always queues, but via > different > > methods. I'd like to be sure to look at the right thing (and, yes, I > could > > extract that from the debug log as well but I'd like me to save that > > time...). > > > > Thanks, > > Rainer > > > >> I immediately ran the test script that sends one message > >> to each facility at each severity and then killed the rsyslog daemon > >> ... > >> so the results between the two *should* be fairly identical, there > is > >> minimal additional syslog traffic on the machine at this time. > >> > >> Immediately, I noticed: > >> > >> host: /tmp> grep -i logserver rsyslog.queue.debug | wc -l > >> 173 > >> host: /tmp> grep -i logserver rsyslog.noqueue.debug | wc -l > >> 142 > >> > >> As I suspected, more logs were being sent to the logserver with > queuing > >> turned on (ie. they weren't getting filtered). I've posted the debug > >> logs here: > >> http://spark.colorado.edu/rsyslog/ > >> > >> I am somewhat new to rsyslog, and I've certainly never trolled > through > >> a > >> debug file. I may have a chance to take a look tonight, but given my > >> familiarity with the software it may be difficult to zero in on > >> anything. > >> > >> Thanks! > >> > >> -- Eric > >> > >> Rainer Gerhards wrote: > >>> Could you create a debug log with such a message? It should tell us > >> what is > >>> going on. Instructions: > >>> > >>> http://www.rsyslog.com/doc/troubleshoot.html > >>> > >>> Rainer > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >>>> Sent: Thursday, February 10, 2011 7:11 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] Queuing subsystem and message filtering > >>>> > >>>> Oh, sorry my mail client must have munged those lines. I am > getting > >> a > >>>> clean start-up and shutdown: > >>>> > >>>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > >>>> swVersion="5.6.3" x-pid="15452" x-info="http://www.rsyslog.com"] > >>>> exiting > >>>> on signal 15. > >>>> Feb 10 11:02:07 host kernel: Kernel logging (proc) stopped. > >>>> Feb 10 11:02:07 host rsyslogd: [origin software="rsyslogd" > >>>> swVersion="5.6.3" x-pid="15884" x-info="http://www.rsyslog.com"] > >> start > >>>> Feb 10 11:02:07 host kernel: imklog 5.6.3, log source = /proc/kmsg > >>>> started. > >>>> > >>>> Here is a better representation of it: > >>>> > >>>> http://pastebin.com/M6Lwseqr > >>>> > >>>> the two '/var/log/everything_to_send' where being used for > >> debugging. I > >>>> actually stacked both of those lines directly above the > >>>> @logserver.colorado.edu line, and that's when I noticed a > >> discrepancy > >>>> between what was getting logged to everything_to_send and > >>>> everything_to_send2. Then I bumped those lines back up above the > >>>> queuing > >>>> directives and observed that debug (and a bunch of other filtered > >>>> messages) were making it to logserver.colorado.edu. > >>>> > >>>> -- Eric > >>>> > >>>> Rainer Gerhards wrote: > >>>> > >>>>> HAve you checked for any error messages from rsyslogd inside your > >>>>> > >>>> logs (do > >>>> > >>>>> you even log them -- many distros don't do by default :-(( ) > >>>>> > >>>>> I am asking because the snippet looks syntactically wrong (then > on > >> a > >>>> separate > >>>> > >>>>> line). > >>>>> > >>>>> Rainer > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Eric Schoeller > >>>>>> Sent: Thursday, February 10, 2011 4:51 PM > >>>>>> To: rsyslog at lists.adiscon.com > >>>>>> Subject: [rsyslog] Queuing subsystem and message filtering > >>>>>> > >>>>>> Hello list, > >>>>>> > >>>>>> I recently tried out message queuing using rsyslog-5.6.3. > >>>>>> > >>>>>> Take the following config snippet: > >>>>>> > >>>>>> ... > >>>>>> # Throw out all messages with debug severity before we log to > the > >>>>>> network > >>>>>> if $syslogseverity-text == 'debug' > >>>>>> then ~ > >>>>>> > >>>>>> $WorkDirectory /var/spool/rsyslog # location for work > >> (spool) > >>>>>> files > >>>>>> $ActionQueueType LinkedList # use asynchronous > >>>>>> > >>>> processing > >>>> > >>>>>> $ActionQueueFileName logserver-queue # set file name, also > >>>>>> > >>>> enables > >>>> > >>>>>> disk mode > >>>>>> $ActionResumeRetryCount -1 # infinite retries on > >> insert > >>>>>> failure > >>>>>> $ActionQueueSaveOnShutdown on # save in-memory data if > >>>>>> rsyslog > >>>>>> shuts down > >>>>>> > >>>>>> # Log anything that hasn't been specifically filtered out with > '~' > >>>>>> > >>>> to > >>>> > >>>>>> logserver > >>>>>> *.* @logserver.colorado.edu > >>>>>> ... > >>>>>> > >>>>>> In this scenario I would assume that all messages with the > >> severity > >>>> of > >>>> > >>>>>> debug don't get logged over the network to > >> 'logserver.colorado.edu'. > >>>>>> But > >>>>>> they do. If I comment all 5 of the directives that activate > >> queuing, > >>>>>> the > >>>>>> debug messages are no longer logged over the network. > >>>>>> > >>>>>> Is this by design, and if so could someone explain why? Is this > a > >>>>>> > >>>> bug? > >>>> > >>>>>> Do I have a configuration problem? > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> Eric Schoeller > >>>>>> University of Colorado, Boulder > >>>>>> Information Technology Services > >>>>>> > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sat Feb 12 10:51:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 12 Feb 2011 10:51:51 +0100 Subject: [rsyslog] how can a parser insert data into a message In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDC0E@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DDC11@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC3B@GRFEXC.intern.adiscon.com> Sorry, I missed this mail initially. To cut a long story short, the sanitizer works much like you describe ;) Perquisite: the message buffer is exactly as long as needed (no extra space available). The sanitizer works in two phases: 1. check if something needs to be done If not, terminate. Otherwise: 2. alloc new buffer, sanitize, replace old buffer I think the Perquisite is not given if messages are so small that they fit into the msg-object provided buffer. In that case, the sanitzer could probably be optimized :) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 11, 2011 8:42 AM > To: rsyslog-users > Subject: Re: [rsyslog] how can a parser insert data into a message > > On Fri, 11 Feb 2011, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> > >> On Fri, 11 Feb 2011, Rainer Gerhards wrote: > >> > >>> Have a look at ./runtime/parser.c, function SanitizeMsg. It builds > a > >>> new buffer and uses MsgSetRawMsg to set the new buffer. > MsgSetRawMsg > >>> handles the "dirty" internals of message object buffer > manipulation. > >>> > >>> Note that it may be quicker to manipulate the buffer pointers > >>> yourself. But then you must be very careful. MsgSetRawMsg should > >>> provide the necessary hints. The thing to keep on your mind is that > up > >>> to a certain message length, a buffer is used from the msg object > >>> itself (thus saving one malloc/free call) whereas for larger size > >>> messages, memory is allocated. You need to keep that straight > during > >>> manipulation. > >> > >> I'll look at it and see how hard it is to separate these two cases. > >> thanks > >> for the pointer here. > > > > Just let me add that I did find it of questionable value to try avoid > the > > malloc here. At least in the sanitization problem, this would have > resulted > > in very complex code. And while saving memory writes and calls to the > malloc > > subsystem is useful, I thought that it would not have brought much > benefit in > > that case. Depending on what you intend to do (well-defined insert at > late > > point) things may be different, though. > > My initial thought is something along the following > > 1. find out how much space is available in whatever buffer the message > is > in (potentially 0 if the buffer is exactly the right size) > > document what needs to happen to adjust how much of the buffer is used > (I've already figured out some of this with the existing parser > modules) > > 2. if there is not enough space, document what the process is to > allocate > a new buffer and make the system use it. > > at this point it should be fairly straightforward to write a routine to > do > something along the lines of 'make sure I have enough space in the > buffer > to add X characters' and have it either return immediatly if there's > enough space or allocate the larger buffer if needed and return after > doing that. > > there will be some things that will need to be documented as side > effects > (pointers into the existing message may be invalid at that point, > including values in the msg structure) > > this could be mis-used (running this routine for every control > character > found could result in many malloc/free pairs for example), and so > examples > will need to be given of doing a 2-pass routine, pass 1 to figure out > what > you want to do, and then make sure there's enough space and do pass 2 > to > modify the buffer as needed. > > Using this for sanitizing would still be slightly less efficient than > the > approach you probably use now (allocate a new buffer, copy things into > it > as you go to construct a new message, then set the message into the > structure), but probably not by more than two copies of the text. As a > result, it may be that the result will be enough cleaner to be worth > the > cost. I'm thinking that the new routine would be to copy the text from > the > old buffer to the new one, then copy everything after your first insert > to > the end of the buffer. after that you are copying data from late in the > buffer to earlier in the buffer, which may even be faster than copying > small amounts of data from one buffer to another as it may result in > better cache behavior. > > in fact, this pattern is probably common enough to make it a routine > itself > > something like > > int InsertIntoRawMsg(int offset, int count) > > inserts at least count spaces into the message at position offset from > the beginning of the message, returns the number of spaces actually > inserted (may be more than the number requested) > > or would it be better to return the number of extra characters > available > in the buffer after the end of the string? > > I figure error checking on the return is not needed because if it can't > allocate the space we need to bail out (with whatever rsyslog does when > it > runs out of memory, probably aborting the message entirely) > > David Lang > > > Rainer > > > >> > >>> As a side-note, it would probably be useful if you could take some > >>> bullet points on how to modify things, so that others can find that > >>> information in the case they want to do that themselves. Could go > to > >>> the wiki or I could include it in the doc set. Just a suggestion, > >>> though... > >> > >> I'll see what I can do. > >> > >> David Lang > >> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >>>> Sent: Friday, February 11, 2011 5:38 AM > >>>> To: rsyslog-users > >>>> Subject: [rsyslog] how can a parser insert data into a message > >>>> > >>>> the various parser modules that I've submitted are all removing > data > >>>> from > >>>> the log message or overwriting the data in place. > >>>> > >>>> But I've now run across a situation where I need to insert > >> information > >>>> into the message. I know that this can be done because the > >> sanitizing > >>>> call > >>>> does exactly this. I am assuming that this is doing something like > >>>> allocating a new string and copying the data into the new string. > >>>> > >>>> the concern is how to do this in a way that will survive the exit > >> from > >>>> the > >>>> module, not confuse any of the many pointers or sizes that are > >>>> involved, > >>>> and make sure everything is properly freed afterwords. > >>>> > >>>> should I just search for the sanitizing routine and copy what it > >> does > >>>> (and > >>>> can you point me at it?), or do you want me to wait until you have > >> time > >>>> to > >>>> write something up on this? > >>>> > >>>> David Lang > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From toddmichael at gmail.com Sun Feb 13 01:22:16 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Sat, 12 Feb 2011 16:22:16 -0800 Subject: [rsyslog] Reliable forwarding Not Working Message-ID: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> I configured reliable forwarding in accordance with instructions here: http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html Version: rsyslog-3.22.1-3.el5_5.1 Configuration: # forward to remote host, queueing to local disk if host is down and memory fills up # work (spool) files directory $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 # in-memory queue; set for asynchronous processing (?) $ActionQueueType LinkedList # failover queue filename; also enables disk mode $ActionQueueFileName failqueue-loghost2 # infinite retries on insert failure $ActionResumeRetryCount -1 # save in-memory data if rsyslog shuts down $ActionQueueSaveOnShutdown on # remote logging of everything *.* @@loghost2:5140 I wanted to test its functionality before going into production. First, I used iptables to block access to the syslog port on the central syslog (syslog-ng) server, simulating a down syslog server: # on loghost2 /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT --reject-with icmp-admin-prohibited I then ran logger through a loop to start creating a pile of messages on the rsyslog client: for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is a test $i"; done I ran this loop twice in an effort to sufficiently fill up memory and initiate dump to disk. While this loop was running I verified that memory consumption for the rsylogd process on the client was increasing. It eventually got to this point: root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 /sbin/rsyslogd -c 3 To be honest, I don't know how much memory it will consume before dumping to disk (feel free to school me on this) so I figured I'd keep going until I saw /var/log/rsyslog directory and files created. This never happened and my second iteration stopped at about 600k and I saw some memory fork errors (though they dumped only to standard error, not log, so I lost them (sorry)). Dump to disk having failed, I next wanted to see if rsyslog would at least resume forward to remote host when it came back up (dumping whatever was in memory to central syslog server). I restarted iptables on the syslog server to restore access to the port, but no logs were forwarded from the rsyslog client. Lastly, I restarted rsyslog, hoping that I would see a dump to disk but this failed as well. I'm sure it's something I'm doing incorrectly. Would appreciate some guidance. Who knows, maybe I just need to create the /var/log/rsyslog directory (assumed rsyslog would create it). While I'm waiting for feedback, I'll probably give that a shot. Thanks. Todd From eschoeller at users.sourceforge.net Sun Feb 13 01:26:48 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Sat, 12 Feb 2011 17:26:48 -0700 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> Message-ID: <4D572548.4030907@users.sourceforge.net> I've been playing around with this recently too. You certainly need to create the spool directory. rsyslog won't do it (but like you, I thought it would). Once I did that, I quickly saw spool files being created there while performing tests similar to yours. My issue is still standing, previous filters within the rsyslog configuration aren't honored when you've enabled disk queuing. Everything gets sent to the log server, even messages specifically thrown away earlier with '~'. I'd be interested to see if you experience similar behavior. -- Eric Todd Michael Bushnell wrote: > I configured reliable forwarding in accordance with instructions here: http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html > > Version: rsyslog-3.22.1-3.el5_5.1 > > Configuration: > > # forward to remote host, queueing to local disk if host is down and memory fills up > # work (spool) files directory > $WorkDirectory /var/log/rsyslog > # start forwarding rule - loghost2 > # in-memory queue; set for asynchronous processing (?) > $ActionQueueType LinkedList > # failover queue filename; also enables disk mode > $ActionQueueFileName failqueue-loghost2 > # infinite retries on insert failure > $ActionResumeRetryCount -1 > # save in-memory data if rsyslog shuts down > $ActionQueueSaveOnShutdown on > # remote logging of everything > *.* @@loghost2:5140 > > I wanted to test its functionality before going into production. > > First, I used iptables to block access to the syslog port on the central syslog (syslog-ng) server, simulating a down syslog server: > # on loghost2 > /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT --reject-with icmp-admin-prohibited > > I then ran logger through a loop to start creating a pile of messages on the rsyslog client: > for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is a test $i"; done > > I ran this loop twice in an effort to sufficiently fill up memory and initiate dump to disk. While this loop was running I verified that memory consumption for the rsylogd process on the client was increasing. It eventually got to this point: > root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 /sbin/rsyslogd -c 3 > > To be honest, I don't know how much memory it will consume before dumping to disk (feel free to school me on this) so I figured I'd keep going until I saw /var/log/rsyslog directory and files created. This never happened and my second iteration stopped at about 600k and I saw some memory fork errors (though they dumped only to standard error, not log, so I lost them (sorry)). > > Dump to disk having failed, I next wanted to see if rsyslog would at least resume forward to remote host when it came back up (dumping whatever was in memory to central syslog server). I restarted iptables on the syslog server to restore access to the port, but no logs were forwarded from the rsyslog client. > > Lastly, I restarted rsyslog, hoping that I would see a dump to disk but this failed as well. > > I'm sure it's something I'm doing incorrectly. Would appreciate some guidance. Who knows, maybe I just need to create the /var/log/rsyslog directory (assumed rsyslog would create it). While I'm waiting for feedback, I'll probably give that a shot. Thanks. > > Todd > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From toddmichael at gmail.com Sun Feb 13 01:32:00 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Sat, 12 Feb 2011 16:32:00 -0800 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> Message-ID: Lo and behold, I created /var/log/rsyslog directory, restarted the process as documented earlier and I'm already seeing logs dumped to files in that subdirectory [root at client rsyslog]# pwd /var/log/rsyslog [root at client rsyslog]# ls -al total 6556 drwxr-x--- 2 root wheel 4096 Feb 12 16:29 . drwxr-xr-x 6 root root 4096 Feb 12 16:08 .. -rw------- 1 root root 1048956 Feb 12 16:18 failqueue-loghost2.00000001 -rw------- 1 root root 1048713 Feb 12 16:20 failqueue-loghost2.00000002 -rw------- 1 root root 1048662 Feb 12 16:23 failqueue-loghost2.00000003 -rw------- 1 root root 1048680 Feb 12 16:26 failqueue-loghost2.00000004 -rw------- 1 root root 1048961 Feb 12 16:27 failqueue-loghost2.00000005 -rw------- 1 root root 1048956 Feb 12 16:29 failqueue-loghost2.00000006 -rw------- 1 root root 352328 Feb 12 16:29 failqueue-loghost2.00000007 What a beautiful thing it is. Sorry for the false alarm. On Feb 12, 2011, at 4:22 PM, Todd Michael Bushnell wrote: > I configured reliable forwarding in accordance with instructions here: http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html > > Version: rsyslog-3.22.1-3.el5_5.1 > > Configuration: > > # forward to remote host, queueing to local disk if host is down and memory fills up > # work (spool) files directory > $WorkDirectory /var/log/rsyslog > # start forwarding rule - loghost2 > # in-memory queue; set for asynchronous processing (?) > $ActionQueueType LinkedList > # failover queue filename; also enables disk mode > $ActionQueueFileName failqueue-loghost2 > # infinite retries on insert failure > $ActionResumeRetryCount -1 > # save in-memory data if rsyslog shuts down > $ActionQueueSaveOnShutdown on > # remote logging of everything > *.* @@loghost2:5140 > > I wanted to test its functionality before going into production. > > First, I used iptables to block access to the syslog port on the central syslog (syslog-ng) server, simulating a down syslog server: > # on loghost2 > /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT --reject-with icmp-admin-prohibited > > I then ran logger through a loop to start creating a pile of messages on the rsyslog client: > for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is a test $i"; done > > I ran this loop twice in an effort to sufficiently fill up memory and initiate dump to disk. While this loop was running I verified that memory consumption for the rsylogd process on the client was increasing. It eventually got to this point: > root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 /sbin/rsyslogd -c 3 > > To be honest, I don't know how much memory it will consume before dumping to disk (feel free to school me on this) so I figured I'd keep going until I saw /var/log/rsyslog directory and files created. This never happened and my second iteration stopped at about 600k and I saw some memory fork errors (though they dumped only to standard error, not log, so I lost them (sorry)). > > Dump to disk having failed, I next wanted to see if rsyslog would at least resume forward to remote host when it came back up (dumping whatever was in memory to central syslog server). I restarted iptables on the syslog server to restore access to the port, but no logs were forwarded from the rsyslog client. > > Lastly, I restarted rsyslog, hoping that I would see a dump to disk but this failed as well. > > I'm sure it's something I'm doing incorrectly. Would appreciate some guidance. Who knows, maybe I just need to create the /var/log/rsyslog directory (assumed rsyslog would create it). While I'm waiting for feedback, I'll probably give that a shot. Thanks. > > Todd > From toddmichael at gmail.com Sun Feb 13 01:41:01 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Sat, 12 Feb 2011 16:41:01 -0800 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <4D572548.4030907@users.sourceforge.net> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> <4D572548.4030907@users.sourceforge.net> Message-ID: Eric, Saw your response when I sent my follow-up. Thanks for the info - creating the directory was the solution. I've not gotten to where you're at as all my tests were with messages that dumped to the same location, but I'll certainly put together a test of your scenario and see what I get. So to clarify: your have your loghost directive AFTER some exclusion rules and all works fine until disk queuing comes into play? Essentially, when disk queuing begins, rsyslog dumps everything to the queue, regardless of rules? Todd On Feb 12, 2011, at 4:26 PM, Eric Schoeller wrote: > I've been playing around with this recently too. You certainly need to create the spool directory. rsyslog won't do it (but like you, I thought it would). Once I did that, I quickly saw spool files being created there while performing tests similar to yours. > > My issue is still standing, previous filters within the rsyslog configuration aren't honored when you've enabled disk queuing. Everything gets sent to the log server, even messages specifically thrown away earlier with '~'. I'd be interested to see if you experience similar behavior. > > -- Eric > > Todd Michael Bushnell wrote: >> I configured reliable forwarding in accordance with instructions here: http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html >> >> Version: rsyslog-3.22.1-3.el5_5.1 >> >> Configuration: >> >> # forward to remote host, queueing to local disk if host is down and memory fills up >> # work (spool) files directory >> $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 >> # in-memory queue; set for asynchronous processing (?) >> $ActionQueueType LinkedList >> # failover queue filename; also enables disk mode >> $ActionQueueFileName failqueue-loghost2 >> # infinite retries on insert failure >> $ActionResumeRetryCount -1 >> # save in-memory data if rsyslog shuts down >> $ActionQueueSaveOnShutdown on >> # remote logging of everything >> *.* @@loghost2:5140 >> >> I wanted to test its functionality before going into production. >> >> First, I used iptables to block access to the syslog port on the central syslog (syslog-ng) server, simulating a down syslog server: >> # on loghost2 >> /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT --reject-with icmp-admin-prohibited >> >> I then ran logger through a loop to start creating a pile of messages on the rsyslog client: >> for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is a test $i"; done >> >> I ran this loop twice in an effort to sufficiently fill up memory and initiate dump to disk. While this loop was running I verified that memory consumption for the rsylogd process on the client was increasing. It eventually got to this point: >> root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 /sbin/rsyslogd -c 3 >> >> To be honest, I don't know how much memory it will consume before dumping to disk (feel free to school me on this) so I figured I'd keep going until I saw /var/log/rsyslog directory and files created. This never happened and my second iteration stopped at about 600k and I saw some memory fork errors (though they dumped only to standard error, not log, so I lost them (sorry)). >> >> Dump to disk having failed, I next wanted to see if rsyslog would at least resume forward to remote host when it came back up (dumping whatever was in memory to central syslog server). I restarted iptables on the syslog server to restore access to the port, but no logs were forwarded from the rsyslog client. >> >> Lastly, I restarted rsyslog, hoping that I would see a dump to disk but this failed as well. >> I'm sure it's something I'm doing incorrectly. Would appreciate some guidance. Who knows, maybe I just need to create the /var/log/rsyslog directory (assumed rsyslog would create it). While I'm waiting for feedback, I'll probably give that a shot. Thanks. >> Todd >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Mon Feb 14 09:12:39 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 14 Feb 2011 00:12:39 -0800 (PST) Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <4D572548.4030907@users.sourceforge.net> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com> <4D572548.4030907@users.sourceforge.net> Message-ID: why don't you define the filters after you define the queue type? given the way that rsyslog interprets the config file, I can easily see how it could loose the effect of filters if you define a queue type after the filters in the config file. Defining a different queue type changes somuch of what rsyslog is doing with the messages that I would not be at all surprised if the process of setting that up lost any previously defined fulter rules. I would also be surprised if that was a order that had ever been tested. David Lang On Sat, 12 Feb 2011, Eric Schoeller wrote: > I've been playing around with this recently too. You certainly need to create > the spool directory. rsyslog won't do it (but like you, I thought it would). > Once I did that, I quickly saw spool files being created there while > performing tests similar to yours. > > My issue is still standing, previous filters within the rsyslog configuration > aren't honored when you've enabled disk queuing. Everything gets sent to the > log server, even messages specifically thrown away earlier with '~'. I'd be > interested to see if you experience similar behavior. > > -- Eric > > Todd Michael Bushnell wrote: >> I configured reliable forwarding in accordance with instructions here: >> http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html >> >> Version: rsyslog-3.22.1-3.el5_5.1 >> >> Configuration: >> >> # forward to remote host, queueing to local disk if host is down and memory >> fills up >> # work (spool) files directory >> $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 >> # in-memory queue; set for asynchronous processing (?) >> $ActionQueueType LinkedList >> # failover queue filename; also enables disk mode >> $ActionQueueFileName failqueue-loghost2 >> # infinite retries on insert failure >> $ActionResumeRetryCount -1 >> # save in-memory data if rsyslog shuts down >> $ActionQueueSaveOnShutdown on >> # remote logging of everything >> *.* @@loghost2:5140 >> >> I wanted to test its functionality before going into production. >> >> First, I used iptables to block access to the syslog port on the central >> syslog (syslog-ng) server, simulating a down syslog server: >> # on loghost2 >> /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT >> --reject-with icmp-admin-prohibited >> >> I then ran logger through a loop to start creating a pile of messages on >> the rsyslog client: >> for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is a test >> $i"; done >> >> I ran this loop twice in an effort to sufficiently fill up memory and >> initiate dump to disk. While this loop was running I verified that memory >> consumption for the rsylogd process on the client was increasing. It >> eventually got to this point: >> root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 >> /sbin/rsyslogd -c 3 >> >> To be honest, I don't know how much memory it will consume before dumping >> to disk (feel free to school me on this) so I figured I'd keep going until >> I saw /var/log/rsyslog directory and files created. This never happened >> and my second iteration stopped at about 600k and I saw some memory fork >> errors (though they dumped only to standard error, not log, so I lost them >> (sorry)). >> >> Dump to disk having failed, I next wanted to see if rsyslog would at least >> resume forward to remote host when it came back up (dumping whatever was in >> memory to central syslog server). I restarted iptables on the syslog >> server to restore access to the port, but no logs were forwarded from the >> rsyslog client. >> >> Lastly, I restarted rsyslog, hoping that I would see a dump to disk but >> this failed as well. >> I'm sure it's something I'm doing incorrectly. Would appreciate some >> guidance. Who knows, maybe I just need to create the /var/log/rsyslog >> directory (assumed rsyslog would create it). While I'm waiting for >> feedback, I'll probably give that a shot. Thanks. >> Todd >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Feb 14 09:51:24 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Feb 2011 09:51:24 +0100 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com><4D572548.4030907@users.sourceforge.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC46@GRFEXC.intern.adiscon.com> Mhhh... am I overlooking something? As far as I can see, he first defines the queue parameters and then a (single, as necessary) action. Do you see more? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, February 14, 2011 9:13 AM > To: eschoeller at users.sourceforge.net; rsyslog-users > Subject: Re: [rsyslog] Reliable forwarding Not Working > > why don't you define the filters after you define the queue type? > > given the way that rsyslog interprets the config file, I can easily see > how it could loose the effect of filters if you define a queue type > after > the filters in the config file. Defining a different queue type changes > somuch of what rsyslog is doing with the messages that I would not be > at > all surprised if the process of setting that up lost any previously > defined fulter rules. > > I would also be surprised if that was a order that had ever been > tested. > > David Lang > > On Sat, 12 Feb 2011, Eric Schoeller wrote: > > > I've been playing around with this recently too. You certainly need > to create > > the spool directory. rsyslog won't do it (but like you, I thought it > would). > > Once I did that, I quickly saw spool files being created there while > > performing tests similar to yours. > > > > My issue is still standing, previous filters within the rsyslog > configuration > > aren't honored when you've enabled disk queuing. Everything gets sent > to the > > log server, even messages specifically thrown away earlier with '~'. > I'd be > > interested to see if you experience similar behavior. > > > > -- Eric > > > > Todd Michael Bushnell wrote: > >> I configured reliable forwarding in accordance with instructions > here: > >> http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html > >> > >> Version: rsyslog-3.22.1-3.el5_5.1 > >> > >> Configuration: > >> > >> # forward to remote host, queueing to local disk if host is down and > memory > >> fills up > >> # work (spool) files directory > >> $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 > >> # in-memory queue; set for asynchronous processing (?) > >> $ActionQueueType LinkedList > >> # failover queue filename; also enables disk mode > >> $ActionQueueFileName failqueue-loghost2 > >> # infinite retries on insert failure > >> $ActionResumeRetryCount -1 > >> # save in-memory data if rsyslog shuts down > >> $ActionQueueSaveOnShutdown on > >> # remote logging of everything > >> *.* @@loghost2:5140 > >> > >> I wanted to test its functionality before going into production. > >> > >> First, I used iptables to block access to the syslog port on the > central > >> syslog (syslog-ng) server, simulating a down syslog server: > >> # on loghost2 > >> /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT > >> --reject-with icmp-admin-prohibited > >> > >> I then ran logger through a loop to start creating a pile of > messages on > >> the rsyslog client: > >> for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is > a test > >> $i"; done > >> > >> I ran this loop twice in an effort to sufficiently fill up memory > and > >> initiate dump to disk. While this loop was running I verified that > memory > >> consumption for the rsylogd process on the client was increasing. > It > >> eventually got to this point: > >> root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 > >> /sbin/rsyslogd -c 3 > >> > >> To be honest, I don't know how much memory it will consume before > dumping > >> to disk (feel free to school me on this) so I figured I'd keep going > until > >> I saw /var/log/rsyslog directory and files created. This never > happened > >> and my second iteration stopped at about 600k and I saw some memory > fork > >> errors (though they dumped only to standard error, not log, so I > lost them > >> (sorry)). > >> > >> Dump to disk having failed, I next wanted to see if rsyslog would at > least > >> resume forward to remote host when it came back up (dumping whatever > was in > >> memory to central syslog server). I restarted iptables on the > syslog > >> server to restore access to the port, but no logs were forwarded > from the > >> rsyslog client. > >> > >> Lastly, I restarted rsyslog, hoping that I would see a dump to disk > but > >> this failed as well. > >> I'm sure it's something I'm doing incorrectly. Would appreciate > some > >> guidance. Who knows, maybe I just need to create the > /var/log/rsyslog > >> directory (assumed rsyslog would create it). While I'm waiting for > >> feedback, I'll probably give that a shot. Thanks. > >> Todd > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Mon Feb 14 10:03:10 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 14 Feb 2011 01:03:10 -0800 (PST) Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC46@GRFEXC.intern.adiscon.com> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com><4D572548.4030907@users.sourceforge.net> <9B6E2A8877C38245BFB15CC491A11DA71DDC46@GRFEXC.intern.adiscon.com> Message-ID: I could be mistaken here, but as I am currently understanding it (and without trying to go back through the mailing list archives) it sounds like what he is doing is some config stuff filter to throw away some log messages define the queue type filter to write log messages to disk and he is saying that if he does not define the queue type, everything works, but once he defines the queue type it ignores the filters that are supposed to throw away some log messages. I'm saying that I suspect that if he rearranged the config file to be some config stuff define the queue type filter to throw away some log messages filter to write log messages to disk I would expect it to work better. David Lang On Mon, 14 Feb 2011, Rainer Gerhards wrote: > Mhhh... am I overlooking something? As far as I can see, he first defines the > queue parameters and then a (single, as necessary) action. Do you see more? > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Monday, February 14, 2011 9:13 AM >> To: eschoeller at users.sourceforge.net; rsyslog-users >> Subject: Re: [rsyslog] Reliable forwarding Not Working >> >> why don't you define the filters after you define the queue type? >> >> given the way that rsyslog interprets the config file, I can easily see >> how it could loose the effect of filters if you define a queue type >> after >> the filters in the config file. Defining a different queue type changes >> somuch of what rsyslog is doing with the messages that I would not be >> at >> all surprised if the process of setting that up lost any previously >> defined fulter rules. >> >> I would also be surprised if that was a order that had ever been >> tested. >> >> David Lang >> >> On Sat, 12 Feb 2011, Eric Schoeller wrote: >> >>> I've been playing around with this recently too. You certainly need >> to create >>> the spool directory. rsyslog won't do it (but like you, I thought it >> would). >>> Once I did that, I quickly saw spool files being created there while >>> performing tests similar to yours. >>> >>> My issue is still standing, previous filters within the rsyslog >> configuration >>> aren't honored when you've enabled disk queuing. Everything gets sent >> to the >>> log server, even messages specifically thrown away earlier with '~'. >> I'd be >>> interested to see if you experience similar behavior. >>> >>> -- Eric >>> >>> Todd Michael Bushnell wrote: >>>> I configured reliable forwarding in accordance with instructions >> here: >>>> http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html >>>> >>>> Version: rsyslog-3.22.1-3.el5_5.1 >>>> >>>> Configuration: >>>> >>>> # forward to remote host, queueing to local disk if host is down and >> memory >>>> fills up >>>> # work (spool) files directory >>>> $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 >>>> # in-memory queue; set for asynchronous processing (?) >>>> $ActionQueueType LinkedList >>>> # failover queue filename; also enables disk mode >>>> $ActionQueueFileName failqueue-loghost2 >>>> # infinite retries on insert failure >>>> $ActionResumeRetryCount -1 >>>> # save in-memory data if rsyslog shuts down >>>> $ActionQueueSaveOnShutdown on >>>> # remote logging of everything >>>> *.* @@loghost2:5140 >>>> >>>> I wanted to test its functionality before going into production. >>>> >>>> First, I used iptables to block access to the syslog port on the >> central >>>> syslog (syslog-ng) server, simulating a down syslog server: >>>> # on loghost2 >>>> /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT >>>> --reject-with icmp-admin-prohibited >>>> >>>> I then ran logger through a loop to start creating a pile of >> messages on >>>> the rsyslog client: >>>> for i in {1..1000000}; do logger -t tmbtest -p local1.info "this is >> a test >>>> $i"; done >>>> >>>> I ran this loop twice in an effort to sufficiently fill up memory >> and >>>> initiate dump to disk. While this loop was running I verified that >> memory >>>> consumption for the rsylogd process on the client was increasing. >> It >>>> eventually got to this point: >>>> root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 >>>> /sbin/rsyslogd -c 3 >>>> >>>> To be honest, I don't know how much memory it will consume before >> dumping >>>> to disk (feel free to school me on this) so I figured I'd keep going >> until >>>> I saw /var/log/rsyslog directory and files created. This never >> happened >>>> and my second iteration stopped at about 600k and I saw some memory >> fork >>>> errors (though they dumped only to standard error, not log, so I >> lost them >>>> (sorry)). >>>> >>>> Dump to disk having failed, I next wanted to see if rsyslog would at >> least >>>> resume forward to remote host when it came back up (dumping whatever >> was in >>>> memory to central syslog server). I restarted iptables on the >> syslog >>>> server to restore access to the port, but no logs were forwarded >> from the >>>> rsyslog client. >>>> >>>> Lastly, I restarted rsyslog, hoping that I would see a dump to disk >> but >>>> this failed as well. >>>> I'm sure it's something I'm doing incorrectly. Would appreciate >> some >>>> guidance. Who knows, maybe I just need to create the >> /var/log/rsyslog >>>> directory (assumed rsyslog would create it). While I'm waiting for >>>> feedback, I'll probably give that a shot. Thanks. >>>> Todd >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Feb 14 10:36:22 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Feb 2011 10:36:22 +0100 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com><4D572548.4030907@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC46@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC4B@GRFEXC.intern.adiscon.com> ah, maybe I misunderstood it. So it is good to getting this straigt. Anyhow... Queue Type for action queues is *per action*. So it really doesn't matter in regard to filters where the queue is defined. So in your sample (a) some config stuff (b) filter to throw away some log messages (c_1) define the queue type (c_2) filter to write log messages to disk Both c_1 and c_2 are the same action. Most importantly, c_1 does not modify anything but the next action immediately following. That means if you have *.* ~ $Action... *.* /logfile logfile should never receive anything, no matter what the queue def is. This is for action queues. For the main queue, it doesn't matter where the typedef is. The queue params are pulled once when the queue is generated and that is AFTER all config statements have been read. I'll try to dig into that use case, but I finally managed to make me very late on my paper deadline. I will probably not be able to do anything but writing this week... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, February 14, 2011 10:03 AM > To: rsyslog-users > Subject: Re: [rsyslog] Reliable forwarding Not Working > > I could be mistaken here, but as I am currently understanding it (and > without trying to go back through the mailing list archives) it sounds > like what he is doing is > > some config stuff > > filter to throw away some log messages > > define the queue type > > filter to write log messages to disk > > > and he is saying that if he does not define the queue type, everything > works, but once he defines the queue type it ignores the filters that > are > supposed to throw away some log messages. > > I'm saying that I suspect that if he rearranged the config file to be > > some config stuff > > define the queue type > > filter to throw away some log messages > > filter to write log messages to disk > > I would expect it to work better. > > David Lang > > On Mon, 14 Feb 2011, Rainer Gerhards wrote: > > > Mhhh... am I overlooking something? As far as I can see, he first > defines the > > queue parameters and then a (single, as necessary) action. Do you see > more? > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Monday, February 14, 2011 9:13 AM > >> To: eschoeller at users.sourceforge.net; rsyslog-users > >> Subject: Re: [rsyslog] Reliable forwarding Not Working > >> > >> why don't you define the filters after you define the queue type? > >> > >> given the way that rsyslog interprets the config file, I can easily > see > >> how it could loose the effect of filters if you define a queue type > >> after > >> the filters in the config file. Defining a different queue type > changes > >> somuch of what rsyslog is doing with the messages that I would not > be > >> at > >> all surprised if the process of setting that up lost any previously > >> defined fulter rules. > >> > >> I would also be surprised if that was a order that had ever been > >> tested. > >> > >> David Lang > >> > >> On Sat, 12 Feb 2011, Eric Schoeller wrote: > >> > >>> I've been playing around with this recently too. You certainly need > >> to create > >>> the spool directory. rsyslog won't do it (but like you, I thought > it > >> would). > >>> Once I did that, I quickly saw spool files being created there > while > >>> performing tests similar to yours. > >>> > >>> My issue is still standing, previous filters within the rsyslog > >> configuration > >>> aren't honored when you've enabled disk queuing. Everything gets > sent > >> to the > >>> log server, even messages specifically thrown away earlier with > '~'. > >> I'd be > >>> interested to see if you experience similar behavior. > >>> > >>> -- Eric > >>> > >>> Todd Michael Bushnell wrote: > >>>> I configured reliable forwarding in accordance with instructions > >> here: > >>>> http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html > >>>> > >>>> Version: rsyslog-3.22.1-3.el5_5.1 > >>>> > >>>> Configuration: > >>>> > >>>> # forward to remote host, queueing to local disk if host is down > and > >> memory > >>>> fills up > >>>> # work (spool) files directory > >>>> $WorkDirectory /var/log/rsyslog # start forwarding rule - loghost2 > >>>> # in-memory queue; set for asynchronous processing (?) > >>>> $ActionQueueType LinkedList > >>>> # failover queue filename; also enables disk mode > >>>> $ActionQueueFileName failqueue-loghost2 > >>>> # infinite retries on insert failure > >>>> $ActionResumeRetryCount -1 > >>>> # save in-memory data if rsyslog shuts down > >>>> $ActionQueueSaveOnShutdown on > >>>> # remote logging of everything > >>>> *.* @@loghost2:5140 > >>>> > >>>> I wanted to test its functionality before going into production. > >>>> > >>>> First, I used iptables to block access to the syslog port on the > >> central > >>>> syslog (syslog-ng) server, simulating a down syslog server: > >>>> # on loghost2 > >>>> /sbin/iptables -I INPUT -p tcp --destination-port 5140 -j REJECT > >>>> --reject-with icmp-admin-prohibited > >>>> > >>>> I then ran logger through a loop to start creating a pile of > >> messages on > >>>> the rsyslog client: > >>>> for i in {1..1000000}; do logger -t tmbtest -p local1.info "this > is > >> a test > >>>> $i"; done > >>>> > >>>> I ran this loop twice in an effort to sufficiently fill up memory > >> and > >>>> initiate dump to disk. While this loop was running I verified > that > >> memory > >>>> consumption for the rsylogd process on the client was increasing. > >> It > >>>> eventually got to this point: > >>>> root 20263 0.2 77.8 2537008 1603712 ? Sl Feb10 6:09 > >>>> /sbin/rsyslogd -c 3 > >>>> > >>>> To be honest, I don't know how much memory it will consume before > >> dumping > >>>> to disk (feel free to school me on this) so I figured I'd keep > going > >> until > >>>> I saw /var/log/rsyslog directory and files created. This never > >> happened > >>>> and my second iteration stopped at about 600k and I saw some > memory > >> fork > >>>> errors (though they dumped only to standard error, not log, so I > >> lost them > >>>> (sorry)). > >>>> > >>>> Dump to disk having failed, I next wanted to see if rsyslog would > at > >> least > >>>> resume forward to remote host when it came back up (dumping > whatever > >> was in > >>>> memory to central syslog server). I restarted iptables on the > >> syslog > >>>> server to restore access to the port, but no logs were forwarded > >> from the > >>>> rsyslog client. > >>>> > >>>> Lastly, I restarted rsyslog, hoping that I would see a dump to > disk > >> but > >>>> this failed as well. > >>>> I'm sure it's something I'm doing incorrectly. Would appreciate > >> some > >>>> guidance. Who knows, maybe I just need to create the > >> /var/log/rsyslog > >>>> directory (assumed rsyslog would create it). While I'm waiting > for > >>>> feedback, I'll probably give that a shot. Thanks. > >>>> Todd > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Feb 14 18:51:51 2011 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 14 Feb 2011 12:51:51 -0500 Subject: [rsyslog] rsyslog multiple property replacer option Message-ID: Hello All, I am trying to add multiple property replacer options to a rsyslog.conf file of a 4.2.0 system. What I need to do is add escape-cc to existing propery replacer rules Here is what I am trying to do $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::escape-cc,sp-if-no-1st-sp%%msg:1:950%" This does not work. However, the following rule where I have only one property replacer option works. I would imagine this would be less efficient though , as it will make multiple passes over the message. $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::escape-cc%%msg:::sp-if-no-1st-sp%%msg:1:950%" Is there something wrong with the comma seperated option list in the first message? Regards, Sayan From rgerhards at hq.adiscon.com Mon Feb 14 18:55:07 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Feb 2011 18:55:07 +0100 Subject: [rsyslog] rsyslog multiple property replacer option In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC55@GRFEXC.intern.adiscon.com> In general, it works the way you use it. The big exception is "sp-if-no-1st-sp", which always returns either a single space or none -- and as such can not be combined. This is atually a hack, which was required in order to make RFC3164 work in all instances. The good news is that the second line you use does not incur any performance hit. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > Sent: Monday, February 14, 2011 6:52 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog multiple property replacer option > > Hello All, > I am trying to add multiple property replacer options to a rsyslog.conf > file > of a 4.2.0 system. > What I need to do is add escape-cc to existing propery replacer rules > > Here is what I am trying to do > $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% > %syslogtag:1:32%%msg:::escape-cc,sp-if-no-1st-sp%%msg:1:950%" > This does not work. > > However, the following rule where I have only one property replacer > option > works. I would imagine this would be less efficient though , as it will > make > multiple passes over the message. > > $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% > %syslogtag:1:32%%msg:::escape-cc%%msg:::sp-if-no-1st-sp%%msg:1:950%" > > Is there something wrong with the comma seperated option list in the > first > message? > > Regards, > Sayan > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From iamsayan at gmail.com Mon Feb 14 19:31:06 2011 From: iamsayan at gmail.com (Sayan Chowdhury) Date: Mon, 14 Feb 2011 13:31:06 -0500 Subject: [rsyslog] rsyslog multiple property replacer option In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC55@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDC55@GRFEXC.intern.adiscon.com> Message-ID: Thanks Rainer, I will just use the second line then. Regards, Sayan On Mon, Feb 14, 2011 at 12:55 PM, Rainer Gerhards wrote: > In general, it works the way you use it. The big exception is > "sp-if-no-1st-sp", which always returns either a single space or none -- > and > as such can not be combined. This is atually a hack, which was required in > order to make RFC3164 work in all instances. The good news is that the > second > line you use does not incur any performance hit. > > HTH > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sayan Chowdhury > > Sent: Monday, February 14, 2011 6:52 PM > > To: rsyslog-users > > Subject: [rsyslog] rsyslog multiple property replacer option > > > > Hello All, > > I am trying to add multiple property replacer options to a rsyslog.conf > > file > > of a 4.2.0 system. > > What I need to do is add escape-cc to existing propery replacer rules > > > > Here is what I am trying to do > > $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% > > %syslogtag:1:32%%msg:::escape-cc,sp-if-no-1st-sp%%msg:1:950%" > > This does not work. > > > > However, the following rule where I have only one property replacer > > option > > works. I would imagine this would be less efficient though , as it will > > make > > multiple passes over the message. > > > > $template forwardformat, "<%PRI%>%TIMESTAMP% %HOSTNAME% > > %syslogtag:1:32%%msg:::escape-cc%%msg:::sp-if-no-1st-sp%%msg:1:950%" > > > > Is there something wrong with the comma seperated option list in the > > first > > message? > > > > Regards, > > Sayan > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From dirk.schulz at kinzesberg.de Tue Feb 15 10:31:16 2011 From: dirk.schulz at kinzesberg.de (Dirk) Date: Tue, 15 Feb 2011 10:31:16 +0100 Subject: [rsyslog] Measuring performance of Rsyslog 3 Message-ID: <4D5A47E4.60107@kinzesberg.de> Hi folks, is there any possibility to gain some performance impression for rsyslogd 3? I would like to count - how many messages are coming in via tcp - how many messages are written to logs totally - how many messages are written to each log Of course I can write scripts to do "wc -l" for the second and third questions, but with some hundred logs this seems not a good idea, and real time results or near time results are preferred. Any idea is greatly appreciated. Dirk From rgerhards at hq.adiscon.com Tue Feb 15 10:32:45 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Feb 2011 10:32:45 +0100 Subject: [rsyslog] Measuring performance of Rsyslog 3 In-Reply-To: <4D5A47E4.60107@kinzesberg.de> References: <4D5A47E4.60107@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC5F@GRFEXC.intern.adiscon.com> I guess you look for performance counters. Unfortunately, there are not present before the introduction of impstat in v5. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk > Sent: Tuesday, February 15, 2011 10:31 AM > To: rsyslog-users > Subject: [rsyslog] Measuring performance of Rsyslog 3 > > Hi folks, > > is there any possibility to gain some performance impression for > rsyslogd 3? I would like to count > - how many messages are coming in via tcp > - how many messages are written to logs totally > - how many messages are written to each log > > Of course I can write scripts to do "wc -l" for the second and third > questions, but with some hundred logs this seems not a good idea, and > real time results or near time results are preferred. > > Any idea is greatly appreciated. > > Dirk > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dirk.schulz at kinzesberg.de Tue Feb 15 17:54:24 2011 From: dirk.schulz at kinzesberg.de (Dirk) Date: Tue, 15 Feb 2011 17:54:24 +0100 Subject: [rsyslog] Measuring performance of Rsyslog 3 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC5F@GRFEXC.intern.adiscon.com> References: <4D5A47E4.60107@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71DDC5F@GRFEXC.intern.adiscon.com> Message-ID: <4D5AAFC0.30809@kinzesberg.de> Am 15.02.11 10:32, schrieb Rainer Gerhards: > I guess you look for performance counters. Unfortunately, there are not > present before the introduction of impstat in v5. We will have a look at that, thanks. We start trying to implant v5 into SLES10. We stumbled upon a strange phenomenon: rsyslog reads messages from a log file and sends them to rsyslog on a central log server which parses them and writes them to log files. The client needs 1 % CPU for that, and the server needs 100 % CPU for that - with only the messages from this one client! Both machines are exactly the same. The configuration on the server is quite complex, so the messages we test with have to be parsed by 550 rules before they match, get written and discarded. Is this asynchronous resource usage "normal"? Or is it specially v3 doing it thus - would we benefit from using v5? Does it depend on the number of rules to be parsed - would we benefit from using regular expressions (assuming this is possible)? Any input, hint or help is greatly appreciated. Dirk > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Dirk >> Sent: Tuesday, February 15, 2011 10:31 AM >> To: rsyslog-users >> Subject: [rsyslog] Measuring performance of Rsyslog 3 >> >> Hi folks, >> >> is there any possibility to gain some performance impression for >> rsyslogd 3? I would like to count >> - how many messages are coming in via tcp >> - how many messages are written to logs totally >> - how many messages are written to each log >> >> Of course I can write scripts to do "wc -l" for the second and third >> questions, but with some hundred logs this seems not a good idea, and >> real time results or near time results are preferred. >> >> Any idea is greatly appreciated. >> >> Dirk >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Feb 15 18:24:44 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Feb 2011 18:24:44 +0100 Subject: [rsyslog] Measuring performance of Rsyslog 3 In-Reply-To: <4D5AAFC0.30809@kinzesberg.de> References: <4D5A47E4.60107@kinzesberg.de><9B6E2A8877C38245BFB15CC491A11DA71DDC5F@GRFEXC.intern.adiscon.com> <4D5AAFC0.30809@kinzesberg.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC68@GRFEXC.intern.adiscon.com> Well, this behaviour certainly looks strange. But it is hard to guess what exactly happens in your 550 rules. I'd say it is best to try out with v6 and report back your findings. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dirk > Sent: Tuesday, February 15, 2011 5:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] Measuring performance of Rsyslog 3 > > Am 15.02.11 10:32, schrieb Rainer Gerhards: > > I guess you look for performance counters. Unfortunately, there are > not > > present before the introduction of impstat in v5. > > We will have a look at that, thanks. We start trying to implant v5 into > SLES10. > > We stumbled upon a strange phenomenon: rsyslog reads messages from a > log > file and sends them to rsyslog on a central log server which parses > them > and writes them to log files. The client needs 1 % CPU for that, and > the > server needs 100 % CPU for that - with only the messages from this one > client! Both machines are exactly the same. > > The configuration on the server is quite complex, so the messages we > test with have to be parsed by 550 rules before they match, get written > and discarded. > > Is this asynchronous resource usage "normal"? Or is it specially v3 > doing it thus - would we benefit from using v5? > Does it depend on the number of rules to be parsed - would we benefit > from using regular expressions (assuming this is possible)? > > Any input, hint or help is greatly appreciated. > > Dirk > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Dirk > >> Sent: Tuesday, February 15, 2011 10:31 AM > >> To: rsyslog-users > >> Subject: [rsyslog] Measuring performance of Rsyslog 3 > >> > >> Hi folks, > >> > >> is there any possibility to gain some performance impression for > >> rsyslogd 3? I would like to count > >> - how many messages are coming in via tcp > >> - how many messages are written to logs totally > >> - how many messages are written to each log > >> > >> Of course I can write scripts to do "wc -l" for the second and third > >> questions, but with some hundred logs this seems not a good idea, > and > >> real time results or near time results are preferred. > >> > >> Any idea is greatly appreciated. > >> > >> Dirk > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Tue Feb 15 21:30:58 2011 From: david at lang.hm (david at lang.hm) Date: Tue, 15 Feb 2011 12:30:58 -0800 (PST) Subject: [rsyslog] Measuring performance of Rsyslog 3 In-Reply-To: <4D5AAFC0.30809@kinzesberg.de> References: <4D5A47E4.60107@kinzesberg.de> <9B6E2A8877C38245BFB15CC491A11DA71DDC5F@GRFEXC.intern.adiscon.com> <4D5AAFC0.30809@kinzesberg.de> Message-ID: On Tue, 15 Feb 2011, Dirk wrote: > Am 15.02.11 10:32, schrieb Rainer Gerhards: >> I guess you look for performance counters. Unfortunately, there are not >> present before the introduction of impstat in v5. > > We will have a look at that, thanks. We start trying to implant v5 into > SLES10. > > We stumbled upon a strange phenomenon: rsyslog reads messages from a log file > and sends them to rsyslog on a central log server which parses them and > writes them to log files. The client needs 1 % CPU for that, and the server > needs 100 % CPU for that - with only the messages from this one client! Both > machines are exactly the same. > > The configuration on the server is quite complex, so the messages we test > with have to be parsed by 550 rules before they match, get written and > discarded. > > Is this asynchronous resource usage "normal"? Or is it specially v3 > doing it thus - would we benefit from using v5? Does it depend on the > number of rules to be parsed - would we benefit from using regular > expressions (assuming this is possible)? > yes, it is very normal for the receiver to use much more CPU than the sender. if you think about what's happening, all the sender needs to do is to read the text, add a bit of formatting, and then send it over the network the receiver needs to receive arbatrary text, parse it to decide what sort of message it is and how it is formatted, then process the rules to decide if each rule applies, and then if the rule does apply, assemble a new output message (potentially changing the text that it has) and writing it out. that being said, there are a lot of ways to improve this. there is a fair amount of overhead in rsyslog when receiving messages as they get moved to and from the queue, the newer versions will move multiple messages at once, so they cut down this overhead a lot. There are a lot of other performance improvements since version 3. you can save 5-10% CPU by having predefined templates for writing the logs to a file instead of using the very flexible runtime defined templates but the big cost (and therefor the big win) will be in working to optimize the rules that you have to evaluate. why do you have so many rules? can you say that once a rule has matched the log none of the other rules apply? (or if you can't say this as a blanket statement, are there cases where you can say this?) do you have some rules that are much more common to match than others? (especially important in combination with the prior question) if you think of your rules logically, do they (or portions of them) form a tree where you can look for something and then branch into two different sets of rules to then evaluate after that (if so, then the new rulesets feature may be the right thing for you) As part of this, the different types of matching rules have very different costs (an if (regex) then arrangement being the highest overhead). it may be worth trying to use different types of matching rules, especially for the most common cases. once we can get an idea of what your rules look like, we may be able to suggest other optimizations. David Lang From mark at thermeon.com Wed Feb 16 10:57:55 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 16 Feb 2011 09:57:55 +0000 Subject: [rsyslog] Sanatize RSyslog Message-ID: Hi, It would appear the rsyslog is not sanitizing the input it receives for example is i send it the string "hello world this is a %s test" it will fail to log it. Is there a way to fix this? Regards Mark From rgerhards at hq.adiscon.com Wed Feb 16 11:20:05 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 11:20:05 +0100 Subject: [rsyslog] Sanatize RSyslog In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC6F@GRFEXC.intern.adiscon.com> please provide details > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Wednesday, February 16, 2011 10:58 AM > To: rsyslog-users > Subject: [rsyslog] Sanatize RSyslog > > Hi, > > It would appear the rsyslog is not sanitizing the input it receives > for example is i send it the string "hello world this is a %s test" it > will fail to log it. Is there a way to fix this? > > Regards > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Wed Feb 16 11:25:51 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 16 Feb 2011 10:25:51 +0000 Subject: [rsyslog] Sanatize RSyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC6F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDC6F@GRFEXC.intern.adiscon.com> Message-ID: No worries, i have just tested this again and it does record the % sign properly when i send it to rsyslog via logger but not when i do so via the ruby syslog module so must be an issue there. Sorry Mark From rgerhards at hq.adiscon.com Wed Feb 16 11:26:51 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 11:26:51 +0100 Subject: [rsyslog] Sanatize RSyslog In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDC6F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC70@GRFEXC.intern.adiscon.com> if you run rsyslog in debug mode, you will probably be able to see what ruby actually sends, and this will probably tell you what is going wrong. > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mark Olliver > Sent: Wednesday, February 16, 2011 11:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Sanatize RSyslog > > No worries, i have just tested this again and it does record the % > sign properly when i send it to rsyslog via logger but not when i do > so via the ruby syslog module so must be an issue there. > > Sorry > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mark at thermeon.com Wed Feb 16 12:45:52 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 16 Feb 2011 11:45:52 +0000 Subject: [rsyslog] Config help Message-ID: Hi All, I have several program that i am using to send data to local6 i am using rsyslog to filter these to their own logs however, some data seems to be written to multiple logs. Do you have any ideas? if $programname == 'USERLOG' then /var/log/user.log &~ if $programname == 'FILERR' then /var/log/filerr.log $ModLoad ommail $ActionMailSMTPServer localhost $ActionMailFrom rsyslog at example.com $ActionMailTo user at example.com $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" $template filerrBody,"There is a file Error on Host: %HOSTNAME%.example.com\n%msg%" $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" $ActionMailSubject filerrSubject $ActionExecOnlyOnceEveryInterval 50 if $programname == 'FILERR' then :ommail:;filerrBody $ModLoad omprog $actionomprogbinary /usr/bin/alert if $programname == 'FILERR' then :omprog:;smsBody if $programname =='FILERR' &~ Now data from program USERLOG correctly only goes into user.log Data from FILERR however, seems to go into filerr.log and also into my later defined local6.log. Any ideas why? Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything after that statement for that facility, Can i restrict it to only taking effect for this programname? Or just this one line below - then i could have different times for sms to email? Thanks Mark From dave at fly.srk.fer.hr Wed Feb 16 13:17:07 2011 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 16 Feb 2011 13:17:07 +0100 Subject: [rsyslog] Race conditions and crashes Message-ID: <20110216121707.GA29230@fly.srk.fer.hr> Hello. I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting thread scheduling class) on CentOS 5.5 (64-bit) and I have a number of crashes. SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of the mavhines in the cluster. SIGABRTs are generated by glibc: *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or corruption (fasttop): 0x00002aaab02bc4c0 *** SIGSEGVs are the usual NULL pointer accesses. I didn't check all core files, but the ones I checked had that condition. I decided to run rsyslog through Sun's Data Race analyzer[1] and it found a few problems. The tool is free and it runs under Linux as well, but it brings Sun's compiler which doesn't handle all of gcc extensions, so I had to change the code to make it compile. The patch is attached. It adds members to empty structs in a few places. Since that compiler doesn't have gcc atomic access builtins, config.h contains this: /* Define if compiler provides atomic builtins */ /* #undef HAVE_ATOMIC_BUILTINS */ /* Define if compiler provides 64 bit atomic builtins */ /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ My test was receiving 4 lines via UDP and writing them to a file and a FIFO. It was as simple as I could make it. Thread scheduling class was not set. The tool found the following problems: Total Races: 4 Experiment: exp1.er Race #1, Vaddr: 0x13909168 Access 1: Read, GetNxt + 0x0000008A, line 346 in "modules.c" Access 2: Write, addModToList + 0x00000131, line 326 in "modules.c" Total Callstack Traces: 1 Race #2, Vaddr: (Multiple Addresses) Access 1: Read, wtpShutdownAll + 0x00000371, line 247 in "wtp.c" Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, line 310 in "wtp.c" Total Callstack Traces: 2 Race #3, Vaddr: (Multiple Addresses) Access 1: Read, thrdDestruct + 0x00000058, line 76 in "threads.c" Access 2: Write, thrdStarter + 0x000001A2, line 197 in "threads.c" Total Callstack Traces: 1 Race #4, Vaddr: 0x1394764c Access 1: Read, processSocket + 0x000000FE, line 314 in "imudp.c" Access 2: Write, thrdTerminateNonCancel + 0x000000CC, line 100 in "threads.c" Total Callstack Traces: 1 What it found really are unprotected memory accesses (ie. bugs), but all of them are in insignificant places: race #1 - module loading race #2 - shutdown all workers race #3 - thread destructor (this one might be responsible for something) race #4 - thread termination on SIGTTIN My production system is a bit more complicated than that. It has UDP and TCP receivers and a few more threads created than the test system. I suppose I could test some more and try to find errors in other places, but before I do I'd like to know if anyone else used tools of this kind on rsyslog. And if so, what the results were. [1] http://download.oracle.com/docs/cd/E19205-01/821-2124/index.html -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr -------------- next part -------------- A non-text attachment was scrubbed... Name: rsyslog-5.6.2-empty_struct.diff Type: text/x-diff Size: 1342 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Wed Feb 16 13:55:49 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 13:55:49 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216121707.GA29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC74@GRFEXC.intern.adiscon.com> Just a quick note, will go through your mail in full later. I regularly use valgrind, which is obviously different, and ran the clang static analyzer in December (or January?) on the code, with a number of minor fixes. I am aware there is a race somewhere and I am trying to find it for a while now. So far, we have been unable to reproduce it in lab. The bugzilla has a couple of entries plus additional information. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, February 16, 2011 1:17 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Race conditions and crashes > > Hello. > > I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting > thread > scheduling class) on CentOS 5.5 (64-bit) and I have a number of > crashes. > SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of the > mavhines in the cluster. > > SIGABRTs are generated by glibc: > > *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or > corruption > (fasttop): 0x00002aaab02bc4c0 *** > > SIGSEGVs are the usual NULL pointer accesses. I didn't check all core > files, but the ones I checked had that condition. > > I decided to run rsyslog through Sun's Data Race analyzer[1] and it > found > a few problems. The tool is free and it runs under Linux as well, but > it > brings Sun's compiler which doesn't handle all of gcc extensions, so I > had > to change the code to make it compile. The patch is attached. It adds > members to empty structs in a few places. > > Since that compiler doesn't have gcc atomic access builtins, config.h > contains this: > > /* Define if compiler provides atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS */ > > /* Define if compiler provides 64 bit atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ > > My test was receiving 4 lines via UDP and writing them to a file and a > FIFO. > It was as simple as I could make it. Thread scheduling class was not > set. > > The tool found the following problems: > > Total Races: 4 Experiment: exp1.er > > Race #1, Vaddr: 0x13909168 > Access 1: Read, GetNxt + 0x0000008A, > line 346 in "modules.c" > Access 2: Write, addModToList + 0x00000131, > line 326 in "modules.c" > Total Callstack Traces: 1 > > Race #2, Vaddr: (Multiple Addresses) > Access 1: Read, wtpShutdownAll + 0x00000371, > line 247 in "wtp.c" > Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, > line 310 in "wtp.c" > Total Callstack Traces: 2 > > Race #3, Vaddr: (Multiple Addresses) > Access 1: Read, thrdDestruct + 0x00000058, > line 76 in "threads.c" > Access 2: Write, thrdStarter + 0x000001A2, > line 197 in "threads.c" > Total Callstack Traces: 1 > > Race #4, Vaddr: 0x1394764c > Access 1: Read, processSocket + 0x000000FE, > line 314 in "imudp.c" > Access 2: Write, thrdTerminateNonCancel + 0x000000CC, > line 100 in "threads.c" > Total Callstack Traces: 1 > > > What it found really are unprotected memory accesses (ie. bugs), but > all > of them are in insignificant places: > > race #1 - module loading > race #2 - shutdown all workers > race #3 - thread destructor (this one might be responsible for > something) > race #4 - thread termination on SIGTTIN > > > My production system is a bit more complicated than that. It has UDP > and > TCP receivers and a few more threads created than the test system. > I suppose I could test some more and try to find errors in other > places, > but before I do I'd like to know if anyone else used tools of this kind > on > rsyslog. And if so, what the results were. > > [1] http://download.oracle.com/docs/cd/E19205-01/821-2124/index.html > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr From dave at fly.srk.fer.hr Wed Feb 16 15:29:30 2011 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 16 Feb 2011 15:29:30 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC74@GRFEXC.intern.adiscon.com> References: <20110216121707.GA29230@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DDC74@GRFEXC.intern.adiscon.com> Message-ID: <20110216142930.GB29230@fly.srk.fer.hr> BTW, is there a way to stop rsyslog aside from sending it SIGTERM? Data collector doesn't like that and then I have problems with viewing results in the GUI. Rainer Gerhards wrote: > Just a quick note, will go through your mail in full later. I regularly use > valgrind, which is obviously different, and ran the clang static analyzer in > December (or January?) on the code, with a number of minor fixes. I am aware > there is a race somewhere and I am trying to find it for a while now. So far, > we have been unable to reproduce it in lab. The bugzilla has a couple of > entries plus additional information. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > > Sent: Wednesday, February 16, 2011 1:17 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Race conditions and crashes > > > > Hello. > > > > I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting > > thread > > scheduling class) on CentOS 5.5 (64-bit) and I have a number of > > crashes. > > SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of the > > mavhines in the cluster. > > > > SIGABRTs are generated by glibc: > > > > *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or > > corruption > > (fasttop): 0x00002aaab02bc4c0 *** > > > > SIGSEGVs are the usual NULL pointer accesses. I didn't check all core > > files, but the ones I checked had that condition. > > > > I decided to run rsyslog through Sun's Data Race analyzer[1] and it > > found > > a few problems. The tool is free and it runs under Linux as well, but > > it > > brings Sun's compiler which doesn't handle all of gcc extensions, so I > > had > > to change the code to make it compile. The patch is attached. It adds > > members to empty structs in a few places. > > > > Since that compiler doesn't have gcc atomic access builtins, config.h > > contains this: > > > > /* Define if compiler provides atomic builtins */ > > /* #undef HAVE_ATOMIC_BUILTINS */ > > > > /* Define if compiler provides 64 bit atomic builtins */ > > /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ > > > > My test was receiving 4 lines via UDP and writing them to a file and a > > FIFO. > > It was as simple as I could make it. Thread scheduling class was not > > set. > > > > The tool found the following problems: > > > > Total Races: 4 Experiment: exp1.er > > > > Race #1, Vaddr: 0x13909168 > > Access 1: Read, GetNxt + 0x0000008A, > > line 346 in "modules.c" > > Access 2: Write, addModToList + 0x00000131, > > line 326 in "modules.c" > > Total Callstack Traces: 1 > > > > Race #2, Vaddr: (Multiple Addresses) > > Access 1: Read, wtpShutdownAll + 0x00000371, > > line 247 in "wtp.c" > > Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, > > line 310 in "wtp.c" > > Total Callstack Traces: 2 > > > > Race #3, Vaddr: (Multiple Addresses) > > Access 1: Read, thrdDestruct + 0x00000058, > > line 76 in "threads.c" > > Access 2: Write, thrdStarter + 0x000001A2, > > line 197 in "threads.c" > > Total Callstack Traces: 1 > > > > Race #4, Vaddr: 0x1394764c > > Access 1: Read, processSocket + 0x000000FE, > > line 314 in "imudp.c" > > Access 2: Write, thrdTerminateNonCancel + 0x000000CC, > > line 100 in "threads.c" > > Total Callstack Traces: 1 > > > > > > What it found really are unprotected memory accesses (ie. bugs), but > > all > > of them are in insignificant places: > > > > race #1 - module loading > > race #2 - shutdown all workers > > race #3 - thread destructor (this one might be responsible for > > something) > > race #4 - thread termination on SIGTTIN > > > > > > My production system is a bit more complicated than that. It has UDP > > and > > TCP receivers and a few more threads created than the test system. > > I suppose I could test some more and try to find errors in other > > places, > > but before I do I'd like to know if anyone else used tools of this kind > > on > > rsyslog. And if so, what the results were. > > > > [1] http://download.oracle.com/docs/cd/E19205-01/821-2124/index.html > > > > -- > > .-. .-. Yes, I am an agent of Satan, but my duties are largely > > (_ \ / _) ceremonial. > > | > > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Wed Feb 16 15:36:50 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 15:36:50 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216142930.GB29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DDC74@GRFEXC.intern.adiscon.com> <20110216142930.GB29230@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC78@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, February 16, 2011 3:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] Race conditions and crashes > > BTW, is there a way to stop rsyslog aside from sending it SIGTERM? Data > collector doesn't like that and then I have problems with viewing > results > in the GUI. No, and I honestly do not see why this may be a problem. There is no other method because one shutdown method seemed sufficient... Rainer > > Rainer Gerhards wrote: > > Just a quick note, will go through your mail in full later. I > regularly use > > valgrind, which is obviously different, and ran the clang static > analyzer in > > December (or January?) on the code, with a number of minor fixes. I > am aware > > there is a race somewhere and I am trying to find it for a while now. > So far, > > we have been unable to reproduce it in lab. The bugzilla has a > couple of > > entries plus additional information. > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > > > Sent: Wednesday, February 16, 2011 1:17 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] Race conditions and crashes > > > > > > Hello. > > > > > > I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting > > > thread > > > scheduling class) on CentOS 5.5 (64-bit) and I have a number of > > > crashes. > > > SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of > the > > > mavhines in the cluster. > > > > > > SIGABRTs are generated by glibc: > > > > > > *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or > > > corruption > > > (fasttop): 0x00002aaab02bc4c0 *** > > > > > > SIGSEGVs are the usual NULL pointer accesses. I didn't check all > core > > > files, but the ones I checked had that condition. > > > > > > I decided to run rsyslog through Sun's Data Race analyzer[1] and it > > > found > > > a few problems. The tool is free and it runs under Linux as well, > but > > > it > > > brings Sun's compiler which doesn't handle all of gcc extensions, > so I > > > had > > > to change the code to make it compile. The patch is attached. It > adds > > > members to empty structs in a few places. > > > > > > Since that compiler doesn't have gcc atomic access builtins, > config.h > > > contains this: > > > > > > /* Define if compiler provides atomic builtins */ > > > /* #undef HAVE_ATOMIC_BUILTINS */ > > > > > > /* Define if compiler provides 64 bit atomic builtins */ > > > /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ > > > > > > My test was receiving 4 lines via UDP and writing them to a file > and a > > > FIFO. > > > It was as simple as I could make it. Thread scheduling class was > not > > > set. > > > > > > The tool found the following problems: > > > > > > Total Races: 4 Experiment: exp1.er > > > > > > Race #1, Vaddr: 0x13909168 > > > Access 1: Read, GetNxt + 0x0000008A, > > > line 346 in "modules.c" > > > Access 2: Write, addModToList + 0x00000131, > > > line 326 in "modules.c" > > > Total Callstack Traces: 1 > > > > > > Race #2, Vaddr: (Multiple Addresses) > > > Access 1: Read, wtpShutdownAll + 0x00000371, > > > line 247 in "wtp.c" > > > Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, > > > line 310 in "wtp.c" > > > Total Callstack Traces: 2 > > > > > > Race #3, Vaddr: (Multiple Addresses) > > > Access 1: Read, thrdDestruct + 0x00000058, > > > line 76 in "threads.c" > > > Access 2: Write, thrdStarter + 0x000001A2, > > > line 197 in "threads.c" > > > Total Callstack Traces: 1 > > > > > > Race #4, Vaddr: 0x1394764c > > > Access 1: Read, processSocket + 0x000000FE, > > > line 314 in "imudp.c" > > > Access 2: Write, thrdTerminateNonCancel + 0x000000CC, > > > line 100 in "threads.c" > > > Total Callstack Traces: 1 > > > > > > > > > What it found really are unprotected memory accesses (ie. bugs), > but > > > all > > > of them are in insignificant places: > > > > > > race #1 - module loading > > > race #2 - shutdown all workers > > > race #3 - thread destructor (this one might be responsible for > > > something) > > > race #4 - thread termination on SIGTTIN > > > > > > > > > My production system is a bit more complicated than that. It has > UDP > > > and > > > TCP receivers and a few more threads created than the test system. > > > I suppose I could test some more and try to find errors in other > > > places, > > > but before I do I'd like to know if anyone else used tools of this > kind > > > on > > > rsyslog. And if so, what the results were. > > > > > > [1] http://download.oracle.com/docs/cd/E19205-01/821- > 2124/index.html > > > > > > -- > > > .-. .-. Yes, I am an agent of Satan, but my duties are > largely > > > (_ \ / _) ceremonial. > > > | > > > | dave at fly.srk.fer.hr > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Feb 16 17:27:55 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 17:27:55 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216121707.GA29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, February 16, 2011 1:17 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Race conditions and crashes > > Hello. > > I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting > thread > scheduling class) on CentOS 5.5 (64-bit) and I have a number of > crashes. > SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of the > mavhines in the cluster. > > SIGABRTs are generated by glibc: > > *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or > corruption > (fasttop): 0x00002aaab02bc4c0 *** > > SIGSEGVs are the usual NULL pointer accesses. I didn't check all core > files, but the ones I checked had that condition. > > I decided to run rsyslog through Sun's Data Race analyzer[1] and it > found > a few problems. The tool is free and it runs under Linux as well, but > it > brings Sun's compiler which doesn't handle all of gcc extensions, so I > had > to change the code to make it compile. The patch is attached. It adds > members to empty structs in a few places. will see that I add that :) Since I use gcc on Solaris, this seems to have slipped my attention ;) > > Since that compiler doesn't have gcc atomic access builtins, config.h > contains this: > > /* Define if compiler provides atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS */ > > /* Define if compiler provides 64 bit atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ > > My test was receiving 4 lines via UDP and writing them to a file and a > FIFO. > It was as simple as I could make it. Thread scheduling class was not > set. did you experience any problems without the analyzer in this setting? As I said, I am searching for this bug but so far we are unable to reproduce (I even got some help from Florian, but so far to no avail...). > > The tool found the following problems: > > Total Races: 4 Experiment: exp1.er > > Race #1, Vaddr: 0x13909168 > Access 1: Read, GetNxt + 0x0000008A, > line 346 in "modules.c" > Access 2: Write, addModToList + 0x00000131, > line 326 in "modules.c" > Total Callstack Traces: 1 > > Race #2, Vaddr: (Multiple Addresses) > Access 1: Read, wtpShutdownAll + 0x00000371, > line 247 in "wtp.c" > Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, > line 310 in "wtp.c" > Total Callstack Traces: 2 > > Race #3, Vaddr: (Multiple Addresses) > Access 1: Read, thrdDestruct + 0x00000058, > line 76 in "threads.c" > Access 2: Write, thrdStarter + 0x000001A2, > line 197 in "threads.c" > Total Callstack Traces: 1 > > Race #4, Vaddr: 0x1394764c > Access 1: Read, processSocket + 0x000000FE, > line 314 in "imudp.c" > Access 2: Write, thrdTerminateNonCancel + 0x000000CC, > line 100 in "threads.c" > Total Callstack Traces: 1 > > > What it found really are unprotected memory accesses (ie. bugs), but > all > of them are in insignificant places: > > race #1 - module loading this is known and really no issue > race #2 - shutdown all workers > race #3 - thread destructor (this one might be responsible for > something) I think they are OK as well, but I will check. Maybe just atomic emulation is missing. May also be that this is a case where it really doesn't matter if dual reads are necessary. > race #4 - thread termination on SIGTTIN sounds interesting, will check. And I think my initial answer was only partly correct. I assumed the tool was something like clang static analyzer. I use valgrind tools very frequently, and there are two thread error detectors, drd and helgrind. Both have pros and cons, and I regularly use both. Unfortunately, some kinds of races do not manifest in valgrind. In any case, I'd suggest you also give it a try if you don't know the tool. It is excellent and has given rsyslog's code quality a real boost when I found it (Thanks to Peter and others for making me aware of it!). Rainer > > > My production system is a bit more complicated than that. It has UDP > and > TCP receivers and a few more threads created than the test system. > I suppose I could test some more and try to find errors in other > places, > but before I do I'd like to know if anyone else used tools of this kind > on > rsyslog. And if so, what the results were. > > [1] http://download.oracle.com/docs/cd/E19205-01/821-2124/index.html > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr From mark at thermeon.com Wed Feb 16 17:55:11 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 16 Feb 2011 16:55:11 +0000 Subject: [rsyslog] postgres log dump Message-ID: Hi, I have just setup the postgres export for rsyslog, however when i view the log via loganalyzer i have noticed the process id fields are blank and instead the syslog tag has both the tag and the process id in it. Is there an easy way to change this and split them back up? Thanks Mark From rgerhards at hq.adiscon.com Wed Feb 16 18:21:38 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 18:21:38 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216121707.GA29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr> Message-ID: <4D5C07A2.4080202@hq.adiscon.com> patch is now merged :) On 02/16/2011 01:17 PM, Dra?en Ka?ar wrote: > Hello. > > I have rsyslog 5.6.2 (+ patches for blocking FIFO write and setting thread > scheduling class) on CentOS 5.5 (64-bit) and I have a number of crashes. > SInce 2011-02-02 there were 27 SIGSEGVs and 35 SIGABRTs on one of the > mavhines in the cluster. > > SIGABRTs are generated by glibc: > > *** glibc detected *** /opt/bulb/sbin/rsyslogd: double free or corruption > (fasttop): 0x00002aaab02bc4c0 *** > > SIGSEGVs are the usual NULL pointer accesses. I didn't check all core > files, but the ones I checked had that condition. > > I decided to run rsyslog through Sun's Data Race analyzer[1] and it found > a few problems. The tool is free and it runs under Linux as well, but it > brings Sun's compiler which doesn't handle all of gcc extensions, so I had > to change the code to make it compile. The patch is attached. It adds > members to empty structs in a few places. > > Since that compiler doesn't have gcc atomic access builtins, config.h > contains this: > > /* Define if compiler provides atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS */ > > /* Define if compiler provides 64 bit atomic builtins */ > /* #undef HAVE_ATOMIC_BUILTINS_64BIT */ > > My test was receiving 4 lines via UDP and writing them to a file and a FIFO. > It was as simple as I could make it. Thread scheduling class was not set. > > The tool found the following problems: > > Total Races: 4 Experiment: exp1.er > > Race #1, Vaddr: 0x13909168 > Access 1: Read, GetNxt + 0x0000008A, > line 346 in "modules.c" > Access 2: Write, addModToList + 0x00000131, > line 326 in "modules.c" > Total Callstack Traces: 1 > > Race #2, Vaddr: (Multiple Addresses) > Access 1: Read, wtpShutdownAll + 0x00000371, > line 247 in "wtp.c" > Access 2: Write, wtpWrkrExecCleanup + 0x000000F2, > line 310 in "wtp.c" > Total Callstack Traces: 2 > > Race #3, Vaddr: (Multiple Addresses) > Access 1: Read, thrdDestruct + 0x00000058, > line 76 in "threads.c" > Access 2: Write, thrdStarter + 0x000001A2, > line 197 in "threads.c" > Total Callstack Traces: 1 > > Race #4, Vaddr: 0x1394764c > Access 1: Read, processSocket + 0x000000FE, > line 314 in "imudp.c" > Access 2: Write, thrdTerminateNonCancel + 0x000000CC, > line 100 in "threads.c" > Total Callstack Traces: 1 > > > What it found really are unprotected memory accesses (ie. bugs), but all > of them are in insignificant places: > > race #1 - module loading > race #2 - shutdown all workers > race #3 - thread destructor (this one might be responsible for something) > race #4 - thread termination on SIGTTIN > > > My production system is a bit more complicated than that. It has UDP and > TCP receivers and a few more threads created than the test system. > I suppose I could test some more and try to find errors in other places, > but before I do I'd like to know if anyone else used tools of this kind on > rsyslog. And if so, what the results were. > > [1] http://download.oracle.com/docs/cd/E19205-01/821-2124/index.html > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dave at fly.srk.fer.hr Wed Feb 16 18:37:54 2011 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 16 Feb 2011 18:37:54 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com> References: <20110216121707.GA29230@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com> Message-ID: <20110216173754.GC29230@fly.srk.fer.hr> Rainer Gerhards wrote: > > My test was receiving 4 lines via UDP and writing them to a file and a > > FIFO. > > It was as simple as I could make it. Thread scheduling class was not > > set. > > did you experience any problems without the analyzer in this setting? As I > said, I am searching for this bug but so far we are unable to reproduce (I > even got some help from Florian, but so far to no avail...). No, I can't reproduce the problem at will. I only see it on production servers under heavy load, but I can't run tools which slow down processing there. OTOH, a while ago I reported perfectly reproducible crash with omruleset. I'll try that one under thread analyzer. And I could take a huge chunk of production data and feed it to the test system to see what happens. > > race #1 - module loading > this is known and really no issue Yes. > > race #2 - shutdown all workers > > race #3 - thread destructor (this one might be responsible for > > something) > I think they are OK as well, but I will check. Maybe just atomic emulation is > missing. May also be that this is a case where it really doesn't matter if > dual reads are necessary. I don't know when thread destructor is called. If only on rsyslog shutdown, then it doesn't really matter. If it can be called during normal processing then maybe that's causing the problem, but I don't think that's the one. > > race #4 - thread termination on SIGTTIN > sounds interesting, will check. I think I just copyed that SIGTTIN from the comment in the code. I didn't send any signals except SIGTERM to terminate the process. I don't know if the thread analyzer inserts its own signal handlers. > And I think my initial answer was only partly correct. I assumed the tool was > something like clang static analyzer. It's not static. Which is nice because it can find things that static analyzers can't, but the not so nice part is that you have to figure out a way to excercize all interesting code paths. One significant difference between production and very light lab load is that the production is using disk queues. They were configured for the lab test as well, but the amount of data was small and nothing was ever written to disk. Maybe the problem is in the disk writing code. And maybe not. :-( I got several more problem reports from the tool. One of them is a real dirty read bug, but the data which is being read is then fed to the dbgprint statement, so that's not causing any problem. In the worst case you'd just get a wrong debug output printed out. And there are a few which look very odd. Like this one: Race #7, Vaddr: 0x41179db4 Access 1: Write, __pthread_unwind + 0x0000003B Access 2: Read, batchProcessed + 0x000000C8, line 1660 in "queue.c" Total Callstack Traces: 1 Callstack Trace 1 Access 1: Write __pthread_unwind + 0x0000003B collector_root + 0x0000004C start_thread + 0x000000D5 Access 2: Read batchProcessed + 0x000000C8, line 1660 in "queue.c" wtiWorker + 0x000003A9, line 304 in "wti.c" wtpWorker + 0x000002A5, line 387 in "wtp.c" collector_root + 0x0000004C start_thread + 0x000000D5 That's from pthread_setcancelstate() call. I just don't want to go into that direction. :-) -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Wed Feb 16 18:49:17 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 18:49:17 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216173754.GC29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com> <20110216173754.GC29230@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC82@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, February 16, 2011 6:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Race conditions and crashes > > Rainer Gerhards wrote: > > > > My test was receiving 4 lines via UDP and writing them to a file > and a > > > FIFO. > > > It was as simple as I could make it. Thread scheduling class was > not > > > set. > > > > did you experience any problems without the analyzer in this setting? > As I > > said, I am searching for this bug but so far we are unable to > reproduce (I > > even got some help from Florian, but so far to no avail...). > > No, I can't reproduce the problem at will. I only see it on production > servers under heavy load, but I can't run tools which slow down > processing > there. > > OTOH, a while ago I reported perfectly reproducible crash with > omruleset. > I'll try that one under thread analyzer. I'll probably try that out next. But I tend to stick a bit to the current scenario -- will see... > > And I could take a huge chunk of production data and feed it to the > test > system to see what happens. > > > > race #1 - module loading > > this is known and really no issue > > Yes. > > > > race #2 - shutdown all workers > > > race #3 - thread destructor (this one might be responsible for > > > something) > > I think they are OK as well, but I will check. Maybe just atomic > emulation is > > missing. May also be that this is a case where it really doesn't > matter if > > dual reads are necessary. > > I don't know when thread destructor is called. If only on rsyslog > shutdown, then it doesn't really matter. If it can be called during > normal > processing then maybe that's causing the problem, but I don't think > that's > the one. It can also be called during normal operation, when the workload reduces and workers sit idle for a while. But all it does is set a flag to indicate the worker should shut down (or "the other way round", that it has shut down), so if the update is done late, it will just result in one more iteration in the shutdown loop, a few ms delay). > > > > race #4 - thread termination on SIGTTIN > > sounds interesting, will check. > > I think I just copyed that SIGTTIN from the comment in the code. I > didn't > send any signals except SIGTERM to terminate the process. I don't know > if > the thread analyzer inserts its own signal handlers. SIGTTIN is used by rsyslog internally to indicate that the thread shall terminate. The problem is that we must wake up blocking API calls, and SIGTTIN does this. > > > And I think my initial answer was only partly correct. I assumed the > tool was > > something like clang static analyzer. > > It's not static. Which is nice because it can find things that static > analyzers can't, but the not so nice part is that you have to figure > out a > way to excercize all interesting code paths. > > One significant difference between production and very light lab load > is > that the production is using disk queues. They were configured for the > lab > test as well, but the amount of data was small and nothing was ever > written to disk. Ah, that's an interesting hint! Disk queue mode is quite different and disk-assisted queues are very complex beasts. Have you tried pure disk mode (I don't think it'll cause the issue to appear, but maybe -- I was not yet on this route). > > Maybe the problem is in the disk writing code. And maybe not. :-( > > I got several more problem reports from the tool. One of them is a real > dirty read bug, but the data which is being read is then fed to the > dbgprint > statement, so that's not causing any problem. In the worst case you'd > just > get a wrong debug output printed out. Also known. The reasoning is that the more you properly syncrhonize inside the debug code, the more you change the runtime characteristics of the code to be tested... So at many places inside the debug system races (with cosmetic results) are deliberately accepted. It will get even worse in newer builds, where I was able to identify one problem by totally disabling all sync code (so far hardcoded, but will become an optional debug switch). The joys of multithreading ;) > > And there are a few which look very odd. Like this one: > > Race #7, Vaddr: 0x41179db4 > Access 1: Write, __pthread_unwind + 0x0000003B > Access 2: Read, batchProcessed + 0x000000C8, > line 1660 in "queue.c" > Total Callstack Traces: 1 > Callstack Trace 1 > Access 1: Write > __pthread_unwind + 0x0000003B > collector_root + 0x0000004C > start_thread + 0x000000D5 > Access 2: Read > batchProcessed + 0x000000C8, line 1660 in "queue.c" > wtiWorker + 0x000003A9, line 304 in "wti.c" > wtpWorker + 0x000002A5, line 387 in "wtp.c" > collector_root + 0x0000004C > start_thread + 0x000000D5 > > That's from pthread_setcancelstate() call. I just don't want to go into > that direction. :-) Was that during shutdown? I don't think there is any other reason pthread_cancel should be called in the recent v5 codebase... Rainer From rgerhards at hq.adiscon.com Wed Feb 16 18:56:18 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Feb 2011 18:56:18 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC82@GRFEXC.intern.adiscon.com> References: <20110216121707.GA29230@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com><20110216173754.GC29230@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DDC82@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC83@GRFEXC.intern.adiscon.com> > > One significant difference between production and very light lab load > > is > > that the production is using disk queues. They were configured for > the > > lab > > test as well, but the amount of data was small and nothing was ever > > written to disk. > > Ah, that's an interesting hint! Disk queue mode is quite different and > disk-assisted queues are very complex beasts. Have you tried pure disk > mode > (I don't think it'll cause the issue to appear, but maybe -- I was not > yet on > this route). One thing: there is an output module "omtesting". Not documented, but sufficient source comments. You can use it to introduce deliberate delays inside the ruleset, so that a queue can build up. Maybe useful. Rainer From david at lang.hm Wed Feb 16 23:01:53 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 16 Feb 2011 14:01:53 -0800 (PST) Subject: [rsyslog] Config help In-Reply-To: References: Message-ID: On Wed, 16 Feb 2011, Mark Olliver wrote: > Hi All, > > I have several program that i am using to send data to local6 i am > using rsyslog to filter these to their own logs however, some data > seems to be written to multiple logs. Do you have any ideas? > > > if $programname == 'USERLOG' then /var/log/user.log > &~ > > if $programname == 'FILERR' then /var/log/filerr.log > $ModLoad ommail > $ActionMailSMTPServer localhost > $ActionMailFrom rsyslog at example.com > $ActionMailTo user at example.com > $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" > $template filerrBody,"There is a file Error on Host: > %HOSTNAME%.example.com\n%msg%" > $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" > $ActionMailSubject filerrSubject > $ActionExecOnlyOnceEveryInterval 50 > if $programname == 'FILERR' then :ommail:;filerrBody > $ModLoad omprog > $actionomprogbinary /usr/bin/alert > if $programname == 'FILERR' then :omprog:;smsBody > > if $programname =='FILERR' > &~ this looks wrong do you really mean to have this partial if statement? David Lang > > > Now data from program USERLOG correctly only goes into user.log Data > from FILERR however, seems to go into filerr.log and also into my > later defined local6.log. Any ideas why? > > Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything > after that statement for that facility, Can i restrict it to only > taking effect for this programname? Or just this one line below - then > i could have different times for sms to email? > > Thanks > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mark at thermeon.com Wed Feb 16 23:26:57 2011 From: mark at thermeon.com (Mark Olliver) Date: Wed, 16 Feb 2011 22:26:57 +0000 Subject: [rsyslog] Config help In-Reply-To: References: Message-ID: The partial if was to try and clear the messages to stop them going to other logs. On 16 Feb 2011 22:01, wrote: > On Wed, 16 Feb 2011, Mark Olliver wrote: > >> Hi All, >> >> I have several program that i am using to send data to local6 i am >> using rsyslog to filter these to their own logs however, some data >> seems to be written to multiple logs. Do you have any ideas? >> >> >> if $programname == 'USERLOG' then /var/log/user.log >> &~ >> >> if $programname == 'FILERR' then /var/log/filerr.log >> $ModLoad ommail >> $ActionMailSMTPServer localhost >> $ActionMailFrom rsyslog at example.com >> $ActionMailTo user at example.com >> $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" >> $template filerrBody,"There is a file Error on Host: >> %HOSTNAME%.example.com\n%msg%" >> $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" >> $ActionMailSubject filerrSubject >> $ActionExecOnlyOnceEveryInterval 50 >> if $programname == 'FILERR' then :ommail:;filerrBody >> $ModLoad omprog >> $actionomprogbinary /usr/bin/alert >> if $programname == 'FILERR' then :omprog:;smsBody >> >> if $programname =='FILERR' >> &~ > > this looks wrong > do you really mean to have this partial if statement? > > David Lang > >> >> >> Now data from program USERLOG correctly only goes into user.log Data >> from FILERR however, seems to go into filerr.log and also into my >> later defined local6.log. Any ideas why? >> >> Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything >> after that statement for that facility, Can i restrict it to only >> taking effect for this programname? Or just this one line below - then >> i could have different times for sms to email? >> >> Thanks >> >> Mark >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Feb 17 00:53:16 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 16 Feb 2011 15:53:16 -0800 (PST) Subject: [rsyslog] Config help In-Reply-To: References: Message-ID: On Wed, 16 Feb 2011, Mark Olliver wrote: > The partial if was to try and clear the messages to stop them going to other > logs. Ok, but that would not do it. If you eliminated the if line, then the &~ line would apply the match criteria of the prior line and stop processing that log message but with the partial if line, rsyslog will see a syntax error, and depending on exactly how it handles that error it coudl affect things after that in the config. David Lang > On 16 Feb 2011 22:01, wrote: >> On Wed, 16 Feb 2011, Mark Olliver wrote: >> >>> Hi All, >>> >>> I have several program that i am using to send data to local6 i am >>> using rsyslog to filter these to their own logs however, some data >>> seems to be written to multiple logs. Do you have any ideas? >>> >>> >>> if $programname == 'USERLOG' then /var/log/user.log >>> &~ >>> >>> if $programname == 'FILERR' then /var/log/filerr.log >>> $ModLoad ommail >>> $ActionMailSMTPServer localhost >>> $ActionMailFrom rsyslog at example.com >>> $ActionMailTo user at example.com >>> $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" >>> $template filerrBody,"There is a file Error on Host: >>> %HOSTNAME%.example.com\n%msg%" >>> $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" >>> $ActionMailSubject filerrSubject >>> $ActionExecOnlyOnceEveryInterval 50 >>> if $programname == 'FILERR' then :ommail:;filerrBody >>> $ModLoad omprog >>> $actionomprogbinary /usr/bin/alert >>> if $programname == 'FILERR' then :omprog:;smsBody >>> >>> if $programname =='FILERR' >>> &~ >> >> this looks wrong >> do you really mean to have this partial if statement? >> >> David Lang >> >>> >>> >>> Now data from program USERLOG correctly only goes into user.log Data >>> from FILERR however, seems to go into filerr.log and also into my >>> later defined local6.log. Any ideas why? >>> >>> Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything >>> after that statement for that facility, Can i restrict it to only >>> taking effect for this programname? Or just this one line below - then >>> i could have different times for sms to email? >>> >>> Thanks >>> >>> Mark >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mark at thermeon.com Thu Feb 17 07:56:52 2011 From: mark at thermeon.com (Mark Olliver) Date: Thu, 17 Feb 2011 06:56:52 +0000 Subject: [rsyslog] Config help In-Reply-To: References: Message-ID: Arh ok. I that case how do i scope the rate limit so that it only effects the one next line instead of everything from there on? Thanks Mark On 16 Feb 2011 23:53, wrote: > On Wed, 16 Feb 2011, Mark Olliver wrote: > >> The partial if was to try and clear the messages to stop them going to other >> logs. > > Ok, but that would not do it. > > If you eliminated the if line, then the &~ line would apply the match > criteria of the prior line and stop processing that log message > > but with the partial if line, rsyslog will see a syntax error, and > depending on exactly how it handles that error it coudl affect things > after that in the config. > > David Lang > >> On 16 Feb 2011 22:01, wrote: >>> On Wed, 16 Feb 2011, Mark Olliver wrote: >>> >>>> Hi All, >>>> >>>> I have several program that i am using to send data to local6 i am >>>> using rsyslog to filter these to their own logs however, some data >>>> seems to be written to multiple logs. Do you have any ideas? >>>> >>>> >>>> if $programname == 'USERLOG' then /var/log/user.log >>>> &~ >>>> >>>> if $programname == 'FILERR' then /var/log/filerr.log >>>> $ModLoad ommail >>>> $ActionMailSMTPServer localhost >>>> $ActionMailFrom rsyslog at example.com >>>> $ActionMailTo user at example.com >>>> $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" >>>> $template filerrBody,"There is a file Error on Host: >>>> %HOSTNAME%.example.com\n%msg%" >>>> $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" >>>> $ActionMailSubject filerrSubject >>>> $ActionExecOnlyOnceEveryInterval 50 >>>> if $programname == 'FILERR' then :ommail:;filerrBody >>>> $ModLoad omprog >>>> $actionomprogbinary /usr/bin/alert >>>> if $programname == 'FILERR' then :omprog:;smsBody >>>> >>>> if $programname =='FILERR' >>>> &~ >>> >>> this looks wrong >>> do you really mean to have this partial if statement? >>> >>> David Lang >>> >>>> >>>> >>>> Now data from program USERLOG correctly only goes into user.log Data >>>> from FILERR however, seems to go into filerr.log and also into my >>>> later defined local6.log. Any ideas why? >>>> >>>> Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything >>>> after that statement for that facility, Can i restrict it to only >>>> taking effect for this programname? Or just this one line below - then >>>> i could have different times for sms to email? >>>> >>>> Thanks >>>> >>>> Mark >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Feb 17 08:33:46 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 16 Feb 2011 23:33:46 -0800 (PST) Subject: [rsyslog] Config help In-Reply-To: References: Message-ID: that I'm less sure of, but what I would assume based on what I understand of rsyslog is that the way to do this would be to put another rate limit after the line as well, resetting it to unlimited. David Lang On Thu, 17 Feb 2011, Mark Olliver wrote: > Arh ok. > > I that case how do i scope the rate limit so that it only effects the one > next line instead of everything from there on? > > Thanks > > Mark > On 16 Feb 2011 23:53, wrote: >> On Wed, 16 Feb 2011, Mark Olliver wrote: >> >>> The partial if was to try and clear the messages to stop them going to > other >>> logs. >> >> Ok, but that would not do it. >> >> If you eliminated the if line, then the &~ line would apply the match >> criteria of the prior line and stop processing that log message >> >> but with the partial if line, rsyslog will see a syntax error, and >> depending on exactly how it handles that error it coudl affect things >> after that in the config. >> >> David Lang >> >>> On 16 Feb 2011 22:01, wrote: >>>> On Wed, 16 Feb 2011, Mark Olliver wrote: >>>> >>>>> Hi All, >>>>> >>>>> I have several program that i am using to send data to local6 i am >>>>> using rsyslog to filter these to their own logs however, some data >>>>> seems to be written to multiple logs. Do you have any ideas? >>>>> >>>>> >>>>> if $programname == 'USERLOG' then /var/log/user.log >>>>> &~ >>>>> >>>>> if $programname == 'FILERR' then /var/log/filerr.log >>>>> $ModLoad ommail >>>>> $ActionMailSMTPServer localhost >>>>> $ActionMailFrom rsyslog at example.com >>>>> $ActionMailTo user at example.com >>>>> $template filerrSubject,"FILE ERROR %HOSTNAME%.example.com %timestamp%" >>>>> $template filerrBody,"There is a file Error on Host: >>>>> %HOSTNAME%.example.com\n%msg%" >>>>> $template smsBody,"%programname%, 44711111111, %hostname% %rawmsg%\n" >>>>> $ActionMailSubject filerrSubject >>>>> $ActionExecOnlyOnceEveryInterval 50 >>>>> if $programname == 'FILERR' then :ommail:;filerrBody >>>>> $ModLoad omprog >>>>> $actionomprogbinary /usr/bin/alert >>>>> if $programname == 'FILERR' then :omprog:;smsBody >>>>> >>>>> if $programname =='FILERR' >>>>> &~ >>>> >>>> this looks wrong >>>> do you really mean to have this partial if statement? >>>> >>>> David Lang >>>> >>>>> >>>>> >>>>> Now data from program USERLOG correctly only goes into user.log Data >>>>> from FILERR however, seems to go into filerr.log and also into my >>>>> later defined local6.log. Any ideas why? >>>>> >>>>> Also $ActionExecOnlyOnceEveryInterval 50 seems to effect everything >>>>> after that statement for that facility, Can i restrict it to only >>>>> taking effect for this programname? Or just this one line below - then >>>>> i could have different times for sms to email? >>>>> >>>>> Thanks >>>>> >>>>> Mark >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Feb 17 15:16:35 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 17 Feb 2011 15:16:35 +0100 Subject: [rsyslog] Race conditions and crashes In-Reply-To: <20110216173754.GC29230@fly.srk.fer.hr> References: <20110216121707.GA29230@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DDC7B@GRFEXC.intern.adiscon.com> <20110216173754.GC29230@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC8E@GRFEXC.intern.adiscon.com> I've just fixed a cosmetic race, but thought you'd like to know. Details in the commit comment: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=39406781e51c7305cb9f35f5 f9fed9c10dea9ecd Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, February 16, 2011 6:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Race conditions and crashes > > Rainer Gerhards wrote: > > > > My test was receiving 4 lines via UDP and writing them to a file > and a > > > FIFO. > > > It was as simple as I could make it. Thread scheduling class was > not > > > set. > > > > did you experience any problems without the analyzer in this setting? > As I > > said, I am searching for this bug but so far we are unable to > reproduce (I > > even got some help from Florian, but so far to no avail...). > > No, I can't reproduce the problem at will. I only see it on production > servers under heavy load, but I can't run tools which slow down > processing > there. > > OTOH, a while ago I reported perfectly reproducible crash with > omruleset. > I'll try that one under thread analyzer. > > And I could take a huge chunk of production data and feed it to the > test > system to see what happens. > > > > race #1 - module loading > > this is known and really no issue > > Yes. > > > > race #2 - shutdown all workers > > > race #3 - thread destructor (this one might be responsible for > > > something) > > I think they are OK as well, but I will check. Maybe just atomic > emulation is > > missing. May also be that this is a case where it really doesn't > matter if > > dual reads are necessary. > > I don't know when thread destructor is called. If only on rsyslog > shutdown, then it doesn't really matter. If it can be called during > normal > processing then maybe that's causing the problem, but I don't think > that's > the one. > > > > race #4 - thread termination on SIGTTIN > > sounds interesting, will check. > > I think I just copyed that SIGTTIN from the comment in the code. I > didn't > send any signals except SIGTERM to terminate the process. I don't know > if > the thread analyzer inserts its own signal handlers. > > > And I think my initial answer was only partly correct. I assumed the > tool was > > something like clang static analyzer. > > It's not static. Which is nice because it can find things that static > analyzers can't, but the not so nice part is that you have to figure > out a > way to excercize all interesting code paths. > > One significant difference between production and very light lab load > is > that the production is using disk queues. They were configured for the > lab > test as well, but the amount of data was small and nothing was ever > written to disk. > > Maybe the problem is in the disk writing code. And maybe not. :-( > > I got several more problem reports from the tool. One of them is a real > dirty read bug, but the data which is being read is then fed to the > dbgprint > statement, so that's not causing any problem. In the worst case you'd > just > get a wrong debug output printed out. > > And there are a few which look very odd. Like this one: > > Race #7, Vaddr: 0x41179db4 > Access 1: Write, __pthread_unwind + 0x0000003B > Access 2: Read, batchProcessed + 0x000000C8, > line 1660 in "queue.c" > Total Callstack Traces: 1 > Callstack Trace 1 > Access 1: Write > __pthread_unwind + 0x0000003B > collector_root + 0x0000004C > start_thread + 0x000000D5 > Access 2: Read > batchProcessed + 0x000000C8, line 1660 in "queue.c" > wtiWorker + 0x000003A9, line 304 in "wti.c" > wtpWorker + 0x000002A5, line 387 in "wtp.c" > collector_root + 0x0000004C > start_thread + 0x000000D5 > > That's from pthread_setcancelstate() call. I just don't want to go into > that direction. :-) > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Thu Feb 17 15:18:02 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Thu, 17 Feb 2011 15:18:02 +0100 Subject: [rsyslog] rsyslog 5.7.4 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC8F@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.7.4, the new v5-beta. This release includes some bug fixes, the new pmsnare module and some improvements. Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-5-7-4-v5-beta/ Download: http://www.rsyslog.com/rsyslog-5-7-4-v5-beta/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From rgerhards at hq.adiscon.com Thu Feb 17 16:20:43 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 17 Feb 2011 16:20:43 +0100 Subject: [rsyslog] SIGSEGV because of double free in msgDestruct In-Reply-To: <20101112123746.GA26430@fly.srk.fer.hr> References: <20101112123746.GA26430@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC90@GRFEXC.intern.adiscon.com> Hi Dra?en, I dug this problem report out and tried to reproduce. I both tried my usual platform under Fedora as well as CentOS 5.5 (64 bit). Unfortunately I did not run into any trouble. Of course, I do not have the program you use, so this may be a difference. I tested with a small program that just read stdin and threw everything it read away. I also tested with writing to a file instead of omprog. I tested on a quad core system and sent 10 million messages. Can you confirm that you still have some trouble with this scenario? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Friday, November 12, 2010 1:38 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] SIGSEGV because of double free in msgDestruct > > Hello. > > I have rsyslog 5.6.0 on CentOS 5.5 with a slightly complex > configuration > and it's crashing. The complete configuration file is attached. The > crash > is perfectly reproducible and it happens very soon after the data > starts > arriving. The program was started with: > > rsyslogd -c5 -x -f rsyslog-datasink.conf > > I have two queues in order to have two thread pools. Input queue just > takes the message from UDP or TCP socket and uses omruleset to pass it > to > the output queue. The output queue then uses omprog to pass the message > to > the external program. Omprog blocks when the pipe to the external > program > is full, so I wanted to have unblocked threads to accept incoming > messages > (which will mostly use UDP). Hence, the configuration has two queues. > > It's possible that I made some error in the configuration and rsyslogd > is > crashing because I'm doing something that I wasn't supposed to do, but > it > didn't detect the faulty configuration early on. > > The whole thing works fine when I have only one queue (created with > $Ruleset) and input and omprog modules on it. But I'd really like to > use > two thread pools. It should be possible to reproduce this problem with > cat > as the omprog binary, although I haven't tried. > > One curiosity (probably unrelated to the problem): $GenerateConfigGraph > at > the end of the config file creates a picture which has only the main > queue, but the queues I configured with $Ruleset directives are not on > the > picture. > > The below is from gdb. The process was started from gdb, so there's no > call to sigsegvHdlr(), which can be seen in the core file when I start > rsyslogd on its own. > > (gdb) info threads > * 8 Thread 0xb4debb90 (LWP 11149) ConsumerReg (pThis=0x80b7988, > pWti=0x80b7cb8) at queue.c:1679 > 7 Thread 0xb57ecb90 (LWP 11148) 0x00d46402 in __kernel_vsyscall () > 6 Thread 0xb61edb90 (LWP 11147) msgDestruct (ppThis=0xb61ed1d4) at > msg.c:790 > 5 Thread 0xb6beeb90 (LWP 11146) 0x00d46402 in __kernel_vsyscall () > 4 Thread 0xb75efb90 (LWP 11145) 0x00d46402 in __kernel_vsyscall () > 3 Thread 0xb7ff0b90 (LWP 11144) 0x00d46402 in __kernel_vsyscall () > 2 Thread 0xb7ff1ac0 (LWP 11111) 0x00d46402 in __kernel_vsyscall () > (gdb) bt > #0 0x00d46402 in __kernel_vsyscall () > #1 0x00b2f040 in raise () from /lib/i686/nosegneg/libc.so.6 > #2 0x00b30a21 in abort () from /lib/i686/nosegneg/libc.so.6 > #3 0x00b67e3b in __libc_message () from /lib/i686/nosegneg/libc.so.6 > #4 0x00b70758 in free () from /lib/i686/nosegneg/libc.so.6 > #5 0x080612ee in msgDestruct (ppThis=0xb4deb1d4) at msg.c:816 > #6 0x08079e35 in DeleteProcessedBatch (pThis=0x80b7988, > pBatch=0x80b7cd0) > at queue.c:1404 > #7 0x0807a3b9 in DequeueConsumableElements (pThis=0x80b7988, > pWti=0x80b7cb8) > at queue.c:1441 > #8 DequeueConsumable (pThis=0x80b7988, pWti=0x80b7cb8) at queue.c:1489 > #9 0x0807a5d7 in DequeueForConsumer (pThis=0x80b7988, pWti=0x80b7cb8) > at queue.c:1626 > #10 ConsumerReg (pThis=0x80b7988, pWti=0x80b7cb8) at queue.c:1679 > #11 0x0807350e in wtiWorker (pThis=0x80b7cb8) at wti.c:315 > #12 0x08072e1f in wtpWorker (arg=0x80b7cb8) at wtp.c:381 > #13 0x00c9b869 in start_thread () from > /lib/i686/nosegneg/libpthread.so.0 > #14 0x00bd9e9e in clone () from /lib/i686/nosegneg/libc.so.6 > > The crash happens in msgDestruct() when it tries to free > pThis->rcvFrom.pfrominet. Valgrind says it's a double free problem. > > The queue mutex used by DequeueForConsumer seems to be properly locked > my > thread 8. From stack frame 10: > > (gdb) p *pThis->mut > $261 = {__data = {__lock = 2, __count = 0, __owner = 11149, __kind = 0, > __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, > __size = > "\002\000\000\000\000\000\000\000\215+\000\000\000\000\000\000\001\000\ > 000\000\000\000\000", __align = 2} > > The value for __lock is curious. It's usually 1 for locked or 0 for > unlocked, but it might have something to do with gdb. It's 1 in the > core > files. pThis->mutThrdMgmt is unlocked. > > I've checked omruleset code and it does a proper deep copy, as far as I > can tell. All the code in msg.c also seems fine. So I don't know what's > happening. > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr From epiphani at gmail.com Thu Feb 17 17:08:56 2011 From: epiphani at gmail.com (Aaron Wiebe) Date: Thu, 17 Feb 2011 11:08:56 -0500 Subject: [rsyslog] SIGSEGV because of double free in msgDestruct In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC90@GRFEXC.intern.adiscon.com> References: <20101112123746.GA26430@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DDC90@GRFEXC.intern.adiscon.com> Message-ID: I've seen a moderate amount of crashes in this area myself. I think this is one of those long-standing bugs that is hiding somewhere - it's visible in 4.6/4.7 as well, and I don't think its necessarily an issue tied to RHEL. -Aaron 2011/2/17 Rainer Gerhards : > Hi Dra?en, > > I dug this problem report out and tried to reproduce. I both tried my usual > platform under Fedora as well as CentOS 5.5 (64 bit). Unfortunately I did not > run into any trouble. Of course, I do not have the program you use, so this > may be a difference. I tested with a small program that just read stdin and > threw everything it read away. I also tested with writing to a file instead > of omprog. I tested on a quad core system and sent 10 million messages. > > Can you confirm that you still have some trouble with this scenario? > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar >> Sent: Friday, November 12, 2010 1:38 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] SIGSEGV because of double free in msgDestruct >> >> Hello. >> >> I have rsyslog 5.6.0 on CentOS 5.5 with a slightly complex >> configuration >> and it's crashing. The complete configuration file is attached. The >> crash >> is perfectly reproducible and it happens very soon after the data >> starts >> arriving. The program was started with: >> >> rsyslogd -c5 -x -f rsyslog-datasink.conf >> >> I have two queues in order to have two thread pools. Input queue just >> takes the message from UDP or TCP socket and uses omruleset to pass it >> to >> the output queue. The output queue then uses omprog to pass the message >> to >> the external program. Omprog blocks when the pipe to the external >> program >> is full, so I wanted to have unblocked threads to accept incoming >> messages >> (which will mostly use UDP). Hence, the configuration has two queues. >> >> It's possible that I made some error in the configuration and rsyslogd >> is >> crashing because I'm doing something that I wasn't supposed to do, but >> it >> didn't detect the faulty configuration early on. >> >> The whole thing works fine when I have only one queue (created with >> $Ruleset) and input and omprog modules on it. But I'd really like to >> use >> two thread pools. It should be possible to reproduce this problem with >> cat >> as the omprog binary, although I haven't tried. >> >> One curiosity (probably unrelated to the problem): $GenerateConfigGraph >> at >> the end of the config file creates a picture which has only the main >> queue, but the queues I configured with $Ruleset directives are not on >> the >> picture. >> >> The below is from gdb. The process was started from gdb, so there's no >> call to sigsegvHdlr(), which can be seen in the core file when I start >> rsyslogd on its own. >> >> (gdb) info threads >> * 8 Thread 0xb4debb90 (LWP 11149) ?ConsumerReg (pThis=0x80b7988, >> ? ? pWti=0x80b7cb8) at queue.c:1679 >> ? 7 Thread 0xb57ecb90 (LWP 11148) ?0x00d46402 in __kernel_vsyscall () >> ? 6 Thread 0xb61edb90 (LWP 11147) ?msgDestruct (ppThis=0xb61ed1d4) at >> msg.c:790 >> ? 5 Thread 0xb6beeb90 (LWP 11146) ?0x00d46402 in __kernel_vsyscall () >> ? 4 Thread 0xb75efb90 (LWP 11145) ?0x00d46402 in __kernel_vsyscall () >> ? 3 Thread 0xb7ff0b90 (LWP 11144) ?0x00d46402 in __kernel_vsyscall () >> ? 2 Thread 0xb7ff1ac0 (LWP 11111) ?0x00d46402 in __kernel_vsyscall () >> (gdb) bt >> #0 ?0x00d46402 in __kernel_vsyscall () >> #1 ?0x00b2f040 in raise () from /lib/i686/nosegneg/libc.so.6 >> #2 ?0x00b30a21 in abort () from /lib/i686/nosegneg/libc.so.6 >> #3 ?0x00b67e3b in __libc_message () from /lib/i686/nosegneg/libc.so.6 >> #4 ?0x00b70758 in free () from /lib/i686/nosegneg/libc.so.6 >> #5 ?0x080612ee in msgDestruct (ppThis=0xb4deb1d4) at msg.c:816 >> #6 ?0x08079e35 in DeleteProcessedBatch (pThis=0x80b7988, >> pBatch=0x80b7cd0) >> ? ? at queue.c:1404 >> #7 ?0x0807a3b9 in DequeueConsumableElements (pThis=0x80b7988, >> pWti=0x80b7cb8) >> ? ? at queue.c:1441 >> #8 ?DequeueConsumable (pThis=0x80b7988, pWti=0x80b7cb8) at queue.c:1489 >> #9 ?0x0807a5d7 in DequeueForConsumer (pThis=0x80b7988, pWti=0x80b7cb8) >> ? ? at queue.c:1626 >> #10 ConsumerReg (pThis=0x80b7988, pWti=0x80b7cb8) at queue.c:1679 >> #11 0x0807350e in wtiWorker (pThis=0x80b7cb8) at wti.c:315 >> #12 0x08072e1f in wtpWorker (arg=0x80b7cb8) at wtp.c:381 >> #13 0x00c9b869 in start_thread () from >> /lib/i686/nosegneg/libpthread.so.0 >> #14 0x00bd9e9e in clone () from /lib/i686/nosegneg/libc.so.6 >> >> The crash happens in msgDestruct() when it tries to free >> pThis->rcvFrom.pfrominet. Valgrind says it's a double free problem. >> >> The queue mutex used by DequeueForConsumer seems to be properly locked >> my >> thread 8. From stack frame 10: >> >> (gdb) p *pThis->mut >> $261 = {__data = {__lock = 2, __count = 0, __owner = 11149, __kind = 0, >> ? ? __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, >> ? __size = >> "\002\000\000\000\000\000\000\000\215+\000\000\000\000\000\000\001\000\ >> 000\000\000\000\000", __align = 2} >> >> The value for __lock is curious. It's usually 1 for locked or 0 for >> unlocked, but it might have something to do with gdb. It's 1 in the >> core >> files. pThis->mutThrdMgmt is unlocked. >> >> I've checked omruleset code and it does a proper deep copy, as far as I >> can tell. All the code in msg.c also seems fine. So I don't know what's >> happening. >> >> -- >> ?.-. ? .-. ? ?Yes, I am an agent of Satan, but my duties are largely >> (_ ?\ / ?_) ? ceremonial. >> ? ? ?| >> ? ? ?| ? ? ? ?dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From dave at fly.srk.fer.hr Thu Feb 17 17:14:21 2011 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 17 Feb 2011 17:14:21 +0100 Subject: [rsyslog] SIGSEGV because of double free in msgDestruct In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC90@GRFEXC.intern.adiscon.com> References: <20101112123746.GA26430@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DDC90@GRFEXC.intern.adiscon.com> Message-ID: <20110217161421.GA31257@fly.srk.fer.hr> Rainer Gerhards wrote: > Hi Dra?en, > > I dug this problem report out and tried to reproduce. I both tried my usual > platform under Fedora as well as CentOS 5.5 (64 bit). Unfortunately I did not > run into any trouble. Of course, I do not have the program you use, so this > may be a difference. I tested with a small program that just read stdin and > threw everything it read away. I also tested with writing to a file instead > of omprog. I tested on a quad core system and sent 10 million messages. > > Can you confirm that you still have some trouble with this scenario? I meant to test that again, but I'm rather bussy at the moment and I expect to have some free time for this eary next week. I'll let you know. FWIW, that crash was on CentOS 5.5 32-bit. I don't know if that's important. It might have something to do with the messages, though, as they weren't random garbage. I'll retest as soon as I can. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From tbergfeld at hq.adiscon.com Fri Feb 18 13:41:01 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 18 Feb 2011 13:41:01 +0100 Subject: [rsyslog] rsyslog 6.1.4 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDCA3@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 6.1.4, the new v6-devel. This release contains all changes and enhancements of the v5 branch. Futher more there are some new bug fixes. Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-6-1-4-devel/ Download: http://www.rsyslog.com/rsyslog-6-1-4-devel/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From toddmichael at gmail.com Mon Feb 21 06:24:08 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Sun, 20 Feb 2011 21:24:08 -0800 Subject: [rsyslog] lmnsd_ptcp and tcp logging work in debug mode, but not daemon Message-ID: <1CA32399-6568-4F42-A436-F80B4E3077A8@gmail.com> Ran into an interesting problem. Deploying rsyslog to 4 servers today, all identical CentOS 5.5 boxes: two worked and two do not. Server/App info: Linux servero01 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux rsyslog-3.22.1-3.el5_5.1 All local logging works, but TCP logging does not. When not working I don't even see any packets being sent when I run tcpdump on the local box. Here's what I find in logs when I start rsyslog (rsyslogd -c 3): Feb 20 21:01:50 server01 rsyslogd: registering object 'nsd_ptcp' failed with error code -2061 Feb 20 21:01:50 server01 rsyslogd-2068: could not load module '/usr/lib64/rsyslog/lmnsd_ptcp.so', rsyslog error -2061 This evening, I decided to throw into debug to try to resolve or at least get more info so smarter people than me (this list) could help me diagnose. When I threw in debug (same startup options other than -d), low and behold, it worked fine. I dumped the debug output to file and searched for lmnsd_ptcp and here's what I see: 3513.740387000:action 1 queue[DA]:Reg/w0: caller requested object 'nsd_ptcp', not found (iRet -3003) 3513.740402000:action 1 queue[DA]:Reg/w0: Requested to load module 'lmnsd_ptcp' 3513.740434000:action 1 queue[DA]:Reg/w0: loading module '/usr/lib64/rsyslog/lmnsd_ptcp.so' 3513.740603000:action 1 queue[DA]:Reg/w0: source file nsd_ptcp.c requested reference for module 'lmnetstrms', reference count now 3 3513.740880000:action 1 queue[DA]:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 1 3513.741541000:action 2 queue[DA]:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 2 3513.750888000:main thread: Loaded Module: Name='lmnsd_ptcp', IFVersion=4, type=library module. 3553.161187000:action 2 queue:Reg/w0: file netstrms.c released module 'lmnsd_ptcp', reference count now 1 3553.161208000:action 2 queue:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 2 Any idea why lmnsd (and tcp logging (i'm sure related)) would work in debug mode, but not otherwise? Any clue on where to go next with this is appreciated. Todd Michael Bushnell toddmichael at gmail.com From toddmichael at gmail.com Mon Feb 21 21:34:48 2011 From: toddmichael at gmail.com (Todd Michael Bushnell) Date: Mon, 21 Feb 2011 12:34:48 -0800 Subject: [rsyslog] lmnsd_ptcp and tcp logging work in debug mode, but not daemon In-Reply-To: <1CA32399-6568-4F42-A436-F80B4E3077A8@gmail.com> References: <1CA32399-6568-4F42-A436-F80B4E3077A8@gmail.com> Message-ID: disregard. someone enabled selinux=enforce on two of my boxes (grrr). all is good. sorry for the fp. On Feb 20, 2011, at 9:24 PM, Todd Michael Bushnell wrote: > Ran into an interesting problem. Deploying rsyslog to 4 servers today, all identical CentOS 5.5 boxes: two worked and two do not. Server/App info: > > Linux servero01 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > rsyslog-3.22.1-3.el5_5.1 > > All local logging works, but TCP logging does not. When not working I don't even see any packets being sent when I run tcpdump on the local box. Here's what I find in logs when I start rsyslog (rsyslogd -c 3): > > Feb 20 21:01:50 server01 rsyslogd: registering object 'nsd_ptcp' failed with error code -2061 > Feb 20 21:01:50 server01 rsyslogd-2068: could not load module '/usr/lib64/rsyslog/lmnsd_ptcp.so', rsyslog error -2061 > > This evening, I decided to throw into debug to try to resolve or at least get more info so smarter people than me (this list) could help me diagnose. When I threw in debug (same startup options other than -d), low and behold, it worked fine. I dumped the debug output to file and searched for lmnsd_ptcp and here's what I see: > > 3513.740387000:action 1 queue[DA]:Reg/w0: caller requested object 'nsd_ptcp', not found (iRet -3003) > 3513.740402000:action 1 queue[DA]:Reg/w0: Requested to load module 'lmnsd_ptcp' > 3513.740434000:action 1 queue[DA]:Reg/w0: loading module '/usr/lib64/rsyslog/lmnsd_ptcp.so' > 3513.740603000:action 1 queue[DA]:Reg/w0: source file nsd_ptcp.c requested reference for module 'lmnetstrms', reference count now 3 > 3513.740880000:action 1 queue[DA]:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 1 > 3513.741541000:action 2 queue[DA]:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 2 > 3513.750888000:main thread: Loaded Module: Name='lmnsd_ptcp', IFVersion=4, type=library module. > 3553.161187000:action 2 queue:Reg/w0: file netstrms.c released module 'lmnsd_ptcp', reference count now 1 > 3553.161208000:action 2 queue:Reg/w0: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 2 > > Any idea why lmnsd (and tcp logging (i'm sure related)) would work in debug mode, but not otherwise? Any clue on where to go next with this is appreciated. > > > From lists at spacerat.ch Tue Feb 22 15:44:55 2011 From: lists at spacerat.ch (Spacerat) Date: Tue, 22 Feb 2011 15:44:55 +0100 Subject: [rsyslog] CentOS/RHEL 4.x rpms Message-ID: <37c385a2394f39061c7cc1a4da322d0c@spacerat.ch> Hi there As the link at to the CentOS/RHEL 4.x Packages is dead, i ask the list for an appropriate rpm source.. Is there anything out there? TIA Spacerat From eschoeller at users.sourceforge.net Wed Feb 23 03:19:25 2011 From: eschoeller at users.sourceforge.net (Eric Schoeller) Date: Tue, 22 Feb 2011 19:19:25 -0700 Subject: [rsyslog] Reliable forwarding Not Working In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC4B@GRFEXC.intern.adiscon.com> References: <397BF207-F38D-495E-8197-4C20B8B3DBF9@gmail.com><4D572548.4030907@users.sourceforge.net><9B6E2A8877C38245BFB15CC491A11DA71DDC46@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDC4B@GRFEXC.intern.adiscon.com> Message-ID: <4D646EAD.8000201@users.sourceforge.net> From brett.dellegrazie at gmail.com Wed Feb 23 10:52:47 2011 From: brett.dellegrazie at gmail.com (Brett Delle Grazie) Date: Wed, 23 Feb 2011 09:52:47 +0000 Subject: [rsyslog] Config guidance requested Message-ID: Hi, I'm using: RHEL 5.6 (x86_64) kernel 2.6.18-238.1.1.el5 rsyslog-3.22.1-3.el5_5.1 I've got a single loghost working in TCP / UDP mode receiving logs from various systems (rsyslog based and normal syslog based). I'm looking for guidance on the configs below - any recommendations for improving performance would be appreciated. Comments are elided for space. The normal config is: $ModLoad imklog $ModLoad imuxsock $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat *.* @@loghost.xxx.xxx *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog;RSYSLOG_TraditionalFileFormat cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot.log The loghost config is: $ModLoad imklog $ModLoad imuxsock $ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template DailyPerHostLogs,"/var/log/syslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/messages.log" :fromhost-ip, !isequal, "127.0.0.1" -?DailyPerHostLogs;TraditionalFormatWithPRI & ~ *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog;RSYSLOG_TraditionalFileFormat cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot.log Thanks, -- Best Regards, Brett Delle Grazie From rgerhards at hq.adiscon.com Wed Feb 23 11:01:04 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Feb 2011 11:01:04 +0100 Subject: [rsyslog] Config guidance requested In-Reply-To: References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDCFB@GRFEXC.intern.adiscon.com> Quite honestly, the best advise - if you have performance problems - is to update to at least version 4. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Brett Delle Grazie > Sent: Wednesday, February 23, 2011 10:53 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Config guidance requested > > Hi, > > I'm using: > RHEL 5.6 (x86_64) > kernel 2.6.18-238.1.1.el5 > rsyslog-3.22.1-3.el5_5.1 > > I've got a single loghost working in TCP / UDP mode receiving logs > from various systems (rsyslog based and normal syslog based). > I'm looking for guidance on the configs below - any recommendations > for improving performance would be appreciated. > > Comments are elided for space. > > The normal config is: > > $ModLoad imklog > $ModLoad imuxsock > $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat > > *.* @@loghost.xxx.xxx > > *.info;mail.none;authpriv.none;cron.none > /var/log/messages > authpriv.* /var/log/secure > mail.* > -/var/log/maillog;RSYSLOG_TraditionalFileFormat > cron.* /var/log/cron > *.emerg * > uucp,news.crit > /var/log/spooler > local7.* > /var/log/boot.log > > > The loghost config is: > > $ModLoad imklog > $ModLoad imuxsock > $ModLoad imudp > $UDPServerRun 514 > $ModLoad imtcp > $InputTCPServerRun 514 > > $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% > %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > $template > DailyPerHostLogs,"/var/log/syslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/me > ssages.log" > :fromhost-ip, !isequal, "127.0.0.1" - > ?DailyPerHostLogs;TraditionalFormatWithPRI > & ~ > > *.info;mail.none;authpriv.none;cron.none > /var/log/messages > authpriv.* /var/log/secure > mail.* > -/var/log/maillog;RSYSLOG_TraditionalFileFormat > cron.* /var/log/cron > *.emerg * > uucp,news.crit > /var/log/spooler > local7.* > /var/log/boot.log > > Thanks, > > -- > Best Regards, > > Brett Delle Grazie > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Wed Feb 23 11:02:57 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 23 Feb 2011 11:02:57 +0100 Subject: [rsyslog] rsyslog 5.7.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDCFC@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.7.5, the new v5-beta. This is a maintainance release. Most importantly, it contains a fix for regressions in imuxsock which could cause loss of information and also induce problem in other tools, e.g. logcheck. Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-5-7-5/ Download: http://www.rsyslog.com/rsyslog-5-7-5-v5-beta/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From brett.dellegrazie at gmail.com Wed Feb 23 11:06:24 2011 From: brett.dellegrazie at gmail.com (Brett Delle Grazie) Date: Wed, 23 Feb 2011 10:06:24 +0000 Subject: [rsyslog] Config guidance requested In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDCFB@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDCFB@GRFEXC.intern.adiscon.com> Message-ID: Hi Rainer, On 23 February 2011 10:01, Rainer Gerhards wrote: > Quite honestly, the best advise - if you have performance problems - is to > update to at least version 4. Thanks - and I will certainly attempt to do that (to current stable version) but in the interim, are there any obvious config issues? e.g. Should I be using LinkedList for the main queue? Particularly on the loghost? > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Brett Delle Grazie >> Sent: Wednesday, February 23, 2011 10:53 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Config guidance requested >> >> Hi, >> >> I'm using: >> RHEL 5.6 (x86_64) >> kernel ?2.6.18-238.1.1.el5 >> rsyslog-3.22.1-3.el5_5.1 >> >> I've got a single loghost working in TCP / UDP mode receiving logs >> from various systems (rsyslog based and normal syslog based). >> I'm looking for guidance on the configs below - any recommendations >> for improving performance would be appreciated. >> >> Comments are elided for space. >> >> The normal config is: >> >> $ModLoad imklog >> $ModLoad imuxsock >> $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat >> >> *.* @@loghost.xxx.xxx >> >> *.info;mail.none;authpriv.none;cron.none >> /var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* >> -/var/log/maillog;RSYSLOG_TraditionalFileFormat >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/cron >> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * >> uucp,news.crit >> /var/log/spooler >> local7.* >> /var/log/boot.log >> >> >> The loghost config is: >> >> $ModLoad imklog >> $ModLoad imuxsock >> $ModLoad imudp >> $UDPServerRun 514 >> $ModLoad imtcp >> $InputTCPServerRun 514 >> >> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% >> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" >> $template >> DailyPerHostLogs,"/var/log/syslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/me >> ssages.log" >> :fromhost-ip, !isequal, "127.0.0.1" - >> ?DailyPerHostLogs;TraditionalFormatWithPRI >> & ~ >> >> *.info;mail.none;authpriv.none;cron.none >> /var/log/messages >> authpriv.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/secure >> mail.* >> -/var/log/maillog;RSYSLOG_TraditionalFileFormat >> cron.* ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?/var/log/cron >> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * >> uucp,news.crit >> /var/log/spooler >> local7.* >> /var/log/boot.log >> Thanks, -- Best Regards, Brett Delle Grazie From rgerhards at hq.adiscon.com Wed Feb 23 11:09:12 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Feb 2011 11:09:12 +0100 Subject: [rsyslog] Config guidance requested In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DDCFB@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDCFD@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: Brett Delle Grazie [mailto:brett.dellegrazie at gmail.com] > Sent: Wednesday, February 23, 2011 11:06 AM > To: Rainer Gerhards > Cc: rsyslog-users > Subject: Re: [rsyslog] Config guidance requested > > Hi Rainer, > > On 23 February 2011 10:01, Rainer Gerhards > wrote: > > Quite honestly, the best advise - if you have performance problems - > is to > > update to at least version 4. > > Thanks - and I will certainly attempt to do that (to current stable > version) but in the interim, > are there any obvious config issues? What's your volume and what's the problem? I don't see anything obviously bad. > > e.g. Should I be using LinkedList for the main queue? Particularly on > the loghost? I don't know the default for v3, but any memory based mode should perform almost equally well (with fixedArray being a very tiny bit faster). But the point is that on average v4 is around 3 to 5 *times* faster than v3. Rainer > > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Brett Delle Grazie > >> Sent: Wednesday, February 23, 2011 10:53 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] Config guidance requested > >> > >> Hi, > >> > >> I'm using: > >> RHEL 5.6 (x86_64) > >> kernel ?2.6.18-238.1.1.el5 > >> rsyslog-3.22.1-3.el5_5.1 > >> > >> I've got a single loghost working in TCP / UDP mode receiving logs > >> from various systems (rsyslog based and normal syslog based). > >> I'm looking for guidance on the configs below - any recommendations > >> for improving performance would be appreciated. > >> > >> Comments are elided for space. > >> > >> The normal config is: > >> > >> $ModLoad imklog > >> $ModLoad imuxsock > >> $ActionForwardDefaultTemplate RSYSLOG_ForwardFormat > >> > >> *.* @@loghost.xxx.xxx > >> > >> *.info;mail.none;authpriv.none;cron.none > >> /var/log/messages > >> authpriv.* > ?/var/log/secure > >> mail.* > >> -/var/log/maillog;RSYSLOG_TraditionalFileFormat > >> cron.* > ?/var/log/cron > >> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * > >> uucp,news.crit > >> /var/log/spooler > >> local7.* > >> /var/log/boot.log > >> > >> > >> The loghost config is: > >> > >> $ModLoad imklog > >> $ModLoad imuxsock > >> $ModLoad imudp > >> $UDPServerRun 514 > >> $ModLoad imtcp > >> $InputTCPServerRun 514 > >> > >> $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% > >> %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > >> $template > >> > DailyPerHostLogs,"/var/log/syslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/me > >> ssages.log" > >> :fromhost-ip, !isequal, "127.0.0.1" - > >> ?DailyPerHostLogs;TraditionalFormatWithPRI > >> & ~ > >> > >> *.info;mail.none;authpriv.none;cron.none > >> /var/log/messages > >> authpriv.* > ?/var/log/secure > >> mail.* > >> -/var/log/maillog;RSYSLOG_TraditionalFileFormat > >> cron.* > ?/var/log/cron > >> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? * > >> uucp,news.crit > >> /var/log/spooler > >> local7.* > >> /var/log/boot.log > >> > > Thanks, > > -- > Best Regards, > > Brett Delle Grazie From lists at spacerat.ch Wed Feb 23 11:22:10 2011 From: lists at spacerat.ch (Spacerat) Date: Wed, 23 Feb 2011 11:22:10 +0100 Subject: [rsyslog] CentOS/RHEL 4.x rpms In-Reply-To: <37c385a2394f39061c7cc1a4da322d0c@spacerat.ch> References: <37c385a2394f39061c7cc1a4da322d0c@spacerat.ch> Message-ID: On Tue, 22 Feb 2011 15:44:55 +0100, Spacerat wrote: > Hi there > > As the link at to the CentOS/RHEL 4.x Packages is dead, i ask the > list for an appropriate rpm source.. > > Is there anything out there? > > TIA > > Spacerat Found it on my own, very old, but looks good withmysql logging. http://packages.sw.be/rsyslog/ From brett.dellegrazie at gmail.com Wed Feb 23 11:26:26 2011 From: brett.dellegrazie at gmail.com (Brett Delle Grazie) Date: Wed, 23 Feb 2011 10:26:26 +0000 Subject: [rsyslog] Config guidance requested In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDCFD@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DDCFB@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DDCFD@GRFEXC.intern.adiscon.com> Message-ID: Hi Rainer, On 23 February 2011 10:09, Rainer Gerhards wrote: >> -----Original Message----- >> From: Brett Delle Grazie [mailto:brett.dellegrazie at gmail.com] >> Sent: Wednesday, February 23, 2011 11:06 AM >> To: Rainer Gerhards >> Cc: rsyslog-users >> Subject: Re: [rsyslog] Config guidance requested >> >> Hi Rainer, >> >> On 23 February 2011 10:01, Rainer Gerhards >> wrote: >> > Quite honestly, the best advise - if you have performance problems - >> is to >> > update to at least version 4. >> >> Thanks - and I will certainly attempt to do that (to current stable >> version) but in the interim, >> are there any obvious config issues? > > > What's your volume and what's the problem? I don't see anything obviously > bad. Sometimes appears as if certain hosts stop logging both locally and remotely. i.e. /var/log/messages is empty. > >> >> e.g. Should I be using LinkedList for the main queue? Particularly on >> the loghost? > I don't know the default for v3, but any memory based mode should perform > almost equally well (with fixedArray being a very tiny bit faster). Default is fixed array according to docs for that version. > > But the point is that on average v4 is around 3 to 5 *times* faster than v3. > Noted - and will attempt but causes maintenance headache since version must be externally tracked. I suppose this will change with RHEL6 Thanks for your help and your time. -- Best Regards, Brett Delle Grazie From pgollucci at p6m7g8.com Fri Feb 25 06:05:50 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Fri, 25 Feb 2011 00:05:50 -0500 Subject: [rsyslog] Fwd: Re: rsyslog5 core dumps a-plenty In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDB4F@GRFEXC.intern.adiscon.com> References: <4D4B3F22.5030300@p6m7g8.com> <9B6E2A8877C38245BFB15CC491A11DA71DDB4F@GRFEXC.intern.adiscon.com> Message-ID: <4D6738AE.1070405@p6m7g8.com> On 2/4/2011 2:59 AM, Rainer Gerhards wrote: > Could you provide me a set of sample messages so that I can check? If you > like, you can mail me privately. I think the root problem is somewhere > message-induced and, if so, it would be very useful to see how it could be > triggered. I already use a large set of "malformed" messages (including a few > gig of truely random data), but none of them trigger the issue :( Sorry, I got swamped at $work with our annual DevOps presentation. I'll follow up tomorrow with some message payloads. I've no issue with public, I'll snip anything I care about if its there at all. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From pgollucci at p6m7g8.com Fri Feb 25 06:11:25 2011 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Fri, 25 Feb 2011 00:11:25 -0500 Subject: [rsyslog] snmp compile borked on fbsd 8+ Message-ID: <4D6739FD.2050003@p6m7g8.com> Hi all, http://portsmon.freebsd.org/portoverview.py?category=sysutils&portname=rsyslog3-snmp s/3/4/5/6/ etc... I can probably fix this one but haven't had time to look at it yet. If you beat me to it great! If not, I'll get around to it eventually. On another note, http://pointyhat.freebsd.org/errorlogs/sparc64-errorlogs/e.7.20101015091133/rsyslog-3.22.2.log.bz2 utmpx.h conversion issues. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From tbergfeld at hq.adiscon.com Fri Feb 25 15:11:18 2011 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 25 Feb 2011 15:11:18 +0100 Subject: [rsyslog] rsyslog 5.7.6 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDD36@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.7.6, the new v5-beta. This is a maintenance release. It contains fixes for a memory leak and a potential abort condition. Both are triggered under non-default (and relatively seldom used) configurations. However, if these conditions are present, the bug is a really bad one. Details to these bugs: fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=225 fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=226 fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=218 Please see the ChangeLog for more details. ChangeLog: http://www.rsyslog.com/changelog-for-5-7-6-v5-beta/ Download: http://www.rsyslog.com/rsyslog-5-7-6-v5-beta/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From ephemeric at gmail.com Fri Feb 25 15:25:53 2011 From: ephemeric at gmail.com (Robert Gabriel) Date: Fri, 25 Feb 2011 16:25:53 +0200 Subject: [rsyslog] Multiple RELP and Action Queues per Ruleset Message-ID: Hello all, I'm very new to rsyslog, so please bear with me... I have managed to get something going, below is what I wanted. 1. High throughput (> 200K). Using multiple queues. 2. Memory-type disk-assisted multiple queues (multiple action queues?). How does $ActionQueue* fit in with multiple rulesets? Is it possible with my existing setup? I would like to spool to disk if any of the rulesets cannot forward by TCP for whatever reason. This reliability is quite important. I see there is $RulesetCreateMainQueue available since 5.3.5+ but I cannot use this for now as I cannot sacrifice stability on the RHEL/CentOS platform and 5.6.x packages won't be available for quite some time (CentOS 6 is only on rsyslog-4.4.2-3.el6.i686.rpm!). 3. Reliable forwarding using RELP. Is there something like $InputRELPServerRun? How, if possible can we do RELP with each ruleset? I tried using $InputRELPServerRun instead of $InputTCPServerRun after $RuleSet but it broke the ruleset match. Can we do something fancy like bind a TCP port to a ruleset and start a RELP listener and forward from the RELP port to the plain TCP port (just guessing here)? Possibly some trickery with Unix domain sockets? OpenSSH local port forwarding? 4. Single/multiple files (option to switch on/off to troubleshoot). In the long run, what seems to better? We have had a debate over this but maybe users can offer real world experience please? 5. Compression. Compressed RELP seemed to work in the forwarding part (no errors in rsyslog restart) but is this a desired config? Should we be doing compression elsewhere if possible like in an ASIC in our Juniper firewall to save precious bandwidth over our WAN link? Has anyone tried OpenSSH compression and local port forwarding? 6. Expression and property-based filters. Which is faster/preferred? 7. Store, filter and forward. What is better in terms of OS throughput, single or multiple files? We are looking at very high throughput like > 200K. If I have ten ruleset inputs all writing to the same file what happens? I'm interested in how concurrent writes are implemented. 8. Versions: Collector: CentOS-5.5 x86_64 on an IBM x3550 M2. kernel 2.6.18-194.32.1.el5 rsyslog4-relp-4.6.5-2.ius.el5.x86_64.rpm. rsyslog4-4.6.5-2.ius.el5.x86_64.rpm librelp-0.1.1-3.el5.kb.x86_64.rpm Relay: CentOS-5.5 x86_64 on an IBM BladeCenter HS22 kernel 2.6.18-194.32.1.el5 rsyslog4-relp-4.6.5-2.ius.el5.x86_64.rpm. rsyslog4-4.6.5-2.ius.el5.x86_64.rpm librelp-0.1.1-3.el5.kb.x86_64.rpm 9. Config: Client collector: For now, I cannot change anything on the client side due to contractual agreements hence the filter match and then forward which will become a ruleset with each source sending to it's own TCP port. At the moment everything is coming in on TCP 514. $ModLoad imklog $ModLoad imuxsock $ModLoad omrelp $ModLoad imudp $UDPServerRun 514 $UDPServerTimeRequery 10 $ModLoad imtcp $InputTCPMaxSessions 500 $InputTCPServerRun 514 $MainMsgQueueSize 100000 $WorkDirectory /media/data/var/spool/ryslog $ActionQueueType LinkedList $ActionQueueFileName gdf $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on if $msg contains 'SymantecServer' then @@(z9)172.nnn.nnn.nnn:4515;RSYSLOG_ForwardFormat if $msg contains 'D75FADC001' then @@(z9)172.nnn.nnn.nnn:4516;RSYSLOG_ForwardFormat if $msg contains 'fw.log' then @@(z9)172.nnn.nnn.nnn:4517;RSYSLOG_ForwardFormat # The below I've got wrong somewhere. Corrections please? #if $pri-text isequal 'local0.info' then @@(z9)172.20.10.30:4517;RSYSLOG_ForwardFormat #:msg, contains, 'SymantecServer' @@(z9)172.20.10.30:4515;RSYSLOG_ForwardFormat #:msg, contains, 'D75FADC001' @@(z9)172.20.10.30:4516;RSYSLOG_ForwardFormat #:pri, isequal, '134' @@(z9)172.20.10.30:4517;RSYSLOG_ForwardFormat :fromhost-ip, isequal, '172.20.26.245' -/media/data/var/log/gdf & ~ local0.info -/media/data/var/log/gdf & ~ kern.* -/var/log/kernel *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* -/var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron *.emerg * uucp,news.crit -/var/log/spooler local7.* -/var/log/boot Relay: The relay will forward the separate TCP streams onto Splunk data inputs as TCP ports and indexed. We want to separate the streams into separate indexes to enable easier analysis of different sourcetypes. Normalisation could be done as far back as the log source, on the collector or in Splunk itself. The end point for the data will be in AlienVault to do SIEM and event correlation. $ModLoad imklog $ModLoad imuxsock $ModLoad imrelp $InputRELPServerRun 4514 $ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 # Is this right ordering? # How does it pertain to the below rulesets? $WorkDirectory /var/spool/rsyslog/ $ActionQueueFileName gdf $ActionQueueType LinkedList $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on kern.* -/var/log/kernel *.info;mail.none;authpriv.none;cron.none -/var/log/messages authpriv.* -/var/log/secure mail.* -/var/log/maillog cron.* -/var/log/cron *.emerg * uucp,news.crit -/var/log/spooler local7.* -/var/log/boot $Ruleset FW1 #:pri, isequal, "134" then @@(z9)172.nnn.nnn.nnn:4517;RSYSLOG_ForwardFormat #*.* -/var/log/fw1 *.* @@172.nnn.nnn.nnn:44517;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset FW1 $InputTCPServerRun 4517 $Ruleset D75FADC001 #:msg, contains, "D75FADC001" @@(z9)172.nnn.nnn.nnn:4516;RSYSLOG_ForwardFormat #*.* -/var/log/D75FADC001 # Will this work? $WorkDirectory /var/spool/rsyslog/ $ActionQueueFileName D75FADC001 $ActionQueueType LinkedList $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.* @@172.nnn.nnn.nnn:44516;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset D75FADC001 $InputTCPServerRun 4516 $Ruleset SEP #:msg, contains, "SymantecServer" @@(z9)172.nnn.nnn.nnn:4515;RSYSLOG_ForwardFormat #*.* -/var/log/sep *.* @@nnn.nnn.nnn.nnn:44515;RSYSLOG_ForwardFormat $InputTCPServerBindRuleset SEP $InputTCPServerRun 4515 From david at lang.hm Fri Feb 25 16:36:55 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 25 Feb 2011 07:36:55 -0800 (PST) Subject: [rsyslog] Multiple RELP and Action Queues per Ruleset In-Reply-To: References: Message-ID: a couple of questions and comments On Fri, 25 Feb 2011, Robert Gabriel wrote: > Hello all, > > I'm very new to rsyslog, so please bear with me... > > I have managed to get something going, below is what I wanted. > > 1. High throughput (> 200K). 200K what? 200K bytes/min, messages/min, bytes/sec, message/sec, ??? > Using multiple queues. > > > 2. Memory-type disk-assisted multiple queues (multiple action queues?). > > How does $ActionQueue* fit in with multiple rulesets? > Is it possible with my existing setup? > I would like to spool to disk if any of the rulesets cannot forward by > TCP for whatever reason. > This reliability is quite important. > > I see there is $RulesetCreateMainQueue available since 5.3.5+ > but I cannot use this for now as I cannot sacrifice stability > on the RHEL/CentOS platform and 5.6.x packages won't be available > for quite some time (CentOS 6 is only on rsyslog-4.4.2-3.el6.i686.rpm!). sticking with the RHEL/CentOS versions is going to really hurt you, the performance improvements since that version are very significant and RedHat is never going to upgrade RHEL 6.0 to a newer version (part of their 'stability' claim), so you are going to be stuck on that version until the next release of RHEL. > 4. Single/multiple files (option to switch on/off to troubleshoot). > > In the long run, what seems to better? We have had a debate over this > but maybe users can offer real world experience please? personally, I hate nor being able to find something because logging wasn't enabled, so I lean towards the 'put everything in one file' mode. > 5. Compression. > > Compressed RELP seemed to work in the forwarding part (no errors in > rsyslog restart) but is this a desired config? > Should we be doing compression elsewhere if possible like in an ASIC > in our Juniper firewall to save > precious bandwidth over our WAN link? > Has anyone tried OpenSSH compression and local port forwarding? text like rsyslog does compress well in bulk, but individual messages are seldom large enough to be worth compressing. a compressed channel may help, but it may also delay the messages as the compression is trying to fill it's buffer before compressing and sending it. > > 6. Expression and property-based filters. > > Which is faster/preferred? property based filters are significantly faster. > 7. Store, filter and forward. > > What is better in terms of OS throughput, single or multiple files? > We are looking at very high throughput like > 200K. > If I have ten ruleset inputs all writing to the same file what happens? > I'm interested in how concurrent writes are implemented. there are versions where concurrent writes are not handled well at all, I don't know the particular version you are using. > The relay will forward the separate TCP streams onto Splunk data > inputs as TCP ports and indexed. > We want to separate the streams into separate indexes to enable easier > analysis of different sourcetypes. > Normalisation could be done as far back as the log source, on the > collector or in Splunk itself. > The end point for the data will be in AlienVault to do SIEM and event > correlation. how many different sources do you have? there is a lot of overhead in splunk per index, and each index really wants it's own chunk of memory so that it can efficiently update itself as new logs are inserted, this is something on the orderof 10G per index by default. splunk is pretty good about being able to search for subsets within an index. Instead of making each host it's own index, consider splitting the indexes by type (juniper vs linux). David Lang From ephemeric at gmail.com Mon Feb 28 06:45:03 2011 From: ephemeric at gmail.com (Robert Gabriel) Date: Mon, 28 Feb 2011 07:45:03 +0200 Subject: [rsyslog] Multiple RELP and Action Queues per Ruleset In-Reply-To: References: Message-ID: On 25 February 2011 17:36, wrote: > a couple of questions and comments >> 1. High throughput (> 200K). 200K messages per hour. > sticking with the RHEL/CentOS versions is going to really hurt you, the > performance improvements since that version are very significant and RedHat > is never going to upgrade RHEL 6.0 to a newer version (part of their > 'stability' claim), so you are going to be stuck on that version until the > next release of RHEL. Can we compile pristine sources or should certain patches be applied as per RHEL/CentOS? > there are versions where concurrent writes are not handled well at all, I > don't know the particular version you are using. CentOS-5.5 x86_64 on an IBM x3550 M2. kernel 2.6.18-194.32.1.el5 rsyslog4-relp-4.6.5-2.ius.el5.x86_64.rpm. rsyslog4-4.6.5-2.ius.el5.x86_64.rpm librelp-0.1.1-3.el5.kb.x86_64.rpm > how many different Splunk sources do you have? There will be plenty. From Jacek.Nykis at betfair.com Mon Feb 28 15:38:39 2011 From: Jacek.Nykis at betfair.com (Jacek Nykis) Date: Mon, 28 Feb 2011 14:38:39 +0000 Subject: [rsyslog] Rsyslog hangs once very few weeks Message-ID: Hi, I am running rsyslog server on my clients and I store events locally and also forward them to remote rsyslog servers. My clients are CentOS 5.5 running rsyslogd 3.22.1. All of them forward data to 2 rsyslog servers using omrelp module. Central servers are running RHEL 6 and rsyslog 4.6.2. Everything works fine most of the time but some of my servers start experiencing some problems once every few weeks. When this happens random clients stop logging any data to the disk and they stop forwarding anything to remote servers. This also affects system responsiveness, it appears that many daemons are waiting for syslog operation to timeout which takes some time. For example ssh login is delayed by about 10 seconds, the same happens to sudo commands and some others as well. The only way I found to resolve the problem is to restart rsyslog process (kill -HUP does not work). I attached strace to hanged process and it hanged with following system call: select(1, NULL, NULL, NULL, {56238, 855000} What is also strange kill -HUP did not work properly. I attached strace to the process during kill -HUP, output below: ) = -1 EINTR (Interrupted system call) --- SIGHUP (Hangup) @ 0 (0) --- rt_sigaction(SIGHUP, {0x409520, [], SA_RESTORER, 0x3b93e0eb10}, NULL, 8) = 0 rt_sigreturn(0x1) = -1 EINTR (Interrupted system call) futex(0x2aaac0009d1c, FUTEX_WAIT_PRIVATE, 1, NULL output hangs on futex system call. My configuration on the clients: ## Modules $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) $ModLoad imtcp.so # provides TCP syslog reception and GSS-API $InputTCPServerRun 514 # start up TCP listener at port 514 $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 $ModLoad omrelp.so # provides RELP syslog transmission ## Global Directives # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $EscapeControlCharactersOnReceive off ## Rules #Log anything (except mail) of level info or higher. #Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none;user.none -/var/log/messages #The authpriv file has restricted access. authpriv.* /var/log/secure #Log all the mail messages in one place. mail.* -/var/log/maillog #Log cron stuff cron.* -/var/log/cron #Everybody gets emergency messages *.emerg * #Save boot messages also to boot.log local7.* /var/log/boot.log ## Include configs from the /etc/rsyslog.d directory $IncludeConfig /etc/rsyslog.d/*.conf ## Log remotely with a fully-qualified hostname $template fmtFQDN,"<%pri%>%timegenerated% hostname %syslogtag% %msg%\n" $WorkDirectory /var/spool/rsyslog $ActionQueueType LinkedList $ActionQueueFileName rsyslog001-buffer $ActionQueueMaxFileSize 10m $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.*; :omrelp:rsyslogserver1:20514;fmtFQDN $ActionQueueType LinkedList $ActionQueueFileName rsyslog002-buffer $ActionQueueMaxFileSize 10m $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on *.*; :omrelp:rsyslogserver2:20514;fmtFQDN Does anybody know what the problem could be and how to resolve it? -- Regards, Jacek ________________________________________________________________________ In order to protect our email recipients, Betfair Group use SkyScan from MessageLabs to scan all Incoming and Outgoing mail for viruses. ________________________________________________________________________ From openlinuxsource at gmail.com Mon Feb 28 16:14:19 2011 From: openlinuxsource at gmail.com (Lee Eric) Date: Mon, 28 Feb 2011 23:14:19 +0800 Subject: [rsyslog] Help: Separate Logs Order Message-ID: Hi, I have a question about separate logs order configuration. I have 4 machines with hostnames server, storage, client1 and client2. The rsyslogs is running at server. The configuration looks like that: $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) $ModLoad imtcp.so $InputTCPServerRun 514 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $template DynFile,"/var/log/system-%HOSTNAME%.log" :source , !isequal , "localhost" ?DynFile & ~ *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot.log And in other machines the configuration looks like this: $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) $ModLoad imtcp.so $InputTCPServerRun 514 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat *.* @@server.herdingcat.internal *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot.log /var/log/system-storage.log, /var/log/system-client1.log, /var/log/system-client2.log created successfully and it works well but there's also a log file /var/log/system-server.log created. I hope server's log is still recorded under /var/log/messages not separated. So I'm wondering is there any good method to achieve that function? Thanks very much. Regards, Eric From david at lang.hm Mon Feb 28 22:38:45 2011 From: david at lang.hm (david at lang.hm) Date: Mon, 28 Feb 2011 13:38:45 -0800 (PST) Subject: [rsyslog] Help: Separate Logs Order In-Reply-To: References: Message-ID: On Mon, 28 Feb 2011, Lee Eric wrote: > Hi, > > I have a question about separate logs order configuration. I have 4 > machines with hostnames server, storage, client1 and client2. The > rsyslogs is running at server. The configuration looks like that: > > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) > $ModLoad imtcp.so > $InputTCPServerRun 514 > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > $template DynFile,"/var/log/system-%HOSTNAME%.log" > :source , !isequal , "localhost" ?DynFile > & ~ > *.info;mail.none;authpriv.none;cron.none /var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* /var/log/cron > *.emerg * > uucp,news.crit /var/log/spooler > local7.* /var/log/boot.log > > And in other machines the configuration looks like this: > > > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) > $ModLoad imtcp.so > $InputTCPServerRun 514 > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > *.* @@server.herdingcat.internal > *.info;mail.none;authpriv.none;cron.none /var/log/messages > authpriv.* /var/log/secure > mail.* -/var/log/maillog > cron.* /var/log/cron > *.emerg * > uucp,news.crit /var/log/spooler > local7.* /var/log/boot.log > > /var/log/system-storage.log, /var/log/system-client1.log, > /var/log/system-client2.log created successfully and it works well but > there's also a log file /var/log/system-server.log created. I hope > server's log is still recorded under /var/log/messages not separated. > So I'm wondering is there any good method to achieve that function? I don't really understand your question here. the second configuration doesn't seem to have any dynafile lines, so it sounds like you are asking about the first one. In the first one you are saying that for any server other than localhost, put the logs in a file /var/log/system-name.log and then throw them away (don't put them in any other logfile) so if you get a log entry from a machine called 'server' it would create the fiel /var/log/system-server.log and then throw away the log. could you please try restating your what you think is being done differently from what you are wanting? David Lang4