From david at lang.hm Wed Oct 1 02:13:54 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:13:54 -0700 (PDT) Subject: [rsyslog] git url? Message-ID: what is the url for the rsyslog git repository? David Lang From mbiebl at gmail.com Wed Oct 1 02:19:05 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 1 Oct 2008 02:19:05 +0200 Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: 2008/10/1 : > what is the url for the rsyslog git repository? gitweb: http://git.adiscon.com/?p=rsyslog.git;a=summary clonable urls: git://git.adiscon.com/git/rsyslog.git http://git.adiscon.com/git/rsyslog.git -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 1 02:38:46 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:38:46 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Wed, 1 Oct 2008, Michael Biebl wrote: > 2008/10/1 : >> what is the url for the rsyslog git repository? > > gitweb: > http://git.adiscon.com/?p=rsyslog.git;a=summary > > clonable urls: > git://git.adiscon.com/git/rsyslog.git > http://git.adiscon.com/git/rsyslog.git thanks, much easier then guessing. now that I have his main branch cloned, what do I need to do to have it fetch all the other branches? David Lang From david at lang.hm Wed Oct 1 02:54:01 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:54:01 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Tue, 30 Sep 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Michael Biebl wrote: > >> 2008/10/1 : >>> what is the url for the rsyslog git repository? >> >> gitweb: >> http://git.adiscon.com/?p=rsyslog.git;a=summary >> >> clonable urls: >> git://git.adiscon.com/git/rsyslog.git >> http://git.adiscon.com/git/rsyslog.git > > thanks, much easier then guessing. > > now that I have his main branch cloned, what do I need to do to have it > fetch all the other branches? found it, git branch -r David Lang From luigi.perroti at googlemail.com Wed Oct 1 10:57:41 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 10:57:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated Message-ID: Hi, I'd like to ask if it's normal that %timegenerated:::date-subseconds% returns a value lower than %timereported:::date-subseconds%. Intuitively I'd say it is since this makes sense by taking in consideration the property names. Still, since the documentation says that timegenerated is the timestamp when the message was received I would have expected a value higher than the one returned by timereported. I'm afraid this is a dumb question but this behaviour got me confused, if anyone could shed some light on this I'd be grateful. Thanks. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 11:24:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 11:24:08 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> It is just returning the subsecond *part* of the timestamp. So while 9:13:14.123 is "higher" than 8:13:14.567, the subsecond part of the later (567) is higher than the subsecond part of the earlier (123). It is important that you are extracting a partial timestamp, which also implies that this is no longer monotonically increasing. Actually, it no longer has any sense of "timing" at all. This is NOT the timestamp converted to nanoseconds (or whatever). I hope this clarifies, if not, keep posting ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 10:58 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Subseconds in timereported and timegenerated > > Hi, > > I'd like to ask if it's normal that %timegenerated:::date-subseconds% > returns a value lower than %timereported:::date-subseconds%. > > Intuitively I'd say it is since this makes sense by taking in > consideration the property names. > Still, since the documentation says that timegenerated is the > timestamp when the message was received I would have expected a value > higher than the one returned by timereported. > > I'm afraid this is a dumb question but this behaviour got me confused, > if anyone could shed some light on this I'd be grateful. Thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 12:00:37 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:00:37 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: Thanks for the thorough explanation but I'm afraid that I still don't understand why I'm seeing this: http://imagebin.ca/view/2EknmP0.html This is how the columns are populated: sent <- %timereported:::date-mysql% sent_subseconds <- %timereported:::date-subseconds% received_by_rsyslog <- %timegenerated:::date-mysql% received_subseconds <- %timegenerated:::date-subseconds% Most probably I didn't get the difference between timereported and timegenerated. Meaning that I should just switch the sent* and received* columns and that nothing is wrong with Rsyslog. I'd be great if you could could confirm this scenario, thanks for your time! Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 12:03:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:03:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Just a quick note while working on a bigger thing: Is the sent timestamp form an instance that received data via the network (data that contained a high-precision timestamp). Also, which version are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks for the thorough explanation but I'm afraid that I still don't > understand why I'm seeing this: > > http://imagebin.ca/view/2EknmP0.html > > This is how the columns are populated: > sent <- %timereported:::date-mysql% > sent_subseconds <- %timereported:::date-subseconds% > received_by_rsyslog <- %timegenerated:::date-mysql% > received_subseconds <- %timegenerated:::date-subseconds% > > Most probably I didn't get the difference between timereported and > timegenerated. > Meaning that I should just switch the sent* and received* columns and > that nothing is wrong with Rsyslog. > > I'd be great if you could could confirm this scenario, thanks for your > time! > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 12:00:47 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 12:00:47 +0200 Subject: [rsyslog] Development of failsafe disk based queue Message-ID: <48E34A4F.308@ecker-software.de> Hi, I am looking for a failsafe solution to store syslog messages localy until they could be send later. I already looked at the disk based memory queue and the disk based queue. Both queue's don't work if you just power down the system immediatly actually loosing the whole queue. I already looked at queue.c and it seemed to me that both queues were not designed for that kind of failure, but I could be wrong there. Since an immediate power down of the system is the major failure which will occure pretty often I need to create a soltution there. Did you already start to develop something addressing that problem? Could you help me extend rsyslog (3.18.4) so that I can develop a new queue myself? I would contribute the code to the rsyslog project if you would like afterwards. bye David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:22:19 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:22:19 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: Here's a better screenshot: http://imagebin.ca/view/8p8hSnT.html Everything is happening on the same machine. I have these modules loaded: imfile, imuxsock, imklog, ommysql. The php-fpm entries are read from a log file by imfile. Right now I'm trying out 3.21.5 but I think I had already seen this behaviour on 3.21.3 but I'm not too sure about the version number. Luigi On 10/1/08, Rainer Gerhards wrote: > Just a quick note while working on a bigger thing: Is the sent timestamp > form an instance that received data via the network (data that contained > a high-precision timestamp). Also, which version are you using? > > Rainer > > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti >> Sent: Wednesday, October 01, 2008 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated >> >> Thanks for the thorough explanation but I'm afraid that I still don't >> understand why I'm seeing this: >> >> http://imagebin.ca/view/2EknmP0.html >> >> This is how the columns are populated: >> sent <- %timereported:::date-mysql% >> sent_subseconds <- %timereported:::date-subseconds% >> received_by_rsyslog <- %timegenerated:::date-mysql% >> received_subseconds <- %timegenerated:::date-subseconds% >> >> Most probably I didn't get the difference between timereported and >> timegenerated. >> Meaning that I should just switch the sent* and received* columns and >> that nothing is wrong with Rsyslog. >> >> I'd be great if you could could confirm this scenario, thanks for your >> time! >> >> Regards, >> Luigi >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 12:28:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:28:09 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Please post your config, that makes it easier for me ;). I think I know where this stems back from - we had two time calls inside rsyslog done, which has been removed in the experimental "perf" git branch. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Here's a better screenshot: > http://imagebin.ca/view/8p8hSnT.html > > Everything is happening on the same machine. > I have these modules loaded: imfile, imuxsock, imklog, ommysql. > The php-fpm entries are read from a log file by imfile. > > Right now I'm trying out 3.21.5 but I think I had already seen this > behaviour on 3.21.3 but I'm not too sure about the version number. > > Luigi > > On 10/1/08, Rainer Gerhards wrote: > > Just a quick note while working on a bigger thing: Is the sent > timestamp > > form an instance that received data via the network (data that > contained > > a high-precision timestamp). Also, which version are you using? > > > > Rainer > > > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > >> Sent: Wednesday, October 01, 2008 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > >> > >> Thanks for the thorough explanation but I'm afraid that I still > don't > >> understand why I'm seeing this: > >> > >> http://imagebin.ca/view/2EknmP0.html > >> > >> This is how the columns are populated: > >> sent <- %timereported:::date-mysql% > >> sent_subseconds <- %timereported:::date-subseconds% > >> received_by_rsyslog <- %timegenerated:::date-mysql% > >> received_subseconds <- %timegenerated:::date-subseconds% > >> > >> Most probably I didn't get the difference between timereported and > >> timegenerated. > >> Meaning that I should just switch the sent* and received* columns > and > >> that nothing is wrong with Rsyslog. > >> > >> I'd be great if you could could confirm this scenario, thanks for > your > >> time! > >> > >> Regards, > >> Luigi > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:32:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:32:26 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. > Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. I doubt there is a software soution against this (one that does not depend on a transactional file system, of course). What prevents you from using a UPS? I'd say that a sudden power-loss is by far the least probable error cause for a system that is configured to do any serious work. Please elaborate why you (or others ;)) consider this case important. > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:37:04 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:37:04 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: Here it is: $ModLoad imfile $ModLoad imuxsock $ModLoad imklog $ModLoad ommysql $ActionOmmysqlServerPort 1234 $MaxMessageSize 8k $EscapeControlCharactersOnReceive off $FileOwner root $FileGroup root $FileCreateMode 0600 $InputFileName /path/to/log/php-fpm.log $InputFileTag php-fpm $InputFileStateFile php-fpm.state $InputFileSeverity notice $InputFileFacility local5 $InputRunFileMonitor $MainMsgQueueType LinkedList $MainMsgQueueSize 20000 $MainMsgQueueFileName main-spool $MainMsgQueueMaxDiskSpace 200M $MainMsgQueueWorkerThreadMinimumMessages 2500 $MainMsgQueueWorkerThreads 1 $MainMsgQueueWorkerTimeoutThreadShutdown 10000 $MainMsgQueueSaveOnShutdown on $template my-rsyslog-schema,"insert into messages(hostname, sent, sent_subseconds, received_by_rsyslog, received_subseconds, process, facility, severity, message) values ('%hostname%', '%timereported:::date-mysql%', %timereported:::date-subseconds%, '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', '%msg%')",sql *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema From rgerhards at hq.adiscon.com Wed Oct 1 12:46:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:46:42 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Thanks, this is useful and helps understanding what is going on. I think what you see is an artifact of the two time() calls I mentioned. To verify, I'd appreciate if you could try out the latest from the perf branch, which can be found here: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=1531c541b294b6c029e5de9a01d4d8499f7e4718;sf=tgz Note that you need to setup the build environment (quite easy, but you need to know). Details at: http://www.rsyslog.com/doc-build_from_repo.html These two timestamps, for internally generated sources, should be the same in that version. Note, however, that, for externally generated messages, timereceived is whatever is recorded inside the message. There are various reasons for this not being correct, clock skew being a highly probably cause (which is also often found). HTH Rainer On Wed, 2008-10-01 at 12:37 +0200, Luigi Perroti wrote: > Here it is: > > $ModLoad imfile > $ModLoad imuxsock > $ModLoad imklog > $ModLoad ommysql > $ActionOmmysqlServerPort 1234 > > $MaxMessageSize 8k > > $EscapeControlCharactersOnReceive off > > $FileOwner root > $FileGroup root > $FileCreateMode 0600 > > $InputFileName /path/to/log/php-fpm.log > $InputFileTag php-fpm > $InputFileStateFile php-fpm.state > $InputFileSeverity notice > $InputFileFacility local5 > $InputRunFileMonitor > > $MainMsgQueueType LinkedList > $MainMsgQueueSize 20000 > $MainMsgQueueFileName main-spool > $MainMsgQueueMaxDiskSpace 200M > $MainMsgQueueWorkerThreadMinimumMessages 2500 > $MainMsgQueueWorkerThreads 1 > $MainMsgQueueWorkerTimeoutThreadShutdown 10000 > $MainMsgQueueSaveOnShutdown on > > $template my-rsyslog-schema,"insert into messages(hostname, sent, > sent_subseconds, received_by_rsyslog, received_subseconds, process, > facility, severity, message) values ('%hostname%', > '%timereported:::date-mysql%', %timereported:::date-subseconds%, > '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, > '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', > '%msg%')",sql > > *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:50:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:50:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> One thing I forgot to mention: a pure disk queue (not a disk-assisted one) gets you as close to your goal as possible (well, mostly - we could, at a considerable performance expense, require synced writing). With that case, all data is immediately stored on disk. You can configure it to also write the meta data out immediately (and again with sync, not yet supported). However, you still have a window of exposure, for example if the power loss happens right in the middle of when the disk actually writes data to the disk sector. I still wonder why this scenario would be useful to address... Rainer On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. > > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 13:02:50 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:02:50 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> Message-ID: <48E358DA.5080104@ecker-software.de> Rainer Gerhards schrieb: >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole >> > queue. > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. >> Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> > > I doubt there is a software soution against this (one that does not > depend on a transactional file system, of course). What prevents you > from using a UPS? I'd say that a sudden power-loss is by far the least > probable error cause for a system that is configured to do any serious > work. > > Please elaborate why you (or others ;)) consider this case important. > > The client systems (about 200 of them planned) are stationed in public places around the world connected to centralized servers through vpn connections over an unreliable network connection. Since space and look requirement is important a UPS won't fit there. There is actually no space for an UPS. The main problem is that customers are actually pulling the plug to restart the system, to charge their laptops or mobile phones or just for the fun of it. The client base image is a read-only system (Knoppix Like) with an extra hard disk for swap and other informations like syslog messages. Since there are no administrators close to the client system the client itself needs to have the capability to send all the missing log information between a network failure and an immediate power down to the central server for error analysis since those are usualy the most important once to pinpoint the cause of the inital error. My approach would be to use a block device directly since a file system if fault-prone if you shut down the system immediatly. Each entry including the header information guarded by a checksum value. It would be actually something like a fixed array based queue just that it would store the information in a block device. But this is just an inital thought. >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if >> > you > >> would like afterwards. >> >> bye >> David Ecker >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 13:15:22 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:15:22 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E35BCA.7030808@ecker-software.de> Rainer Gerhards schrieb: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > If that would work it would be perfect. For testing I could actually pass the correct fctl flag inside of queue.c hardcoded if that is all that is needed. > I still wonder why this scenario would be useful to address... > > > David > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 13:48:32 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:48:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E35BCA.7030808@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <48E35BCA.7030808@ecker-software.de> Message-ID: <1222861712.2682.450.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 13:15 +0200, David Ecker wrote: > Rainer Gerhards schrieb: > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > > If that would work it would be perfect. For testing I could actually > pass the correct fctl flag inside of queue.c hardcoded if that is all > that is needed. that works ;) You just need to fine-tune the queue params. If you find problems with that, I am more than happy to help. > > I still wonder why this scenario would be useful to address... very interesting scenario. Never thought about such one :) If you need to enhance the queue (for block devices), I can provide a some hints. But, given the other priorities which are required by a much broader user base, involving me more than a hint here or there would probably require a consulting contract. Sorry for being so bluntly, but I think it is always good to set the right expectation level. Rainer From david at lang.hm Wed Oct 1 13:55:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 04:55:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. are you sure about the disk based queue? per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue can be set to do a commit of the metadata after each message. Disk Queues Disk queues use disk drives for buffering. The important fact is that the always use the disk and do not buffer anything in memory. Thus, the queue is ultra-reliable, but by far the slowest mode. For regular use cases, this queue mode is not recommended. It is useful if log data is so important that it must not be lost, even in extreme cases. When a disk queue is written, it is done in chunks. Each chunk receives its individual file. Files are named with a prefix (set via the "$QueueFilename" config directive) and followed by a 7-digit number (starting at one and incremented for each file). Chunks are 10mb by default, a different size can be set via the"$QueueMaxFileSize" config directive. Note that the size limit is not a sharp one: rsyslog always writes one complete queue entry, even if it violates the size limit. So chunks are actually a little but (usually less than 1k) larger then the configured size. Each chunk also has a different size for the same reason. If you observe different chunk sizes, you can relax: this is not a problem. Writing in chunks is used so that processed data can quickly be deleted and is free for other uses - while at the same time keeping no artificial upper limit on disk space used. If a disk quota is set (instructions further below), be sure that the quota/chunk size allows at least two chunks to be written. Rsyslog currently does not check that and will fail miserably if a single chunk is over the quota. Creating new chunks costs performance but provides quicker ability to free disk space. The 10mb default is considered a good compromise between these two. However, it may make sense to adapt these settings to local policies. For example, if a disk queue is written on a dedicated 200gb disk, it may make sense to use a 2gb (or even larger) chunk size. Please note, however, that the disk queue by default does not update its housekeeping structures every time it writes to disk. This is for performance reasons. In the event of failure, data will still be lost (except when manually is mangled with the file structures). However, disk queues can be set to write bookkeeping information on checkpoints (every n records), so that this can be made ultra-reliable, too. If the checkpoint interval is set to one, no data can be lost, but the queue is exceptionally slow. Each queue can be placed on a different disk for best performance and/or isolation. This is currently selected by specifying different $WorkDirectory config directives before the queue creation statement. To create a disk queue, use the "$QueueType Disk" config directive. Checkpoint intervals can be specified via "$QueueCheckpointInterval", with 0 meaning no checkpoints. you also need to specificly enable syncing (from http://www.rsyslog.com/doc-v3compatibility.html ) Output File Syncing Rsyslogd tries to keep as compatible to stock syslogd as possible. As such, it retained stock syslogd's default of syncing every file write if not specified otherwise (by placing a dash in front of the output file name). While this was a useful feature in past days where hardware was much less reliable and UPS seldom, this no longer is useful in today's worl. Instead, the syncing is a high performace hit. With it, rsyslogd writes files around 50 *times* slower than without it. It also affects overall system performance due to the high IO activity. In rsyslog v3, syncing has been turned off by default. This is done via a specific configuration directive "$ActionFileEnableSync on/off" which is off by default. So even if rsyslogd finds sync selector lines, it ignores them by default. In order to enable file syncing, the administrator must specify "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that syncing only happens in some installations where the administrator actually wanted that (performance-intense) feature. In the fast majority of cases (if not all), this dramatically increases rsyslogd performance without any negative effects. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. with checkpoint interval set to 1 and syncing enabled the data should be in on the disk safely (assuming you have hardware that supports this) and a power-off won't affect it. David Lang > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 13:57:55 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:57:55 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> David, the file syncing mentioned in the compatibility doc applies to the output action, only. The queue does never do synchronous writes - I always assumed that a critical system would have a UPS and could never think (so far) about a valid reason for not having it. So the queue would need to have an extra option to do sync writes. Obviously, that's not a big deal. Performance, of course, will be extremely terrible with such a setup... Rainer On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Hi, > > > > I am looking for a failsafe solution to store syslog messages localy > > until they could be send later. I already looked at the disk based > > memory queue and the disk based queue. Both queue's don't work if you > > just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that the > always use the disk and do not buffer anything in memory. Thus, the queue > is ultra-reliable, but by far the slowest mode. For regular use cases, > this queue mode is not recommended. It is useful if log data is so > important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk receives > its individual file. Files are named with a prefix (set via the > "$QueueFilename" config directive) and followed by a 7-digit > number (starting at one and incremented for each file). Chunks are 10mb by > default, a different size can be set via the"$QueueMaxFileSize" > config directive. Note that the size limit is not a sharp one: rsyslog > always writes one complete queue entry, even if it violates the size > limit. So chunks are actually a little but (usually less than 1k) larger > then the configured size. Each chunk also has a different size for the > same reason. If you observe different chunk sizes, you can relax: this is > not a problem. > > Writing in chunks is used so that processed data can quickly be deleted > and is free for other uses - while at the same time keeping no artificial > upper limit on disk space used. If a disk quota is set (instructions > further below), be sure that the quota/chunk size allows at least two > chunks to be written. Rsyslog currently does not check that and will fail > miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to free > disk space. The 10mb default is considered a good compromise between these > two. However, it may make sense to adapt these settings to local policies. > For example, if a disk queue is written on a dedicated 200gb disk, it may > make sense to use a 2gb (or even larger) chunk size. > > Please note, however, that the disk queue by default does not update its > housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, disk > queues can be set to write bookkeeping information on checkpoints (every n > records), so that this can be made ultra-reliable, too. If the checkpoint > interval is set to one, no data can be lost, but the queue is > exceptionally slow. > > Each queue can be placed on a different disk for best performance and/or > isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write if > not specified otherwise (by placing a dash in front of the output file > name). While this was a useful feature in past days where hardware was > much less reliable and UPS seldom, this no longer is useful in today's > worl. Instead, the syncing is a high performace hit. With it, rsyslogd > writes files around 50 *times* slower than without it. It also affects > overall system performance due to the high IO activity. In rsyslog v3, > syncing has been turned off by default. This is done via a specific > configuration directive "$ActionFileEnableSync on/off" which is off by > default. So even if rsyslogd finds sync selector lines, it ignores them by > default. In order to enable file syncing, the administrator must specify > "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > syncing only happens in some installations where the administrator > actually wanted that (performance-intense) feature. In the fast majority > of cases (if not all), this dramatically increases rsyslogd performance > without any negative effects. > > > > > I already looked at queue.c and it seemed to me that both queues were > > not designed for that kind of failure, but I could be wrong there. Since > > an immediate power down of the system is the major failure which will > > occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should be > in on the disk safely (assuming you have hardware that supports this) and > a power-off won't affect it. > > David Lang > > > > > Did you already start to develop something addressing that problem? > > Could you help me extend rsyslog (3.18.4) so that I can develop a new > > queue myself? I would contribute the code to the rsyslog project if you > > would like afterwards. > > > > bye > > David Ecker > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:02:37 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:02:37 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E358DA.5080104@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> <48E358DA.5080104@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Rainer Gerhards schrieb: >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole >>> >> queue. >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. >>> Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> >> I doubt there is a software soution against this (one that does not >> depend on a transactional file system, of course). What prevents you >> from using a UPS? I'd say that a sudden power-loss is by far the least >> probable error cause for a system that is configured to do any serious >> work. >> >> Please elaborate why you (or others ;)) consider this case important. >> >> > The client systems (about 200 of them planned) are stationed in public > places around the world connected to centralized servers through vpn > connections over an unreliable network connection. Since space and look > requirement is important a UPS won't fit there. There is actually no > space for an UPS. The main problem is that customers are actually > pulling the plug to restart the system, to charge their laptops or > mobile phones or just for the fun of it. you can get UPS systems that are PCI cards, completly internal. they still may not fit, but you at least have a chance. > The client base image is a read-only system (Knoppix Like) with an extra > hard disk for swap and other informations like syslog messages. Since > there are no administrators close to the client system the client itself > needs to have the capability to send all the missing log information > between a network failure and an immediate power down to the central > server for error analysis since those are usualy the most important once > to pinpoint the cause of the inital error. > > My approach would be to use a block device directly since a file system > if fault-prone if you shut down the system immediatly. Each entry > including the header information guarded by a checksum value. It would > be actually something like a fixed array based queue just that it would > store the information in a block device. But this is just an inital thought. you are inventing a new filesystem here. it's not that easy to be reliable becouse the disk can lie to you. unless you are doing interesting things at the ATA/SCSI command level the disk may re-order your writes and may cache them in memory on the drive for an unknown time before actually writing them if you need reliable writes at anything close to a reasonable speed you need to have a battery backed cache or solid state drive in your machine (and the solid state drives are not all fast to write to) David Lang >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if >>> >> you >> >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 14:05:42 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:05:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E36796.9060509@ecker-software.de> Hi, should I use 3.18.4 (latest stable, I would preferr that one) or do I need the latest development version? I would actually alter queue.c directly changing the fctl flags in the disk based queue (O_DIRECT,O_SYNC,.O_NOATIME). Performance is not really an issue. There will be only 1000 to 2000 Messages per hour in peak times bye David Ecker Rainer Gerhards schrieb: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. > > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. > > Performance, of course, will be extremely terrible with such a setup... > > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >>> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:07:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:07:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > I still wonder why this scenario would be useful to address... not all uses of rsyslog are for simple system logs. it's a good general purpose log tool, and there are some cases where you want to be as sure as you possibly can be that once a message has been acknowledged it has no chance of being lost. useing some form of solid-state reliable storage (battery backed ram on a raid controller, a battery backed ram disk, a flash disk) it is possible (but not nessasarily cheap) to get the ability to do tens to hundreds of thousands of writes + syncs per second David Lang > Rainer > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:07:32 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:07:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <48E36804.3030107@ecker-software.de> Hi David, the actuall problem was that the qi file was not correct after an immediate restart. All messages were actually correctly written except the qi file. bye David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based > queue can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that > the always use the disk and do not buffer anything in memory. Thus, > the queue is ultra-reliable, but by far the slowest mode. For regular > use cases, this queue mode is not recommended. It is useful if log > data is so important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk > receives its individual file. Files are named with a prefix (set via > the "$QueueFilename" config directive) and followed by a > 7-digit number (starting at one and incremented for each file). Chunks > are 10mb by default, a different size can be set via > the"$QueueMaxFileSize" config directive. Note that the size > limit is not a sharp one: rsyslog always writes one complete queue > entry, even if it violates the size limit. So chunks are actually a > little but (usually less than 1k) larger then the configured size. > Each chunk also has a different size for the same reason. If you > observe different chunk sizes, you can relax: this is not a problem. > > Writing in chunks is used so that processed data can quickly be > deleted and is free for other uses - while at the same time keeping no > artificial upper limit on disk space used. If a disk quota is set > (instructions further below), be sure that the quota/chunk size allows > at least two chunks to be written. Rsyslog currently does not check > that and will fail miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to > free disk space. The 10mb default is considered a good compromise > between these two. However, it may make sense to adapt these settings > to local policies. For example, if a disk queue is written on a > dedicated 200gb disk, it may make sense to use a 2gb (or even larger) > chunk size. > > Please note, however, that the disk queue by default does not update > its housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, > disk queues can be set to write bookkeeping information on checkpoints > (every n records), so that this can be made ultra-reliable, too. If > the checkpoint interval is set to one, no data can be lost, but the > queue is exceptionally slow. > > Each queue can be placed on a different disk for best performance > and/or isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write > if not specified otherwise (by placing a dash in front of the output > file name). While this was a useful feature in past days where > hardware was much less reliable and UPS seldom, this no longer is > useful in today's worl. Instead, the syncing is a high performace hit. > With it, rsyslogd writes files around 50 *times* slower than without > it. It also affects overall system performance due to the high IO > activity. In rsyslog v3, syncing has been turned off by default. This > is done via a specific configuration directive "$ActionFileEnableSync > on/off" which is off by default. So even if rsyslogd finds sync > selector lines, it ignores them by default. In order to enable file > syncing, the administrator must specify "$ActionFileEnableSync on" at > the top of rsyslog.conf. This ensures that syncing only happens in > some installations where the administrator actually wanted that > (performance-intense) feature. In the fast majority of cases (if not > all), this dramatically increases rsyslogd performance without any > negative effects. > > > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should > be in on the disk safely (assuming you have hardware that supports > this) and a power-off won't affect it. > > David Lang > > > >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:09:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:09:33 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E36796.9060509@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <48E36796.9060509@ecker-software.de> Message-ID: <1222862973.2682.456.camel@rgf9dev.intern.adiscon.com> I am currently working on the queue engine in the devel branch. In that light, it could make sense to start with that branch. Also, I won't make any non-bug related changes in the stable version, so if you run into anything I can quickly change, that will happen in devel. Rainer On Wed, 2008-10-01 at 14:05 +0200, David Ecker wrote: > Hi, > > should I use 3.18.4 (latest stable, I would preferr that one) or do I > need the latest development version? I would actually alter queue.c > directly changing the fctl flags in the disk based queue > (O_DIRECT,O_SYNC,.O_NOATIME). > > Performance is not really an issue. There will be only 1000 to 2000 > Messages per hour in peak times > > bye > David Ecker > > Rainer Gerhards schrieb: > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > > > Performance, of course, will be extremely terrible with such a setup... > > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >>> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >>> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 14:09:53 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 14:09:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: I installed the snapshot but apparently the issue persists. If there's something else that I can do, now or when you have more time, let me know. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 14:12:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:12:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > I still wonder why this scenario would be useful to address... > > not all uses of rsyslog are for simple system logs. it's a good general > purpose log tool, and there are some cases where you want to be as sure as > you possibly can be that once a message has been acknowledged it has no > chance of being lost. I designed the engine for audit-class reliability. However, I assumed that the rest of the system is also playing in that class. Doing everything with a potential power failure in mind creates a lot of extra demands. And I have never heard of anybody doing serious datacenter work without a proper UPS. Is this *really* an issue? Rainer > > useing some form of solid-state reliable storage (battery backed ram on a > raid controller, a battery backed ram disk, a flash disk) it is possible > (but not nessasarily cheap) to get the ability to do tens to hundreds of > thousands of writes + syncs per second > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, > >> > >> I am looking for a failsafe solution to store syslog messages localy > >> until they could be send later. I already looked at the disk based > >> memory queue and the disk based queue. Both queue's don't work if you > >> just power down the system immediatly actually loosing the whole queue. > >> I already looked at queue.c and it seemed to me that both queues were > >> not designed for that kind of failure, but I could be wrong there. Since > >> an immediate power down of the system is the major failure which will > >> occure pretty often I need to create a soltution there. > >> > >> Did you already start to develop something addressing that problem? > >> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >> queue myself? I would contribute the code to the rsyslog project if you > >> would like afterwards. > >> > >> bye > >> David Ecker > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 14:15:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:15:14 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863314.2682.463.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:09 +0200, Luigi Perroti wrote: > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. That's interesting. Could you please get me a debug log (you can mail it privately). Here is info about how to do that: http://www.rsyslog.com/doc-troubleshoot.html Rainer From david at lang.hm Wed Oct 1 14:25:18 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:25:18 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. ouch. > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. good > Performance, of course, will be extremely terrible with such a setup... only if you have to wait for a spinning disk to do the write. this is the same problem that databases have. they need to guarentee that once the database tells the writing program that the data is written it will be there even if the system looses power immediatly. if you run a database on standard desktop hardware (and it doesn't have this safety disabled) you cannot do more then about 80 writes/second. If you upgrade to the super speedy 15K rpm drives you can do ~160 writes/second. given that you need to write the data + metadata it gets even uglier, so what the databases do (and some journaling filesystems) is to write a log that says what they are going to do, sync that, and then later write the data to the actual files (updating the journal when they complete the write) it sounds like you order your write correctly for a disk-based queue, but you would need the option of issuing the syncs (probably when you do the checkpoints) if you do this on the wrong hardware (say a laptop 5200 rpm drive or the wrong flash drive), the fact that you need to do four writes per log entry (data to queue, metadata to queue, data to output, update metadata for queue) could drop you to below 15 logs/sec (60/4 but then you loose time to seeking as well) however, with the correct drive to write to (say a $2,400 80G fusion-io flash card that can do ~100k IO ops/sec) you should be able to sustain 20,000 logs/sec. realisticly very few people need the sustained write capacity that you would get from such a setup. but if you go with a $500-$700 raid card with a battery-backed cache you get very similar performance, but with some possibility that you can't sustain it forever. David Lang > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, David Ecker wrote: >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:29:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:29:03 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > ouch. > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > good > > > Performance, of course, will be extremely terrible with such a setup... > > only if you have to wait for a spinning disk to do the write. I agree to the rest of your argument below. But the question raised here was in regard to a system without any battery backup. So I would need to wait. Even then, in the worst case, I think it would be possible that the disk does only a partial write. I am not sure if that's really the case with today's disk drives (which I think have capacitors to prevent this scenario), but with past drives this could happen (I know all too well - a few years ago that cost me a weekend ;)). Rainer > > this is the same problem that databases have. they need to guarentee that > once the database tells the writing program that the data is written it > will be there even if the system looses power immediatly. > > if you run a database on standard desktop hardware (and it doesn't have > this safety disabled) you cannot do more then about 80 writes/second. If > you upgrade to the super speedy 15K rpm drives you can do ~160 > writes/second. > > given that you need to write the data + metadata it gets even uglier, so > what the databases do (and some journaling filesystems) is to write a log > that says what they are going to do, sync that, and then later write the > data to the actual files (updating the journal when they complete the > write) > > it sounds like you order your write correctly for a disk-based queue, but > you would need the option of issuing the syncs (probably when you do the > checkpoints) > > if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > wrong flash drive), the fact that you need to do four writes per log entry > (data to queue, metadata to queue, data to output, update metadata for > queue) could drop you to below 15 logs/sec (60/4 but then you loose time > to seeking as well) > > however, with the correct drive to write to (say a $2,400 80G fusion-io > flash card that can do ~100k IO ops/sec) you should be able to sustain > 20,000 logs/sec. > > realisticly very few people need the sustained write capacity that you > would get from such a setup. but if you go with a $500-$700 raid card with > a battery-backed cache you get very similar performance, but with some > possibility that you can't sustain it forever. > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:35:17 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:35:17 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> One thing I forgot to mention: a pure disk queue (not a disk-assisted >>> one) gets you as close to your goal as possible (well, mostly - we >>> could, at a considerable performance expense, require synced writing). >>> With that case, all data is immediately stored on disk. You can >>> configure it to also write the meta data out immediately (and again with >>> sync, not yet supported). However, you still have a window of exposure, >>> for example if the power loss happens right in the middle of when the >>> disk actually writes data to the disk sector. >>> >>> I still wonder why this scenario would be useful to address... >> >> not all uses of rsyslog are for simple system logs. it's a good general >> purpose log tool, and there are some cases where you want to be as sure as >> you possibly can be that once a message has been acknowledged it has no >> chance of being lost. > > I designed the engine for audit-class reliability. However, I assumed > that the rest of the system is also playing in that class. Doing > everything with a potential power failure in mind creates a lot of extra > demands. And I have never heard of anybody doing serious datacenter work > without a proper UPS. Is this *really* an issue? Yes. UPSs fail. generators fail power cords come loose. power cords get unplugged by someone who thinks they are unplugging a different system people bump power switches on power strips. power supplies are defective I had one production outage where a visiting tech pulled a power cord from an overhead plug and dropped it on the ground, where it happened to hit the power switch on a power strip. I've had high-end systems with redundant power supplies go down becouse of faulty hardware that decided to disble both power supplies at once (it turned out that there was a defect in the whole batch of servers, but it took IBM several weeks to figure out what was going on) I've had UPS systems blow up (literally) I've had a datacenter go down becouse the it was running on generator power (due to other issues), and the refueling guy filled the tank incorrectly and got air bubbles into the fuel system, a few min later the 500Kw diesel generator couldn't maintain constant speed and the safety triggers kicked in and disabled it. it's amazing the things that happen in real-life David Lang > Rainer >> >> useing some form of solid-state reliable storage (battery backed ram on a >> raid controller, a battery backed ram disk, a flash disk) it is possible >> (but not nessasarily cheap) to get the ability to do tens to hundreds of >> thousands of writes + syncs per second >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >>>> Hi, >>>> >>>> I am looking for a failsafe solution to store syslog messages localy >>>> until they could be send later. I already looked at the disk based >>>> memory queue and the disk based queue. Both queue's don't work if you >>>> just power down the system immediatly actually loosing the whole queue. >>>> I already looked at queue.c and it seemed to me that both queues were >>>> not designed for that kind of failure, but I could be wrong there. Since >>>> an immediate power down of the system is the major failure which will >>>> occure pretty often I need to create a soltution there. >>>> >>>> Did you already start to develop something addressing that problem? >>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>> queue myself? I would contribute the code to the rsyslog project if you >>>> would like afterwards. >>>> >>>> bye >>>> David Ecker >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:39:54 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:39:54 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >> >> ouch. >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >> >> good >> >>> Performance, of course, will be extremely terrible with such a setup... >> >> only if you have to wait for a spinning disk to do the write. > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. no UPS is not nessasarily the same as no battey backup. you could use a compact flash drive and probably get better performance/reliability than spinning disks with no battery at all. > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). current disks do not have capacitors to prevent partial writes or to flush their caches. but options like the linux ext3 data-journaled make it so that you have your data in the journal safely, and the various solid-state options solve that problem. David Lang > Rainer > >> >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:45:06 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:45:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E370D2.7080901@ecker-software.de> Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >>> >> ouch. >> >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >>> >> good >> >> >>> Performance, of course, will be extremely terrible with such a setup... >>> >> only if you have to wait for a spinning disk to do the write. >> > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. > > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). > > Rainer > Hi, as long as you do sector based writes (512 byte per sector, usual) you can be sure that the write wasn"t partial.. Writing more than one sector or not starting at a correct offset (n*512,n=0,1,2,...x) might result in a partial write. I'll already tested that with my devel client here. So fencing each sector with a crc32 value would help detecting errors during a write operation. This is actually only a problem if you are writing directly to a block device like any filesystem does and yes, reordering is definitly a problem. So validating the content written to the disk afterwards is important. If writing through a filesystem reserving space in the destination file beforehand actually minimizes errors since the file system table doesn't have to be updated (you should also use the Flag O_NOATIME for that case). See for example VMWare ESX VMDK file handling. David > >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>> >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:55:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:55:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E370D2.7080901@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> Message-ID: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: [snip] > as long as you do sector based writes (512 byte per sector, usual) you > can be sure that the write wasn"t partial.. Writing more than one sector > or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > a partial write. I'll already tested that with my devel client here. So > fencing each sector with a crc32 value would help detecting errors > during a write operation. This is actually only a problem if you are > writing directly to a block device like any filesystem does and yes, > reordering is definitly a problem. So validating the content written to > the disk afterwards is important. > > If writing through a filesystem reserving space in the destination file > beforehand actually minimizes errors since the file system table doesn't > have to be updated (you should also use the Flag O_NOATIME for that > case). See for example VMWare ESX VMDK file handling. Well, first of all let me re-iterate that I do not intend to do a block device driver for rsyslog (but I definitely do not object getting one contributed ;)). Still thinking about the case and thinking about non-solid-state, non-internal-battery-backed-up disk, I can't see how you can be sure the data will be written. David just told me there are no capacitors. So if power fails, it fails rather quickly. So how can you be sure the disk will be able to finish writing that sector? Let's say the drive has begun to write the sector and been able to write the first 5 bytes. Now power fails. No capacitors, no battery-backup, so why should there be enough power to drive the disk write head for another 507 bytes? It the drives assures it can do that, it needs capacitors - doesn't it? Am I overlooking something obvious? Rainer From rgerhards at hq.adiscon.com Wed Oct 1 14:57:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:57:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> David, going back to the higher layer: do you say that immediate power failure is a case that you consider needed to be addressed in an enterprise logging system? Anybody else with an opinion? Rainer On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> > >>> David, > >>> > >>> the file syncing mentioned in the compatibility doc applies to the > >>> output action, only. > >> > >> ouch. > >> > >>> The queue does never do synchronous writes - I always assumed that a > >>> critical system would have a UPS and could never think (so far) about a > >>> valid reason for not having it. So the queue would need to have an extra > >>> option to do sync writes. Obviously, that's not a big deal. > >> > >> good > >> > >>> Performance, of course, will be extremely terrible with such a setup... > >> > >> only if you have to wait for a spinning disk to do the write. > > > > I agree to the rest of your argument below. But the question raised here > > was in regard to a system without any battery backup. So I would need to > > wait. > > no UPS is not nessasarily the same as no battey backup. > > you could use a compact flash drive and probably get better > performance/reliability than spinning disks with no battery at all. > > > Even then, in the worst case, I think it would be possible that the disk > > does only a partial write. I am not sure if that's really the case with > > today's disk drives (which I think have capacitors to prevent this > > scenario), but with past drives this could happen (I know all too well - > > a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. > > David Lang > > > Rainer > > > >> > >> this is the same problem that databases have. they need to guarentee that > >> once the database tells the writing program that the data is written it > >> will be there even if the system looses power immediatly. > >> > >> if you run a database on standard desktop hardware (and it doesn't have > >> this safety disabled) you cannot do more then about 80 writes/second. If > >> you upgrade to the super speedy 15K rpm drives you can do ~160 > >> writes/second. > >> > >> given that you need to write the data + metadata it gets even uglier, so > >> what the databases do (and some journaling filesystems) is to write a log > >> that says what they are going to do, sync that, and then later write the > >> data to the actual files (updating the journal when they complete the > >> write) > >> > >> it sounds like you order your write correctly for a disk-based queue, but > >> you would need the option of issuing the syncs (probably when you do the > >> checkpoints) > >> > >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > >> wrong flash drive), the fact that you need to do four writes per log entry > >> (data to queue, metadata to queue, data to output, update metadata for > >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time > >> to seeking as well) > >> > >> however, with the correct drive to write to (say a $2,400 80G fusion-io > >> flash card that can do ~100k IO ops/sec) you should be able to sustain > >> 20,000 logs/sec. > >> > >> realisticly very few people need the sustained write capacity that you > >> would get from such a setup. but if you go with a $500-$700 raid card with > >> a battery-backed cache you get very similar performance, but with some > >> possibility that you can't sustain it forever. > >> > >> David Lang > >> > >>> Rainer > >>> > >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >>>> On Wed, 1 Oct 2008, David Ecker wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I am looking for a failsafe solution to store syslog messages localy > >>>>> until they could be send later. I already looked at the disk based > >>>>> memory queue and the disk based queue. Both queue's don't work if you > >>>>> just power down the system immediatly actually loosing the whole queue. > >>>> > >>>> are you sure about the disk based queue? > >>>> > >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >>>> can be set to do a commit of the metadata after each message. > >>>> > >>>> Disk Queues > >>>> > >>>> Disk queues use disk drives for buffering. The important fact is that the > >>>> always use the disk and do not buffer anything in memory. Thus, the queue > >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, > >>>> this queue mode is not recommended. It is useful if log data is so > >>>> important that it must not be lost, even in extreme cases. > >>>> > >>>> When a disk queue is written, it is done in chunks. Each chunk receives > >>>> its individual file. Files are named with a prefix (set via the > >>>> "$QueueFilename" config directive) and followed by a 7-digit > >>>> number (starting at one and incremented for each file). Chunks are 10mb by > >>>> default, a different size can be set via the"$QueueMaxFileSize" > >>>> config directive. Note that the size limit is not a sharp one: rsyslog > >>>> always writes one complete queue entry, even if it violates the size > >>>> limit. So chunks are actually a little but (usually less than 1k) larger > >>>> then the configured size. Each chunk also has a different size for the > >>>> same reason. If you observe different chunk sizes, you can relax: this is > >>>> not a problem. > >>>> > >>>> Writing in chunks is used so that processed data can quickly be deleted > >>>> and is free for other uses - while at the same time keeping no artificial > >>>> upper limit on disk space used. If a disk quota is set (instructions > >>>> further below), be sure that the quota/chunk size allows at least two > >>>> chunks to be written. Rsyslog currently does not check that and will fail > >>>> miserably if a single chunk is over the quota. > >>>> > >>>> Creating new chunks costs performance but provides quicker ability to free > >>>> disk space. The 10mb default is considered a good compromise between these > >>>> two. However, it may make sense to adapt these settings to local policies. > >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may > >>>> make sense to use a 2gb (or even larger) chunk size. > >>>> > >>>> Please note, however, that the disk queue by default does not update its > >>>> housekeeping structures every time it writes to disk. This is for > >>>> performance reasons. In the event of failure, data will still be lost > >>>> (except when manually is mangled with the file structures). However, disk > >>>> queues can be set to write bookkeeping information on checkpoints (every n > >>>> records), so that this can be made ultra-reliable, too. If the checkpoint > >>>> interval is set to one, no data can be lost, but the queue is > >>>> exceptionally slow. > >>>> > >>>> Each queue can be placed on a different disk for best performance and/or > >>>> isolation. This is currently selected by specifying different > >>>> $WorkDirectory config directives before the queue creation statement. > >>>> > >>>> To create a disk queue, use the "$QueueType Disk" config > >>>> directive. Checkpoint intervals can be specified via > >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> you also need to specificly enable syncing (from > >>>> http://www.rsyslog.com/doc-v3compatibility.html ) > >>>> > >>>> Output File Syncing > >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >>>> such, it retained stock syslogd's default of syncing every file write if > >>>> not specified otherwise (by placing a dash in front of the output file > >>>> name). While this was a useful feature in past days where hardware was > >>>> much less reliable and UPS seldom, this no longer is useful in today's > >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >>>> writes files around 50 *times* slower than without it. It also affects > >>>> overall system performance due to the high IO activity. In rsyslog v3, > >>>> syncing has been turned off by default. This is done via a specific > >>>> configuration directive "$ActionFileEnableSync on/off" which is off by > >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by > >>>> default. In order to enable file syncing, the administrator must specify > >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >>>> syncing only happens in some installations where the administrator > >>>> actually wanted that (performance-intense) feature. In the fast majority > >>>> of cases (if not all), this dramatically increases rsyslogd performance > >>>> without any negative effects. > >>>> > >>>> > >>>> > >>>>> I already looked at queue.c and it seemed to me that both queues were > >>>>> not designed for that kind of failure, but I could be wrong there. Since > >>>>> an immediate power down of the system is the major failure which will > >>>>> occure pretty often I need to create a soltution there. > >>>> > >>>> with checkpoint interval set to 1 and syncing enabled the data should be > >>>> in on the disk safely (assuming you have hardware that supports this) and > >>>> a power-off won't affect it. > >>>> > >>>> David Lang > >>>> > >>>> > >>>> > >>>>> Did you already start to develop something addressing that problem? > >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>>>> queue myself? I would contribute the code to the rsyslog project if you > >>>>> would like afterwards. > >>>>> > >>>>> bye > >>>>> David Ecker > >>>>> > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 15:09:08 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:09:08 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37674.9090408@ecker-software.de> Hi, [quote] Depending on the media and the block device driver design, individual sector writes may not be atomic. A physical sector in typical devices is 512 bytes. In most cases, physical sector writes are atomic (either completely written, or not modified at all). A truly reliable file system, however, cannot count on this. [/quote] In most cases it works but some way of validating the data is needed if you want ultra reliability, which I don't need. If the last few messages a few seconds before an immediate shutdown are lost but all other messages are send correctly afterwards then that would be OK in my case. I'll just test version 2.21.5 with the altered open behauvior. The disk based queue-array developed by myself is just a fallback solution if the disk-based queue doesn't work with an immediate shutdown. David Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. >> > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:11:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:11:09 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Sorry, I overlooked this mail in the big bunch of messages. That's good reasoning. To cover these scenarios, we need to do everything with syncing. This also means that you can not use any of the disk-assisted modes, because in these modes we always try to keep things in memory in order to save writes. So while you have convinced me things can go wrong, I'd still say that is is very unusual (at least very costly) to care for all these things. But, of course, there are situations where it is needed. I'll probably see that I provide a facility to open files in "always sync" mode, but that for sure will not be the default setting ;) But even with the fast solid state disks (and similar methods) you mention, I think there will be a severe impact on performance because everything now needs to go through two write (data+metadata) and two read (again, data+metadata) OS call where we currently simply update an in-memory structure. Just out of curiosity: do you expect the majority of you rollouts to be using such methods? Rainer On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > > ... And I have never heard of anybody doing serious datacenter work > > without a proper UPS. Is this *really* an issue? > > Yes. > > UPSs fail. > generators fail > power cords come loose. > power cords get unplugged by someone who thinks they are unplugging a > different system > people bump power switches on power strips. > power supplies are defective > > I had one production outage where a visiting tech pulled a power cord from > an overhead plug and dropped it on the ground, where it happened to hit > the power switch on a power strip. > > I've had high-end systems with redundant power supplies go down becouse of > faulty hardware that decided to disble both power supplies at once (it > turned out that there was a defect in the whole batch of servers, but it > took IBM several weeks to figure out what was going on) > > I've had UPS systems blow up (literally) > > I've had a datacenter go down becouse the it was running on generator > power (due to other issues), and the refueling guy filled the tank > incorrectly and got air bubbles into the fuel system, a few min later the > 500Kw diesel generator couldn't maintain constant speed and the safety > triggers kicked in and disabled it. > > it's amazing the things that happen in real-life From david at lang.hm Wed Oct 1 15:17:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:17:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. I'm replying out of order as I see things anyway > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. I think you are saying that we must use the disk-only mode which is correct. > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. absolutly!! > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) thanks. > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. given the performance gains that we have seen by eliminating syscalls, it will hurt to add these back in, even with solid-state disks. that being said, it looks like the output module is nowhere close to being the limit (when I could get a good, stable reading on it, it looked like it was eating ~15% cpu compared to the input module at 100%) so it may not make much of a difference. > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? absolutly not. I have one case I am considering (the one I am talking to you about more efficiant database writes) that would be this paranoid, but the rest of it will be optimized for speed (battery-backed disk caches on the final server, but everything else can just use ram) David Lang > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:17:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37674.9090408@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <48E37674.9090408@ecker-software.de> Message-ID: <1222867062.2682.492.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:09 +0200, David Ecker wrote: > Hi, > > [quote] > Depending on the media and the block device driver design, individual > sector writes may > not be atomic. A physical sector in typical devices is 512 bytes. In > most cases, physical > sector writes are atomic (either completely written, or not modified at > all). A truly > reliable file system, however, cannot count on this. > [/quote] > > In most cases it works exactly - in most cases. That means it does not work always. > but some way of validating the data how can you validate if there is no power and the machine is off? > is needed if > you want ultra reliability, which I don't need. If the last few messages > a few seconds before an immediate shutdown are lost but all other > messages are send correctly afterwards then that would be OK in my case. but we can not guarantee that, at least not in all cases. Let's assume the disk died in the middle of the write access. Chances are good you'll never be able to read that sector again. Using a journaling file system will help, but without it, you may just have destroyed the sector that contained the .qi file. So on next startup the .qi is either not readable at all or not pointing at the correct information. The end result can be total loss of information. This scenario is probably acceptable in your case, because it is really, really highly unlikely. But it still exists. > I'll just test version 2.21.5 with the altered open behauvior. The disk > based queue-array developed by myself is just a fallback solution if the > disk-based queue doesn't work with an immediate shutdown. If it does not work under the constraints described here, this would point to a problem in the queue implementation (I have to admit the reason to provide a capability to write periodic qi file updates was related to a scenario like this, though not thought in this extreme ;)). Rainer > > David > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > > > [snip] > > > > > >> as long as you do sector based writes (512 byte per sector, usual) you > >> can be sure that the write wasn"t partial.. Writing more than one sector > >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > >> a partial write. I'll already tested that with my devel client here. So > >> fencing each sector with a crc32 value would help detecting errors > >> during a write operation. This is actually only a problem if you are > >> writing directly to a block device like any filesystem does and yes, > >> reordering is definitly a problem. So validating the content written to > >> the disk afterwards is important. > >> > >> If writing through a filesystem reserving space in the destination file > >> beforehand actually minimizes errors since the file system table doesn't > >> have to be updated (you should also use the Flag O_NOATIME for that > >> case). See for example VMWare ESX VMDK file handling. > >> > > > > Well, first of all let me re-iterate that I do not intend to do a block > > device driver for rsyslog (but I definitely do not object getting one > > contributed ;)). > > > > Still thinking about the case and thinking about non-solid-state, > > non-internal-battery-backed-up disk, I can't see how you can be sure the > > data will be written. David just told me there are no capacitors. So if > > power fails, it fails rather quickly. So how can you be sure the disk > > will be able to finish writing that sector? Let's say the drive has > > begun to write the sector and been able to write the first 5 bytes. Now > > power fails. No capacitors, no battery-backup, so why should there be > > enough power to drive the disk write head for another 507 bytes? It the > > drives assures it can do that, it needs capacitors - doesn't it? > > > > Am I overlooking something obvious? > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 15:18:55 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:18:55 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? in an audit-grade logging system yes, in an enterprise-grade system probably not. David Lang > Anybody else with an opinion? > Rainer > > On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> David, >>>>> >>>>> the file syncing mentioned in the compatibility doc applies to the >>>>> output action, only. >>>> >>>> ouch. >>>> >>>>> The queue does never do synchronous writes - I always assumed that a >>>>> critical system would have a UPS and could never think (so far) about a >>>>> valid reason for not having it. So the queue would need to have an extra >>>>> option to do sync writes. Obviously, that's not a big deal. >>>> >>>> good >>>> >>>>> Performance, of course, will be extremely terrible with such a setup... >>>> >>>> only if you have to wait for a spinning disk to do the write. >>> >>> I agree to the rest of your argument below. But the question raised here >>> was in regard to a system without any battery backup. So I would need to >>> wait. >> >> no UPS is not nessasarily the same as no battey backup. >> >> you could use a compact flash drive and probably get better >> performance/reliability than spinning disks with no battery at all. >> >>> Even then, in the worst case, I think it would be possible that the disk >>> does only a partial write. I am not sure if that's really the case with >>> today's disk drives (which I think have capacitors to prevent this >>> scenario), but with past drives this could happen (I know all too well - >>> a few years ago that cost me a weekend ;)). >> >> current disks do not have capacitors to prevent partial writes or to flush >> their caches. but options like the linux ext3 data-journaled make it so >> that you have your data in the journal safely, and the various solid-state >> options solve that problem. >> >> David Lang >> >>> Rainer >>> >>>> >>>> this is the same problem that databases have. they need to guarentee that >>>> once the database tells the writing program that the data is written it >>>> will be there even if the system looses power immediatly. >>>> >>>> if you run a database on standard desktop hardware (and it doesn't have >>>> this safety disabled) you cannot do more then about 80 writes/second. If >>>> you upgrade to the super speedy 15K rpm drives you can do ~160 >>>> writes/second. >>>> >>>> given that you need to write the data + metadata it gets even uglier, so >>>> what the databases do (and some journaling filesystems) is to write a log >>>> that says what they are going to do, sync that, and then later write the >>>> data to the actual files (updating the journal when they complete the >>>> write) >>>> >>>> it sounds like you order your write correctly for a disk-based queue, but >>>> you would need the option of issuing the syncs (probably when you do the >>>> checkpoints) >>>> >>>> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >>>> wrong flash drive), the fact that you need to do four writes per log entry >>>> (data to queue, metadata to queue, data to output, update metadata for >>>> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >>>> to seeking as well) >>>> >>>> however, with the correct drive to write to (say a $2,400 80G fusion-io >>>> flash card that can do ~100k IO ops/sec) you should be able to sustain >>>> 20,000 logs/sec. >>>> >>>> realisticly very few people need the sustained write capacity that you >>>> would get from such a setup. but if you go with a $500-$700 raid card with >>>> a battery-backed cache you get very similar performance, but with some >>>> possibility that you can't sustain it forever. >>>> >>>> David Lang >>>> >>>>> Rainer >>>>> >>>>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am looking for a failsafe solution to store syslog messages localy >>>>>>> until they could be send later. I already looked at the disk based >>>>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>>>> just power down the system immediatly actually loosing the whole queue. >>>>>> >>>>>> are you sure about the disk based queue? >>>>>> >>>>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>>>> can be set to do a commit of the metadata after each message. >>>>>> >>>>>> Disk Queues >>>>>> >>>>>> Disk queues use disk drives for buffering. The important fact is that the >>>>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>>>> this queue mode is not recommended. It is useful if log data is so >>>>>> important that it must not be lost, even in extreme cases. >>>>>> >>>>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>>>> its individual file. Files are named with a prefix (set via the >>>>>> "$QueueFilename" config directive) and followed by a 7-digit >>>>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>>>> default, a different size can be set via the"$QueueMaxFileSize" >>>>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>>>> always writes one complete queue entry, even if it violates the size >>>>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>>>> then the configured size. Each chunk also has a different size for the >>>>>> same reason. If you observe different chunk sizes, you can relax: this is >>>>>> not a problem. >>>>>> >>>>>> Writing in chunks is used so that processed data can quickly be deleted >>>>>> and is free for other uses - while at the same time keeping no artificial >>>>>> upper limit on disk space used. If a disk quota is set (instructions >>>>>> further below), be sure that the quota/chunk size allows at least two >>>>>> chunks to be written. Rsyslog currently does not check that and will fail >>>>>> miserably if a single chunk is over the quota. >>>>>> >>>>>> Creating new chunks costs performance but provides quicker ability to free >>>>>> disk space. The 10mb default is considered a good compromise between these >>>>>> two. However, it may make sense to adapt these settings to local policies. >>>>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>>>> make sense to use a 2gb (or even larger) chunk size. >>>>>> >>>>>> Please note, however, that the disk queue by default does not update its >>>>>> housekeeping structures every time it writes to disk. This is for >>>>>> performance reasons. In the event of failure, data will still be lost >>>>>> (except when manually is mangled with the file structures). However, disk >>>>>> queues can be set to write bookkeeping information on checkpoints (every n >>>>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>>>> interval is set to one, no data can be lost, but the queue is >>>>>> exceptionally slow. >>>>>> >>>>>> Each queue can be placed on a different disk for best performance and/or >>>>>> isolation. This is currently selected by specifying different >>>>>> $WorkDirectory config directives before the queue creation statement. >>>>>> >>>>>> To create a disk queue, use the "$QueueType Disk" config >>>>>> directive. Checkpoint intervals can be specified via >>>>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> you also need to specificly enable syncing (from >>>>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>>>> >>>>>> Output File Syncing >>>>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>>>> such, it retained stock syslogd's default of syncing every file write if >>>>>> not specified otherwise (by placing a dash in front of the output file >>>>>> name). While this was a useful feature in past days where hardware was >>>>>> much less reliable and UPS seldom, this no longer is useful in today's >>>>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>>>> writes files around 50 *times* slower than without it. It also affects >>>>>> overall system performance due to the high IO activity. In rsyslog v3, >>>>>> syncing has been turned off by default. This is done via a specific >>>>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>>>> default. In order to enable file syncing, the administrator must specify >>>>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>>>> syncing only happens in some installations where the administrator >>>>>> actually wanted that (performance-intense) feature. In the fast majority >>>>>> of cases (if not all), this dramatically increases rsyslogd performance >>>>>> without any negative effects. >>>>>> >>>>>> >>>>>> >>>>>>> I already looked at queue.c and it seemed to me that both queues were >>>>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>>>> an immediate power down of the system is the major failure which will >>>>>>> occure pretty often I need to create a soltution there. >>>>>> >>>>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>>>> in on the disk safely (assuming you have hardware that supports this) and >>>>>> a power-off won't affect it. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> >>>>>>> Did you already start to develop something addressing that problem? >>>>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>>>> would like afterwards. >>>>>>> >>>>>>> bye >>>>>>> David Ecker >>>>>>> >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 15:20:29 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:20:29 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E3791D.3050304@ecker-software.de> Hi, this is already the second version of this systems we develop. Not being able to do error analysis beacause of missing log data was one of the big problems including filesystem crashes. Having all logs and being able to proof the cause also helps a lot afterwards not only for creating a workaround for the incident but also to proof that the Service Level Agreement wasn't violated. bye David Ecker Rainer Gerhards schrieb: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. > > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. > > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) > > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. > > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? > > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >>> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:24:33 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:24:33 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? one possible thing is tht if the write has not completed then the system sending you the logs has not received confirmation that you have the log yet, so they are the ones responsible for it. it's only after you acknowledge the message (via relp or equivalent) that you are required to not loose the log message. with ext3 and data=journaled you can do this. the writes to the journal are done by the filesystem in such a way that they can be considered atomic (either they happen and are reliable, or they can be treated as if they never happened), once that write (or rather the sync on that write) has completed the data is safe. the filesystem will later modify the actual sector on disk, but it jumps through hoops to make that safe from power outages. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:24:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:24:24 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. mhhh... This sounds like you still may be in trouble if the file system crashes (as the queue files are stored inside that system). Rainer From david at lang.hm Wed Oct 1 15:31:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:31:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). pick the right filesystem and you should still be safe ext3+data=journaled is safe databases do the journaling themselves and like to store their journal files on ext2. you can use a log-structured filesystem that never overwrites data. it writes the new sector to a new place on disk then modifies pointers to tell the filesystem that the data is in the new place instead of the old place (the results in horrible fragmentation in many cases, but it's _very_ safe) some of this can be done without doing a sync for every write with barriers, which can guarentee that a write before the barrier takes place before any writes after the barrier. this lets you do things like modify the queue and then (after you know the data is safe) modify the metadata to indicate that the new data is there. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:31:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:31:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > one possible thing is tht if the write has not completed then the system > sending you the logs has not received confirmation that you have the log > yet, so they are the ones responsible for it. I concur. But as I understood the scenario here, the log messages are emited from a process *inside* the failing machine. So that process fails, too, and we do not have any interim that has a copy of the data. So if it is not persisted to disk, it is lost. Anyhow, this requirement has been relaxed in later posting. Also note that I was just thinking about the physical layer, considering a single physical write - far away from rsyslog. > it's only after you acknowledge the message (via relp or equivalent) that > you are required to not loose the log message. I concur and this is how RELP handles this. Well, actually I think there currently is a very slight (but still existing) window of exposure. I think RELP acks when the message is submited to the queue engine. That does not necessarily mean the message is already present on disk. Also, I think some mild duplication of messages may happen with RELP in a power fail scenario. It is not doing a two-phase commit, thought it tries very hard to get a perfect understanding of what is written and what not. I could check this with spec/code, but I think this is not justified at this point in time ;) > > with ext3 and data=journaled you can do this. the writes to the journal > are done by the filesystem in such a way that they can be considered > atomic (either they happen and are reliable, or they can be treated as if > they never happened), once that write (or rather the sync on that write) > has completed the data is safe. the filesystem will later modify the > actual sector on disk, but it jumps through hoops to make that safe from > power outages. > sure, definitely. And I assume a sync'ed write will return only after all this has happened. So journaled ext3 should be able to solve the problem described here. Rainer From david at lang.hm Wed Oct 1 15:35:01 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:35:01 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> Even then, in the worst case, I think it would be possible that the disk >> does only a partial write. I am not sure if that's really the case with >> today's disk drives (which I think have capacitors to prevent this >> scenario), but with past drives this could happen (I know all too well - >> a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. actually, I need to correct my answer. I know that disks do not have capacitors large enough to write their buffer. I'm pretty sure that they don't have capacitors large enough to write an entire track. but they may have capacitors large enough to finish the sector they are on before stopping (considering the number of sectors in a track nowdays this is a _very_ sort time) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:34:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:34:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: > some of this can be done without doing a sync for every write with > barriers, which can guarentee that a write before the barrier takes place > before any writes after the barrier. this lets you do things like modify > the queue and then (after you know the data is safe) modify the metadata > to indicate that the new data is there. That sounds interesting - can you point me to some API documentation? A good place to do that would be inside the queue after the qi file has been persisted. That could reduce performance toll of synchronous writes. Especially if loss of a few messages is acceptable (as David Ecker just said). Rainer From david at ecker-software.de Wed Oct 1 15:35:18 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:35:18 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37C96.3080904@ecker-software.de> I know. I used reiserfs before. The client systems were turned off like 4 times a day (average). At that point we had only 60 clients running. At least once a day one client needed to be reimaged since the filesystem was broke. The system itself wrote too much to the local disk during operation. So the failure rate was actually below 0,5% but it keept a constand buzz for the support center. I just was lucky that I implemented a failback image so that the systems could reimage themselfs without anybody going there. In this since the I would do it like this: Since the filesystem would only store the queue files of the syslog and nothing else I would check the filesystem at startup. If it's broken I would reformat the fs automtically. So that would leave me with a loss of 0,5% of all logs. Thats a lot better than nothing. I just have to make sure, that a rollback of the fs leaves the syslog data queue intact. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. >> > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:40:30 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:40:30 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > >> one possible thing is tht if the write has not completed then the system >> sending you the logs has not received confirmation that you have the log >> yet, so they are the ones responsible for it. > > I concur. But as I understood the scenario here, the log messages are > emited from a process *inside* the failing machine. So that process > fails, too, and we do not have any interim that has a copy of the data. > So if it is not persisted to disk, it is lost. Anyhow, this requirement > has been relaxed in later posting. > > Also note that I was just thinking about the physical layer, considering > a single physical write - far away from rsyslog. > >> it's only after you acknowledge the message (via relp or equivalent) that >> you are required to not loose the log message. > > I concur and this is how RELP handles this. Well, actually I think there > currently is a very slight (but still existing) window of exposure. I > think RELP acks when the message is submited to the queue engine. That > does not necessarily mean the message is already present on disk. right now it doesn't make a difference (since the queue doesn't sync the data), but if/when this is added checking that the call to queue the data doesn't return until after that point should be done. > Also, > I think some mild duplication of messages may happen with RELP in a > power fail scenario. It is not doing a two-phase commit, thought it > tries very hard to get a perfect understanding of what is written and > what not. I could check this with spec/code, but I think this is not > justified at this point in time ;) no matter what checks you do, it's always possible for things to fail after succeeding on doing the work and before the acknowlegement gets back to the sender (with relp, if the receiving machine sends the acknowledgement, but the sending machine crashes before fully processing it, the receiving machine has no way of detecting this) on the other hand, this is fairly easy to deal with by making messages include a sequence number to guarentee that they are all unique, and then have something filter out duplicates later. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:39:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:39:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868396.2682.511.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:35 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, david at lang.hm wrote: > > > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > >> Even then, in the worst case, I think it would be possible that the disk > >> does only a partial write. I am not sure if that's really the case with > >> today's disk drives (which I think have capacitors to prevent this > >> scenario), but with past drives this could happen (I know all too well - > >> a few years ago that cost me a weekend ;)). > > > > current disks do not have capacitors to prevent partial writes or to flush > > their caches. but options like the linux ext3 data-journaled make it so > > that you have your data in the journal safely, and the various solid-state > > options solve that problem. > > actually, I need to correct my answer. > > I know that disks do not have capacitors large enough to write their > buffer. absolutely > > I'm pretty sure that they don't have capacitors large enough to write an > entire track. > but they may have capacitors large enough to finish the sector they are on > before stopping (considering the number of sectors in a track nowdays this > is a _very_ sort time) *That* I had expected. Especially if you think about the low-level sync marks which otherwise could be affected. I think loss of disk could otherwise be the extreme result of a power failure (or at least loss of disk track). But you never know what a failing write arm does... So I would think (but have no evidence) that current disk drives have capacitors large enough to finish the write AND shut down the write mechanism in an orderly manner. As you say, there are "not many electrons" required to ensure this. Rainer From david at lang.hm Wed Oct 1 15:42:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:42:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: >> some of this can be done without doing a sync for every write with >> barriers, which can guarentee that a write before the barrier takes place >> before any writes after the barrier. this lets you do things like modify >> the queue and then (after you know the data is safe) modify the metadata >> to indicate that the new data is there. > > That sounds interesting - can you point me to some API documentation? A > good place to do that would be inside the queue after the qi file has > been persisted. That could reduce performance toll of synchronous > writes. Especially if loss of a few messages is acceptable (as David > Ecker just said). I'll try to find it (I've seen it discussed on the linux-kernel mailing list by the filesystem developers, but I don't know how it's exposed to userspace) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:46:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:46:52 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868812.2682.515.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:40 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > > > >> one possible thing is tht if the write has not completed then the system > >> sending you the logs has not received confirmation that you have the log > >> yet, so they are the ones responsible for it. > > > > I concur. But as I understood the scenario here, the log messages are > > emited from a process *inside* the failing machine. So that process > > fails, too, and we do not have any interim that has a copy of the data. > > So if it is not persisted to disk, it is lost. Anyhow, this requirement > > has been relaxed in later posting. > > > > Also note that I was just thinking about the physical layer, considering > > a single physical write - far away from rsyslog. > > > >> it's only after you acknowledge the message (via relp or equivalent) that > >> you are required to not loose the log message. > > > > I concur and this is how RELP handles this. Well, actually I think there > > currently is a very slight (but still existing) window of exposure. I > > think RELP acks when the message is submited to the queue engine. That > > does not necessarily mean the message is already present on disk. > > right now it doesn't make a difference (since the queue doesn't sync the > data), but if/when this is added checking that the call to queue the data > doesn't return until after that point should be done. I just checked the code. Actually, librelp already ensures this. It acks only after the callback has successfully completed. > > > Also, > > I think some mild duplication of messages may happen with RELP in a > > power fail scenario. It is not doing a two-phase commit, thought it > > tries very hard to get a perfect understanding of what is written and > > what not. I could check this with spec/code, but I think this is not > > justified at this point in time ;) > > no matter what checks you do, it's always possible for things to fail > after succeeding on doing the work and before the acknowlegement gets back > to the sender (with relp, if the receiving machine sends the > acknowledgement, but the sending machine crashes before fully processing > it, the receiving machine has no way of detecting this) RELP has a "Window of uncertainty", in which peers negotiate where to restart. But power failure was not on my spec list, so this only works if the shutdown was relatively clean. > > on the other hand, this is fairly easy to deal with by making messages > include a sequence number to guarentee that they are all unique, and then > have something filter out duplicates later. well said - among others one reason why RELP currently is not receiving more attention that just letting it mature... Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 15:59:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:59:34 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37C96.3080904@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> Message-ID: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> David (Ecker ;)), I'd appreciate if you let me know your results after you have tested with $..QueueCheckpointInterval 1 Rainer From david at lang.hm Wed Oct 1 16:10:38 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 07:10:38 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David (Ecker ;)), > > I'd appreciate if you let me know your results after you have tested > with $..QueueCheckpointInterval 1 also, try mounting the filesystem -o sync this should make everything on it a sync write without Rainer having to change his code (won't work for other situations, but will definantly work for a test) David Lang From david at ecker-software.de Wed Oct 1 16:19:45 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 16:19:45 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E38701.80707@ecker-software.de> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My guess is, that the O_DIRECT in combination with the O_SYNC flag (turning of cache) will have an impact. I won't be able to test it tomorrow and since Friday is a vaction in germany I'll be able to tell you on monday the results of my tests. Thanks for your help. I was actually positivly suprised by the constructiveness of our discussion. David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > >> David (Ecker ;)), >> >> I'd appreciate if you let me know your results after you have tested >> with $..QueueCheckpointInterval 1 >> > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 1 16:32:28 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:32:28 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? > > Anybody else with an opinion? Yes, I have an opinion: no. Enterprise-grade applications need to make the best effort they can to be fault-tolerant and reliable, but we're following an exponential curve - at some point, you _have_ to give over to the hardware and OS engineers to do their job properly. Don't reinvent the hammer & chisel to reinvent the wheel - software can _never_ account for all hardware failures and should never be expected to do so. RB From rgerhards at hq.adiscon.com Wed Oct 1 16:39:36 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 16:39:36 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> Message-ID: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Hi RB, I guess you've seen in the discussion (well, if you read all these mails, I just noticed how many they were...) I am coming from the same camp. But after all the dust settles, it looks like a very minor change to rsyslog to allow the OS to do what it is expected to do. The queue logic already cares about proper sequence (at least it should), the only thing that is missing is to tell the OS to sync when we need it. The spot to do so is also well defined, that is when the qi file is persisted (because that completes the "queue transaction"). I will verify with the code, but my current guess is that no more than 10 lines of code will be needed to support this functionality. If so, I think it is worth it. More complex questions may arise (I have some on my mind) and that will probably go beyond a general-purpose solution. Rainer On Wed, 2008-10-01 at 08:32 -0600, RB wrote: > On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > > going back to the higher layer: do you say that immediate power failure > > is a case that you consider needed to be addressed in an enterprise > > logging system? > > > > Anybody else with an opinion? > > Yes, I have an opinion: no. Enterprise-grade applications need to > make the best effort they can to be fault-tolerant and reliable, but > we're following an exponential curve - at some point, you _have_ to > give over to the hardware and OS engineers to do their job properly. > Don't reinvent the hammer & chisel to reinvent the wheel - software > can _never_ account for all hardware failures and should never be > expected to do so. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Wed Oct 1 16:40:21 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:40:21 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. Having all logs and being > able to proof the cause also helps a lot afterwards not only for > creating a workaround for the incident but also to proof that the > Service Level Agreement wasn't violated. Somewhat randomly selecting one to respond to - you guys generated a lot of chatter. My view as both a software and hardware engineer on this is simple: never try to solve a hardware problem with a software solution. You will never win. It is possible to account for many edge cases, but no matter how paranoid you may be, hardware always trumps software - it has the last word, regardless of what you may try to do. RB From aoz.syn at gmail.com Wed Oct 1 16:50:24 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:50:24 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> > I guess you've seen in the discussion (well, if you read all these > mails, I just noticed how many they were...) ;-) I was wondering why my hip was rattling during the entire commute today. > persisted (because that completes the "queue transaction"). I will > verify with the code, but my current guess is that no more than 10 lines > of code will be needed to support this functionality. If so, I think it > is worth it. If it's something you can solve without worrying about block-level writes and whether the underlying drive (if indeed there is even a "drive") has a battery-backed cache or sufficient capacitor charge to write your data, I'm all for it. Even better if it's POSIX. Although a block driver and other filesystem-bypassing solutions may be interesting in limited cases, I'd rather not see anyone stab their wife over it. From rgerhards at hq.adiscon.com Wed Oct 1 17:18:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:18:11 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> David, On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David (Ecker ;)), > > > > I'd appreciate if you let me know your results after you have tested > > with $..QueueCheckpointInterval 1 > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) I after all this discussion I will take a few minutes to see if I can capture results in code. There are two things and I guess you have an opinion on that. First of all, the doc I have on fsync() is a bit cryptic. It says: ---- fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)). ---- so it looks like fsync() is sufficient to flush both the data as well as the size information (so that data is not only present but can actually be read ;). But in the next paragraph, the man page says: ---- Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed. ---- which sound much like the opposite. I know there are many flavours of *nix and even though this is a POSIX call, things may be different in different OS... Do you have an opinion of we can assume a call to fsync() on the descriptor in questions solves the issue? This would probably avoid the need to open with always synced writes and thus could offer a somewhat better performance. Also, the thought about O_NOATIME looks interesting. For internal queue files, I think the last access time is of very limited interest, so it would probably be useful to always specify this option (at least by default). Feedback appreciated. Rainer From david at lang.hm Wed Oct 1 17:41:11 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:41:11 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) > > I after all this discussion I will take a few minutes to see if I can > capture results in code. There are two things and I guess you have an > opinion on that. > > First of all, the doc I have on fsync() is a bit cryptic. It says: > > ---- > fsync() transfers ("flushes") all modified in-core data of (i.e., > modified buffer cache pages for) the file referred to by the file > descriptor fd to the disk device (or other permanent storage device) > where that file resides. The call blocks until the device reports that > the transfer has completed. It also flushes metadata information > associated with the file (see stat(2)). > ---- > > so it looks like fsync() is sufficient to flush both the data as well as > the size information (so that data is not only present but can actually > be read ;). But in the next paragraph, the man page says: > > ---- > Calling fsync() does not necessarily ensure that the entry in the > directory containing the file has also reached disk. For that an > explicit fsync() on a file descriptor for the directory is also needed. > ---- > > which sound much like the opposite. I know there are many flavours of > *nix and even though this is a POSIX call, things may be different in > different OS... > > Do you have an opinion of we can assume a call to fsync() on the > descriptor in questions solves the issue? This would probably avoid the > need to open with always synced writes and thus could offer a somewhat > better performance. my understanding is that you may need to call both. if you have pre-allocated the file you would not need to call fsync on the directory, if you are dynamicly extending it you will need to. > Also, the thought about O_NOATIME looks interesting. For internal queue > files, I think the last access time is of very limited interest, so it > would probably be useful to always specify this option (at least by > default). in this case noatime won't make much difference. normally you want it so that when you read a file it doesn't have to update the 'last accessed' time in the directory, but since we are writing to these just about every time we access them we are having to update the 'last modified' time there anyway. I am assuming that even though the data is being written to the disk queue file, it's still available in memory so you aren't having to do a disk read to read the message from the queue. If this is not the case then noatime could make a difference if the file is opened O_SYNC becouse the read would then require the directory update to be synced before you could do anything else. If you are dong the fsync calls yourself it shouldn't matter much becouse the atime update will (probably) not get pushed out to disk before you are going back and modifying the queue to indicate that you wrote the message to the output (when the 'last modified' times will get updated) David Lang > Feedback appreciated. > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 17:43:52 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:43:52 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E38701.80707@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > of cache) will have an impact. O_DIRECT is doing very different things. I don't think you need to worry about those things, having the data not go into the OS cache is a drawback not an advantage becouse it means that when you go to pull the data back out of the file it will need to actually touch disk. it also imposes significant alignment issues on the application that I don't think you want to have to desl with. David Lang > I won't be able to test it tomorrow and since Friday is a vaction in > germany I'll be able to tell you on monday the results of my tests. > > Thanks for your help. I was actually positivly suprised by the > constructiveness of our discussion. > David Ecker > > david at lang.hm schrieb: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >>> >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 17:44:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:44:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > > of cache) will have an impact. > > O_DIRECT is doing very different things. I don't think you need to worry > about those things, having the data not go into the OS cache is a drawback > not an advantage becouse it means that when you go to pull the data back > out of the file it will need to actually touch disk. it also imposes > significant alignment issues on the application that I don't think you > want to have to desl with. plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... Rainer From david at lang.hm Wed Oct 1 17:45:16 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:45:16 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. Having all logs and being >> able to proof the cause also helps a lot afterwards not only for >> creating a workaround for the incident but also to proof that the >> Service Level Agreement wasn't violated. > > Somewhat randomly selecting one to respond to - you guys generated a > lot of chatter. > > My view as both a software and hardware engineer on this is simple: > never try to solve a hardware problem with a software solution. You > will never win. It is possible to account for many edge cases, but no > matter how paranoid you may be, hardware always trumps software - it > has the last word, regardless of what you may try to do. I'm not understanding the point you are trying to make. are you saying that it's a bad idea to try and have an option to do the syncing we are talking about for the queue? David Lang From david at lang.hm Wed Oct 1 17:48:57 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:48:57 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I guess you've seen in the discussion (well, if you read all these >> mails, I just noticed how many they were...) > > ;-) I was wondering why my hip was rattling during the entire commute today. > >> persisted (because that completes the "queue transaction"). I will >> verify with the code, but my current guess is that no more than 10 lines >> of code will be needed to support this functionality. If so, I think it >> is worth it. > > If it's something you can solve without worrying about block-level > writes and whether the underlying drive (if indeed there is even a > "drive") has a battery-backed cache or sufficient capacitor charge to > write your data, I'm all for it. Even better if it's POSIX. > > Although a block driver and other filesystem-bypassing solutions may > be interesting in limited cases, I'd rather not see anyone stab their > wife over it. I agree that trying to bypass the filesystem is highly questionable, and not something for a core change (as always a contributed vrsion can be tested to see if it makes a difference) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 17:58:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:58:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: <1222876701.2682.541.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:48 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, RB wrote: > > >> I guess you've seen in the discussion (well, if you read all these > >> mails, I just noticed how many they were...) > > > > ;-) I was wondering why my hip was rattling during the entire commute today. > > > >> persisted (because that completes the "queue transaction"). I will > >> verify with the code, but my current guess is that no more than 10 lines > >> of code will be needed to support this functionality. If so, I think it > >> is worth it. > > > > If it's something you can solve without worrying about block-level > > writes and whether the underlying drive (if indeed there is even a > > "drive") has a battery-backed cache or sufficient capacitor charge to > > write your data, I'm all for it. Even better if it's POSIX. > > > > Although a block driver and other filesystem-bypassing solutions may > > be interesting in limited cases, I'd rather not see anyone stab their > > wife over it. > > I agree that trying to bypass the filesystem is highly questionable, and > not something for a core change (as always a contributed vrsion can be > tested to see if it makes a difference) Just FYI: preliminary analysis indicates that is probably around the 10 lines of code that need to be added in stream.c, maybe a few more. The idea is that I can set a flag similar to O_SYNC on stream creation but then sync when the "atomic" writes are done. This may save some few cycles over an O_SYNC open(). However, there need to be a few config settings, which in turn need to be passed down to the queue and stream class. That adds more code, maybe around 100 lines (the config interface needs to be redone, thus the many LOC required, it's on the todo list...). An alternative is to use a simple global $AllWritesSync on/off option, which would probably be sufficient and cut down changes required. For a robust implementation, some more analysis is required (including thinking about the implications of fsync()). So it is not totally trivial, but well doable. I am just not sure if I'll do it immediately, there are many things in the work queue. Testing effort is probably much bigger than implementation effort, there are so many cases to check out... Rainer From aoz.syn at gmail.com Wed Oct 1 18:00:10 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 10:00:10 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> > I'm not understanding the point you are trying to make. > > are you saying that it's a bad idea to try and have an option to do the > syncing we are talking about for the queue? Sync: not as long as it's optional. Worrying about sector writes and capacitors/battery-backed cache on the underlying drives: yes. Unless rsyslog starts writing to raw devices, it is my opinion that once it reasonably hands the data off to the filesystem, it becomes a kernel and/or hardware problem and unnecessarily complex for a userspace application to govern. I'm all for anything rsyslog can do to encourage proper behavior (like calling sync()) without getting into the kernel/hardware space. RB From david at lang.hm Wed Oct 1 18:07:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 09:07:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I'm not understanding the point you are trying to make. >> >> are you saying that it's a bad idea to try and have an option to do the >> syncing we are talking about for the queue? > > Sync: not as long as it's optional. Worrying about sector writes and > capacitors/battery-backed cache on the underlying drives: yes. Unless > rsyslog starts writing to raw devices, it is my opinion that once it > reasonably hands the data off to the filesystem, it becomes a kernel > and/or hardware problem and unnecessarily complex for a userspace > application to govern. I'm all for anything rsyslog can do to > encourage proper behavior (like calling sync()) without getting into > the kernel/hardware space. Ok, we are on the same page then. in case I confused anyone, the reason I went into detail on the hardware side of things was to explain how proper hardware selection could result in good performance while poor hardware selection would result is dismal performance (and to give people who aren't familiar with the options some hints as to what they could do) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 18:51:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 18:51:52 +0200 Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> [going back to the list with Lorenzo's permission] Hi Lorenzo, finally... I did not realize that the queue is running in direct mode, but now I saw it and can reproduce a failure. That should make things much better. Anyhow, it is very unusual that the main message queue runs in direct mode, so I guess it is a side-effect of a config file. In those files you sent me, I don't see it, but I see a .d directory is included. May there be something? In any case, I hope I will be able to fix the bug now that I can reproduce it. Looks like a productive day today :) Thanks for your persistence, Rainer > -----Original Message----- > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > Sent: Wednesday, October 01, 2008 1:46 PM > To: Rainer Gerhards > Subject: Re: [rsyslog] abort case > > Sorry for the delay, I've just run helgrind's head, bau as you can see, > it > crashed almost as soon as I started. > > The log is enclosed; I'm now checking-out head and will retry. > > Yours, > > lorenzo > > RG> I have created a new version with one slight change, to be found in > RG> the helgrind branch. > RG> > RG> There is also a new valgrind tool called drd inside the valgrind > RG> development tree. I think you already downloaded that tree. If so, > RG> could you please replace > RG> > RG> Valgrind --tool=helgrind .. rsyslogd ... > RG> > RG> with > RG> > RG> Valgrind --tool=drd .. rsyslogd ... > RG> > RG> Drd does an even better job than helgrind. I also changed the > source > RG> to remove some debug-system related warning, which otherwise would > RG> clutter up the error message. But unfortunately, in my lab I did > not > RG> find any more problems, except for the small change I mentioned. > But > RG> that one affects program termination (and in a very subtle way), so > it > RG> should not change anything for you. I'd still be interested in a > new > RG> run, including debug info, from you. > RG> > > > +-------------------------+-------------------------------------------- > --+ > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > | > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > Vergata" | > | | Via O. Raimondo 18 ** I-00173 ROMA ** > ITALY | > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > | > +-------------------------+-------------------------------------------- > --+ From rgerhards at hq.adiscon.com Wed Oct 1 19:02:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 19:02:15 +0200 Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 01, 2008 6:52 PM > To: Lorenzo M. Catucci > Cc: rsyslog-users > Subject: Re: [rsyslog] abort case > > [going back to the list with Lorenzo's permission] > > Hi Lorenzo, > > finally... I did not realize that the queue is running in direct mode, > but now I saw it and can reproduce a failure. That should make things > much better. Anyhow, it is very unusual that the main message queue > runs in direct mode, so I guess it is a side-effect of a config file. > In those files you sent me, I don't see it, but I see a .d directory is > included. May there be something? In any case, I hope I will be able to > fix the bug now that I can reproduce it. > > Looks like a productive day today :) > > Thanks for your persistence, > Rainer > > > -----Original Message----- > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > > Sent: Wednesday, October 01, 2008 1:46 PM > > To: Rainer Gerhards > > Subject: Re: [rsyslog] abort case > > > > Sorry for the delay, I've just run helgrind's head, bau as you can > see, > > it > > crashed almost as soon as I started. > > > > The log is enclosed; I'm now checking-out head and will retry. > > > > Yours, > > > > lorenzo > > > > RG> I have created a new version with one slight change, to be found > in > > RG> the helgrind branch. > > RG> > > RG> There is also a new valgrind tool called drd inside the valgrind > > RG> development tree. I think you already downloaded that tree. If > so, > > RG> could you please replace > > RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... > > RG> > > RG> with > > RG> > > RG> Valgrind --tool=drd .. rsyslogd ... > > RG> > > RG> Drd does an even better job than helgrind. I also changed the > > source > > RG> to remove some debug-system related warning, which otherwise > would > > RG> clutter up the error message. But unfortunately, in my lab I did > > not > > RG> find any more problems, except for the small change I mentioned. > > But > > RG> that one affects program termination (and in a very subtle way), > so > > it > > RG> should not change anything for you. I'd still be interested in a > > new > > RG> run, including debug info, from you. > > RG> > > > > > > +-------------------------+------------------------------------------ > -- > > --+ > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > > | > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > > Vergata" | > > | | Via O. Raimondo 18 ** I-00173 ROMA ** > > ITALY | > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > > | > > +-------------------------+------------------------------------------ > -- > > --+ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:05:31 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:05:31 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> [going back to the list with Lorenzo's permission] RG> [...] RG> RG> finally... I did not realize that the queue is running in direct mode, RG> but now I saw it and can reproduce a failure. That should make things RG> much better. Anyhow, it is very unusual that the main message queue RG> runs in direct mode, so I guess it is a side-effect of a config file. RG> No, they are really the only config files present on the server (rsyslog.conf is in /etc, the other one is the only thing inside /etc/rsyslog.d/) Really, I wouldn't go back to a saner configuration until we zeroed on the bug... still, nowhere I touched the MainQueue options! Hope to hear from you soon! Yours, lorenzo RG> RG> In those files you sent me, I don't see it, but I see a .d directory RG> is included. May there be something? In any case, I hope I will be RG> able to fix the bug now that I can reproduce it. RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:16:23 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:16:23 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: It just crashed under drd (sending both direct and through the list, it should get to you!). I've restarted under helgrind, just for the sake of not wasting the upcoming night... Hear you tomorrow! Yours, lorenzo On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) RG> RG> Rainer RG> RG> > -----Original Message----- RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards RG> > Sent: Wednesday, October 01, 2008 6:52 PM RG> > To: Lorenzo M. Catucci RG> > Cc: rsyslog-users RG> > Subject: Re: [rsyslog] abort case RG> > RG> > [going back to the list with Lorenzo's permission] RG> > RG> > Hi Lorenzo, RG> > RG> > finally... I did not realize that the queue is running in direct mode, RG> > but now I saw it and can reproduce a failure. That should make things RG> > much better. Anyhow, it is very unusual that the main message queue RG> > runs in direct mode, so I guess it is a side-effect of a config file. RG> > In those files you sent me, I don't see it, but I see a .d directory is RG> > included. May there be something? In any case, I hope I will be able to RG> > fix the bug now that I can reproduce it. RG> > RG> > Looks like a productive day today :) RG> > RG> > Thanks for your persistence, RG> > Rainer RG> > RG> > > -----Original Message----- RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] RG> > > Sent: Wednesday, October 01, 2008 1:46 PM RG> > > To: Rainer Gerhards RG> > > Subject: Re: [rsyslog] abort case RG> > > RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can RG> > see, RG> > > it RG> > > crashed almost as soon as I started. RG> > > RG> > > The log is enclosed; I'm now checking-out head and will retry. RG> > > RG> > > Yours, RG> > > RG> > > lorenzo RG> > > RG> > > RG> I have created a new version with one slight change, to be found RG> > in RG> > > RG> the helgrind branch. RG> > > RG> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind RG> > > RG> development tree. I think you already downloaded that tree. If RG> > so, RG> > > RG> could you please replace RG> > > RG> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... RG> > > RG> RG> > > RG> with RG> > > RG> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... RG> > > RG> RG> > > RG> Drd does an even better job than helgrind. I also changed the RG> > > source RG> > > RG> to remove some debug-system related warning, which otherwise RG> > would RG> > > RG> clutter up the error message. But unfortunately, in my lab I did RG> > > not RG> > > RG> find any more problems, except for the small change I mentioned. RG> > > But RG> > > RG> that one affects program termination (and in a very subtle way), RG> > so RG> > > it RG> > > RG> should not change anything for you. I'd still be interested in a RG> > > new RG> > > RG> run, including debug info, from you. RG> > > RG> RG> > > RG> > > RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione RG> > > | RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor RG> > > Vergata" | RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** RG> > > ITALY | RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 RG> > > | RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > _______________________________________________ RG> > rsyslog mailing list RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog RG> > http://www.rsyslog.com RG> _______________________________________________ RG> rsyslog mailing list RG> http://lists.adiscon.net/mailman/listinfo/rsyslog RG> http://www.rsyslog.com RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:20:48 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:20:48 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: This time I attach the file! Sorry! On Wed, 1 Oct 2008, Lorenzo M. Catucci wrote: LMC> It just crashed under drd (sending both direct and through the list, it LMC> should get to you!). I've restarted under helgrind, just for the sake of LMC> not wasting the upcoming night... LMC> LMC> Hear you tomorrow! Yours, LMC> LMC> lorenzo LMC> LMC> On Wed, 1 Oct 2008, Rainer Gerhards wrote: LMC> LMC> RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) LMC> RG> LMC> RG> Rainer LMC> RG> LMC> RG> > -----Original Message----- LMC> RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- LMC> RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards LMC> RG> > Sent: Wednesday, October 01, 2008 6:52 PM LMC> RG> > To: Lorenzo M. Catucci LMC> RG> > Cc: rsyslog-users LMC> RG> > Subject: Re: [rsyslog] abort case LMC> RG> > LMC> RG> > [going back to the list with Lorenzo's permission] LMC> RG> > LMC> RG> > Hi Lorenzo, LMC> RG> > LMC> RG> > finally... I did not realize that the queue is running in direct mode, LMC> RG> > but now I saw it and can reproduce a failure. That should make things LMC> RG> > much better. Anyhow, it is very unusual that the main message queue LMC> RG> > runs in direct mode, so I guess it is a side-effect of a config file. LMC> RG> > In those files you sent me, I don't see it, but I see a .d directory is LMC> RG> > included. May there be something? In any case, I hope I will be able to LMC> RG> > fix the bug now that I can reproduce it. LMC> RG> > LMC> RG> > Looks like a productive day today :) LMC> RG> > LMC> RG> > Thanks for your persistence, LMC> RG> > Rainer LMC> RG> > LMC> RG> > > -----Original Message----- LMC> RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] LMC> RG> > > Sent: Wednesday, October 01, 2008 1:46 PM LMC> RG> > > To: Rainer Gerhards LMC> RG> > > Subject: Re: [rsyslog] abort case LMC> RG> > > LMC> RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can LMC> RG> > see, LMC> RG> > > it LMC> RG> > > crashed almost as soon as I started. LMC> RG> > > LMC> RG> > > The log is enclosed; I'm now checking-out head and will retry. LMC> RG> > > LMC> RG> > > Yours, LMC> RG> > > LMC> RG> > > lorenzo LMC> RG> > > LMC> RG> > > RG> I have created a new version with one slight change, to be found LMC> RG> > in LMC> RG> > > RG> the helgrind branch. LMC> RG> > > RG> LMC> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind LMC> RG> > > RG> development tree. I think you already downloaded that tree. If LMC> RG> > so, LMC> RG> > > RG> could you please replace LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> with LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> Drd does an even better job than helgrind. I also changed the LMC> RG> > > source LMC> RG> > > RG> to remove some debug-system related warning, which otherwise LMC> RG> > would LMC> RG> > > RG> clutter up the error message. But unfortunately, in my lab I did LMC> RG> > > not LMC> RG> > > RG> find any more problems, except for the small change I mentioned. LMC> RG> > > But LMC> RG> > > RG> that one affects program termination (and in a very subtle way), LMC> RG> > so LMC> RG> > > it LMC> RG> > > RG> should not change anything for you. I'd still be interested in a LMC> RG> > > new LMC> RG> > > RG> run, including debug info, from you. LMC> RG> > > RG> LMC> RG> > > LMC> RG> > > LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione LMC> RG> > > | LMC> RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor LMC> RG> > > Vergata" | LMC> RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** LMC> RG> > > ITALY | LMC> RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 LMC> RG> > > | LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > _______________________________________________ LMC> RG> > rsyslog mailing list LMC> RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> > http://www.rsyslog.com LMC> RG> _______________________________________________ LMC> RG> rsyslog mailing list LMC> RG> http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> http://www.rsyslog.com LMC> RG> LMC> LMC> +-------------------------+----------------------------------------------+ LMC> | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | LMC> | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | LMC> | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | LMC> | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | LMC> +-------------------------+----------------------------------------------+ +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Thu Oct 2 15:14:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:14:34 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Thanks to the debug data provided, I could find out that the timestamps stem back to imfile and the way (sequence) in which it generates the timestamps. I will take the opportunity and fix it in a way that also improves performance. I just thought I let you know, more info follows when I have written the fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 2:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Oct 2 15:54:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:54:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> I (think) I have now fixed it, but not verified all cases. I would appreciate if you could give the new snapshot a try: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=39000a62024510cd62607 200e6100e3cd7c05005;sf=tgz Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, October 02, 2008 3:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks to the debug data provided, I could find out that the timestamps > stem back to imfile and the way (sequence) in which it generates the > timestamps. I will take the opportunity and fix it in a way that also > improves performance. I just thought I let you know, more info follows > when I have written the fix. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > > Sent: Wednesday, October 01, 2008 2:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > > > I installed the snapshot but apparently the issue persists. > > > > If there's something else that I can do, now or when you have more > > time, let me know. > > > > Regards, > > Luigi > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Oct 2 17:29:30 2008 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 2 Oct 2008 17:29:30 +0200 Subject: [rsyslog] could not load TLS module even when it's there! Message-ID: <0B1B9163138571439EAF804646F3F6AA05917715@GIBSVWIN004X.partygaming.local> Hi list, I'm setting up rsyslog to use TLS. The server-side comes up fine and I hope it's expecting encrypted traffic... Anyway, when I restart rsyslog on a client which is supposed to forward its syslogs to the logserver I see the following error message: -- ... rsyslogd: # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so'/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 : No such file or directory -- ...but funny enough: -- # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so -- What am I doing wrong?? TIA, Martin From luigi.perroti at googlemail.com Fri Oct 3 17:00:57 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Fri, 3 Oct 2008 17:00:57 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: Everything's fine here. Now I get the same subsecond part for all the events. This is how it should be since they are being generated on the same host. If it's of relevance I'm using these modules: imfile, imuxsock, imklog, ommysql. I'm also listening on a couple of additional unix sockets. Everything seems to be working fine. Thanks for the quick turnaround. Regards, Luigi From rgerhards at hq.adiscon.com Fri Oct 3 17:47:41 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 3 Oct 2008 17:47:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F28A@grfint2.intern.adiscon.com> Excellent, thanks for the confirmation. The modules are no longer relevant, they all get the time from a consistent call now (not doing this was part of the problem, if not the whole ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Friday, October 03, 2008 5:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Everything's fine here. Now I get the same subsecond part for all the > events. > This is how it should be since they are being generated on the same > host. > > If it's of relevance I'm using these modules: > imfile, imuxsock, imklog, ommysql. > I'm also listening on a couple of additional unix sockets. > Everything seems to be working fine. > > Thanks for the quick turnaround. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Oct 6 11:24:05 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:24:05 +0200 Subject: [rsyslog] HUP action Message-ID: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Hi List, I again have a backwards compatibility question. HUP, in sysklogd, means a full restart of the syslogd. This is done because the configuration may have been changed and to apply the new one, the previous one must be shut down. This is even more so the case in rsyslog, which has multiple dynamically loadable plugins and all that... rsyslog has implemented HUP processing to be compatible to sysklogd. However, in practice (as far as I know) HUP is almost always being used to close the output files after log rotation. An automatted config reaload indication usually does not happen (but operators use HUP after doing a config change). HUP, as it currently is, is an extremely expensive operation. Keep in mind that it is a full daemon restart, so, among other things, it unloads modules and tears down (TCP, RELP, TLS, ...) connections. In most cases, though, this would not be needed. I now consider changing the way HUP works. I would like to model it in a way that closes files and clears caches (where useful), but not do a full restart. Initially, I thought to use a special signal (e.g. USR2) for this purpose. But still logrotate, as a well-known example, would cause a daemon restart where none is needed. So I now think it would be best to change HUP in the way described above. Those that need to apply a config change would then need to do a regular daemon restart - the same thing that is done with other daemons. In order to keep compatible, I would add an $HUPisRestart option, by default off, which could be used to re-enable previous behavior. I would appreciate feedback on this plan. Are there any concerns? Is this a good idea or a bad one? Does it break something I have not yet mentioned? Thanks, Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 11:42:36 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 11:42:36 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> RG> [...] RG> RG> Initially, I thought to use a special signal (e.g. USR2) for this RG> purpose. But still logrotate, as a well-known example, would cause a RG> daemon restart where none is needed. So I now think it would be best to RG> change HUP in the way described above. Those that need to apply a config RG> change would then need to do a regular daemon restart - the same thing RG> that is done with other daemons. In order to keep compatible, I would RG> add an $HUPisRestart option, by default off, which could be used to RG> re-enable previous behavior. RG> Still, I'd like to have a way to force full restart without really tearing down and reloading the server. What about (optionally?) binding current HUP behaviour to USR2 adding a default on option like $USR2isRestart? Cheers, and a nice forthweek to all, yours lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 11:44:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:44:25 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Thanks for the feedback, but please let me ask one question: On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: > Still, I'd like to have a way to force full restart without really tearing > down and reloading the server. What about (optionally?) binding current > HUP behaviour to USR2 adding a default on option like $USR2isRestart? Why is this useful? I am trying to understand the use cases and why a "normal" restart is not desirable. Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 13:33:05 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 13:33:05 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> Thanks for the feedback, but please let me ask one question: RG> RG> On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: RG> RG> > Still, I'd like to have a way to force full restart without really tearing RG> > down and reloading the server. What about (optionally?) binding current RG> > HUP behaviour to USR2 adding a default on option like $USR2isRestart? RG> RG> Why is this useful? I am trying to understand the use cases and why a RG> "normal" restart is not desirable. RG> Habits, pid stability, monitoring systems, one less thing to break... etc. Yours, lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 13:56:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 13:56:50 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Lorenzo, > Habits, pid stability, monitoring systems, one less thing to break... I understand habits and pid stability. What do you mean by "monitoring system" and "one less thing to break"? Anyone else with an opinion? Rainer From david at ecker-software.de Mon Oct 6 14:41:25 2008 From: david at ecker-software.de (David Ecker) Date: Mon, 06 Oct 2008 14:41:25 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EA0775.7050105@ecker-software.de> Hi, 3.21.5 created an assertion error. rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void *)0)' failed. close to the end. Propably short before the abort signal. I mounted /rsyslog (ext3) with the option : sync Here is a copy of my rsyslog.conf: #--------------------------------------- $ModLoad imuxsock.so $ModLoad imklog.so $WorkDirectory /rsyslog $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 1m $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 #--------------------------------------- I attached the output from" rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 Actually only one messagefile was written, no .qi file was created. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My >>> guess is, that the O_DIRECT in combination with the O_SYNC flag (turning >>> of cache) will have an impact. >>> >> O_DIRECT is doing very different things. I don't think you need to worry >> about those things, having the data not go into the OS cache is a drawback >> not an advantage becouse it means that when you go to pull the data back >> out of the file it will need to actually touch disk. it also imposes >> significant alignment issues on the application that I don't think you >> want to have to desl with. >> > > plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: error.txt URL: From rgerhards at hq.adiscon.com Mon Oct 6 14:43:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 14:43:12 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EA0775.7050105@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Please use the version from git. I didn't realize that the bug affects normal operations, but obviously it does. This is fixed and I'll see that I release 3.21.6 ASAP, but I am not sure if I manage to do this today. Gitweb available at http://git.adiscon.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Monday, October 06, 2008 2:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi, > > 3.21.5 created an assertion error. > > rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void > *)0)' failed. > > close to the end. Propably short before the abort signal. > > I mounted /rsyslog (ext3) with the option : sync > > Here is a copy of my rsyslog.conf: > #--------------------------------------- > $ModLoad imuxsock.so > $ModLoad imklog.so > > $WorkDirectory /rsyslog > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 1m > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > #--------------------------------------- > > I attached the output from" > > rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 > > Actually only one messagefile was written, no .qi file was created. > > bye > David Ecker > > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Already did both with 2.18.3 but'll try again with 3.21.5 and > 3.18.4. My > >>> guess is, that the O_DIRECT in combination with the O_SYNC flag > (turning > >>> of cache) will have an impact. > >>> > >> O_DIRECT is doing very different things. I don't think you need to > worry > >> about those things, having the data not go into the OS cache is a > drawback > >> not an advantage becouse it means that when you go to pull the data > back > >> out of the file it will need to actually touch disk. it also imposes > >> significant alignment issues on the application that I don't think > you > >> want to have to desl with. > >> > > > > plus rsyslog does not care about the alignment (at this time), so I > think it is dangerous... > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From hks.private at gmail.com Mon Oct 6 16:49:26 2008 From: hks.private at gmail.com ((private) HKS) Date: Mon, 6 Oct 2008 10:49:26 -0400 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: In general, I understand HUP to be exactly what you're talking - a reload rather than a restart. I see little to lose by enforcing that in rsyslogd. The pid stability argument has no relevance for me or the tools I use since I just pull it out of pidfiles anyway. -HKS On Mon, Oct 6, 2008 at 7:56 AM, Rainer Gerhards wrote: > Lorenzo, > >> Habits, pid stability, monitoring systems, one less thing to break... > > I understand habits and pid stability. What do you mean by "monitoring > system" and "one less thing to break"? > > Anyone else with an opinion? > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 6 17:08:07 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:08:07 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > Hi List, > > I again have a backwards compatibility question. > > HUP, in sysklogd, means a full restart of the syslogd. This is done > because the configuration may have been changed and to apply the new > one, the previous one must be shut down. This is even more so the case > in rsyslog, which has multiple dynamically loadable plugins and all > that... > > rsyslog has implemented HUP processing to be compatible to sysklogd. > > However, in practice (as far as I know) HUP is almost always being used > to close the output files after log rotation. An automatted config > reaload indication usually does not happen (but operators use HUP after > doing a config change). > > HUP, as it currently is, is an extremely expensive operation. Keep in > mind that it is a full daemon restart, so, among other things, it > unloads modules and tears down (TCP, RELP, TLS, ...) connections. In > most cases, though, this would not be needed. > > I now consider changing the way HUP works. I would like to model it in a > way that closes files and clears caches (where useful), but not do a > full restart. > > Initially, I thought to use a special signal (e.g. USR2) for this > purpose. But still logrotate, as a well-known example, would cause a > daemon restart where none is needed. So I now think it would be best to > change HUP in the way described above. Those that need to apply a config > change would then need to do a regular daemon restart - the same thing > that is done with other daemons. In order to keep compatible, I would > add an $HUPisRestart option, by default off, which could be used to > re-enable previous behavior. > > I would appreciate feedback on this plan. Are there any concerns? Is > this a good idea or a bad one? Does it break something I have not yet > mentioned? could you do something along the lines of checking to see if the config file changed, and if so do a full restart, otherwise just the flush/close? if you know when you started you may be able to just check the last-modified time of the config file and do a restart if it's been modified after you started. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:08:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:08:08 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> > could you do something along the lines of checking to see if the config > file changed, and if so do a full restart, otherwise just the > flush/close? > > if you know when you started you may be able to just check the > last-modified time of the config file and do a restart if it's been > modified after you started. That's a very interesting idea. Do you think this criterion is sufficient? Rainer From david at lang.hm Mon Oct 6 17:15:40 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:15:40 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> could you do something along the lines of checking to see if the > config >> file changed, and if so do a full restart, otherwise just the >> flush/close? >> >> if you know when you started you may be able to just check the >> last-modified time of the config file and do a restart if it's been >> modified after you started. > > That's a very interesting idea. Do you think this criterion is > sufficient? I was thinking of other possible items (doing a checksum of the file, doing a syntactic check to see if the new is equivalent to the old, etc) and I think that they don't end up adding noticable value the only way the last-modified check would fail is if your clock goes backwards (and goes backwards far enough that the 'new' file is modified before you last started). two ways to deal with that 1. store the last-modified time of the file when you start up and re-read it if it has changed (still vunerable if the clock moved back to that exact second, but a very small windows) 2. document the issue and tell people that if their clock goes backwards and the HUP doesn't re-read the config file do a 'touch' of the file and try again. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:19:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:19:23 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AC@grfint2.intern.adiscon.com> This sounds very reasonable. Thanks for the good suggestion! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, October 06, 2008 5:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > >> could you do something along the lines of checking to see if the > > config > >> file changed, and if so do a full restart, otherwise just the > >> flush/close? > >> > >> if you know when you started you may be able to just check the > >> last-modified time of the config file and do a restart if it's been > >> modified after you started. > > > > That's a very interesting idea. Do you think this criterion is > > sufficient? > > I was thinking of other possible items (doing a checksum of the file, > doing a syntactic check to see if the new is equivalent to the old, > etc) > and I think that they don't end up adding noticable value > > the only way the last-modified check would fail is if your clock goes > backwards (and goes backwards far enough that the 'new' file is > modified > before you last started). two ways to deal with that > > 1. store the last-modified time of the file when you start up and re- > read > it if it has changed (still vunerable if the clock moved back to that > exact second, but a very small windows) > > 2. document the issue and tell people that if their clock goes > backwards > and the HUP doesn't re-read the config file do a 'touch' of the file > and > try again. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Mon Oct 6 17:45:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 17:45:56 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 : > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >>> could you do something along the lines of checking to see if the >> config >>> file changed, and if so do a full restart, otherwise just the >>> flush/close? >>> >>> if you know when you started you may be able to just check the >>> last-modified time of the config file and do a restart if it's been >>> modified after you started. >> >> That's a very interesting idea. Do you think this criterion is >> sufficient? Don't forget $IncludeConfig Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 17:47:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:47:46 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Args... thanks Michael, this makes life a little less easy ;) So we do not longer have a clear indication. Would it make sense to require that the main config file is touched when something in the includes is changed? This could be documented. And a restart of course pulls everything. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Monday, October 06, 2008 5:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > 2008/10/6 : > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > >>> could you do something along the lines of checking to see if the > >> config > >>> file changed, and if so do a full restart, otherwise just the > >>> flush/close? > >>> > >>> if you know when you started you may be able to just check the > >>> last-modified time of the config file and do a restart if it's been > >>> modified after you started. > >> > >> That's a very interesting idea. Do you think this criterion is > >> sufficient? > > Don't forget $IncludeConfig > > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Oct 6 18:29:03 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 6 Oct 2008 10:29:03 -0600 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> On Mon, Oct 6, 2008 at 08:49, (private) HKS wrote: > In general, I understand HUP to be exactly what you're talking - a > reload rather than a restart. I see little to lose by enforcing that > in rsyslogd. Ditto - IMHO SIGHUP has come to signify 'change what you can without restarting' or a "hot restart". Some daemons are capable of fully re-configuring on the fly, i.e. "warm restart". It would be rather complex but one could mark particular configuration pragmas as requiring a full tear-down and issue a warning if a warm restart was issued and those had changed. The third option (which many poorly-written init scripts do anyway) is a "cold restart", where the process is killed off and manually re-started, allowing full-blown reconfiguration. I don't care what signals are used for what, beyond the [de facto?] standard of 'skip a beat' when given SIGHUP. Beyond the scope of this discussion, I've not looked at your configuration parsing but wonder if you're not parsing it into an internal, static structure and handling bits from there. If you were, it would be trivial to parse your new configuration, see how/whether it differs from the running one, and shift over if able - like failsafe firmware in embedded systems. That would also offer interesting implications for alternative configuration syntaxes. RB From rgerhards at hq.adiscon.com Mon Oct 6 18:34:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:34:12 +0200 Subject: [rsyslog] HUP action In-Reply-To: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> Message-ID: <1223310852.2682.566.camel@rgf9dev.intern.adiscon.com> On Mon, 2008-10-06 at 10:29 -0600, RB wrote: > Beyond the scope of this discussion, I've not looked at your > configuration parsing but wonder if you're not parsing it into an > internal, static structure and handling bits from there. If you were, > it would be trivial to parse your new configuration, see how/whether > it differs from the running one, and shift over if able - like > failsafe firmware in embedded systems. That would also offer > interesting implications for alternative configuration syntaxes. That is quite complex, because the configuration defines what can be configured. With the plug-in architecture, we do not have any fixed memory structures. So, we would need to spawn a new process, load plugins there, check what in-memory structures they create and, if they differ, than use this new configuration instead of the old one. Rainer From mbiebl at gmail.com Mon Oct 6 18:37:31 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 18:37:31 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 Rainer Gerhards : > Args... thanks Michael, this makes life a little less easy ;) So we do > not longer have a clear indication. > > Would it make sense to require that the main config file is touched when > something in the includes is changed? This could be documented. And a > restart of course pulls everything. I don't think that would be such a good idea. It is not very intuitive and people are not used to such a behaviour. They (at least I do) expect "reload" to reload the complete configuration. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 18:46:16 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:46:16 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> So let me try to sum up where we are now: - it is considered useful to have a full daemon restart be done via HUP - we can not sufficiently simply detect whether the configuration has changed or not - there are situations where it is useful to have the ability to just close files, clear caches etc - people do not like existing things be used in new ways (least surprise principle) So I conclude: - SIGHUP, as ugly as it is, must stay with existing semantics - a new signal can be defined to just do file closure etc Unfortunately, this means that most systems will still use the terribly expensive during e.g. logrotation. However, this is considered acceptable because a) it always was this way, b) a higher demand environment then has options to avoid that. Over time, package maintainers my get maintainers of logrotate involved to change the HUP to the new signal. Am I grasping this right? Rainer On Mon, 2008-10-06 at 18:37 +0200, Michael Biebl wrote: > 2008/10/6 Rainer Gerhards : > > Args... thanks Michael, this makes life a little less easy ;) So we do > > not longer have a clear indication. > > > > Would it make sense to require that the main config file is touched when > > something in the includes is changed? This could be documented. And a > > restart of course pulls everything. > > I don't think that would be such a good idea. > > It is not very intuitive and people are not used to such a behaviour. > They (at least I do) expect "reload" to reload the complete configuration. > > > Michael > From linggao at us.ibm.com Mon Oct 6 20:44:40 2008 From: linggao at us.ibm.com (Ling Gao) Date: Mon, 6 Oct 2008 14:44:40 -0400 Subject: [rsyslog] log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From dt.abernathy at gmail.com Tue Oct 7 18:00:24 2008 From: dt.abernathy at gmail.com (Dan Abernathy) Date: Tue, 7 Oct 2008 10:00:24 -0600 Subject: [rsyslog] FreeBSD and rsyslog Message-ID: Hi, I've been working on integrating rsyslog into a FreeBSD environment, and the following issues and questions have come up along the way: 1. The security facility isn't deprecated in FreeBSD, and I've noticed that this is aliased to the auth facility in rsyslog. I have make a quick patch for this on FreeBSD in order to log to the security facility again, and added the ntp facility (also supported). These changes may be more suitable for the FreeBSD ports tree. 2. The other FreeBSD facility in use is console, which logs all writes to /dev/console to the console log facility. I'm not sure how important this feature is, and it's certainly something I can live without, but it doesn't seem supported in rsyslog. 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. I was curious about support for /var/run/logpriv, however I can't find a great deal of information about how and where this socket is used - other than for privileged applications. 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP address is probably at the top of my list right now. Is there any feature to achieve this that I'm missing? Any thoughts, suggestions and discussion welcome. Dan. From hks.private at gmail.com Tue Oct 7 18:38:16 2008 From: hks.private at gmail.com ((private) HKS) Date: Tue, 7 Oct 2008 12:38:16 -0400 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: On Tue, Oct 7, 2008 at 12:00 PM, Dan Abernathy wrote: > Hi, > > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: > > 1. The security facility isn't deprecated in FreeBSD, and I've noticed that > this is aliased to the auth facility in rsyslog. I have make a quick patch > for this on FreeBSD in order to log to the security facility again, and > added the ntp facility (also supported). These changes may be more suitable > for the FreeBSD ports tree. Excellent, that's good news. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. > > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Can't answer either of these. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? I assume you're running rsyslogd v3. If you're using v2, check the man pages (they're still relevant at that point). When you create the UDP/TCP server in the rsyslog.conf file, you give it the address and port you want it to bind to: $UDPServerAddress 10.123.0.19 $UDPServerRun 514 Hope that helps. -HKS From aoz.syn at gmail.com Tue Oct 7 18:53:05 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 7 Oct 2008 10:53:05 -0600 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: <4255c2570810070953j7d489dfcre45b6515d9d68b5b@mail.gmail.com> > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: You're not alone, I've just gotten distracted along the way and not chased down any issues. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. Curious, I'd not carefully looked at that - it's a character device. No idea. > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Should be addressable with imuxsock and permissions; not much to that. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? Configuration pragma $UDPServerAddress From linggao at us.ibm.com Tue Oct 7 19:39:09 2008 From: linggao at us.ibm.com (Ling Gao) Date: Tue, 7 Oct 2008 13:39:09 -0400 Subject: [rsyslog] Fw: log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From rgerhards at hq.adiscon.com Wed Oct 8 11:04:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 11:04:26 +0200 Subject: [rsyslog] my responsiveness Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D4@grfint2.intern.adiscon.com> Hi folks, unfortunately, I am currently a bit unresponsive to mailing list posts. The reason is that I am reviewing the threading model, which is quite complex and does not play well with any distraction. Also, there is some additional (paid) work to be done and this obviously has some priority. Thanks to all who contribute their knowledge. I try my best to catch up with the posts. I think the situation will become much better in a few days at latest. Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:42:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:42:27 +0200 Subject: [rsyslog] Fw: log forwarding in version 3 In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D9@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ling Gao > Sent: Tuesday, October 07, 2008 7:39 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Fw: log forwarding in version 3 > > > > Hi, > We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. > We'd > like to have logs sent from tier3 to tier1 through tier2. (tier3- > >tier2-> > tier1). > > In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to > enable them receiving remote logs. > $ModLoad imudp.so > $UDPServerRun 514 > > In tier2, we have set up the following in etc/rsyslog.conf to send the > logs > to tier1: > *.* @tier1 > > In tier3, we have set up the following in /etc/rsyslog.conf to send the > logs to tier2 : > *.* @tier2 > > Question: > 1. What other setting we need to enable in tier2 so that the logs from > tier3 can be forwarded to tier1 via tier2? (something like -h option > in > lower versions). None (but keep the firewalls in mind!) > 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some > are > Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using > rsyslog > v3, do we need to change all to the compatibility mode? The compatibility mode affects the way the config file and command line options are interpreted. It does not change anything other than those settings. Review the compatibility guide if there are some problems with any of your settings (my guess is "no") and if so, adjust them. It is recommended to run each instance in native mode. Rainer From r.bhatia at ipax.at Wed Oct 8 12:49:05 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 08 Oct 2008 12:49:05 +0200 Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EC9021.1000701@ipax.at> Rainer Gerhards wrote: > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not no knowing the code, i suggest md5/sha1/... checksum the whole file after integrating everything including rsyslog.d/* into one flat file. > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) i, currently, do not care that much about this issue - at least regarding the reload vs. restart issue. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 8 12:56:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:56:11 +0200 Subject: [rsyslog] HUP action In-Reply-To: <48EC9021.1000701@ipax.at> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <48EC9021.1000701@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 08, 2008 12:49 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > Rainer Gerhards wrote: > > - it is considered useful to have a full daemon restart be done via > HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > no knowing the code, i suggest md5/sha1/... checksum the whole file > after integrating everything including rsyslog.d/* into one flat > file. Quite honestly, I do not like the extra effort. I now have to define two signals in any case, so let the user use the right one for what he intends to do ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:58:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:58:26 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com><48EC9021.1000701@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DB@grfint2.intern.adiscon.com> > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > > Sent: Wednesday, October 08, 2008 12:49 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP action > > > > Rainer Gerhards wrote: > > > - it is considered useful to have a full daemon restart be done via > > HUP > > > - we can not sufficiently simply detect whether the configuration > > > has changed or not > > > > no knowing the code, i suggest md5/sha1/... checksum the whole file > > after integrating everything including rsyslog.d/* into one flat > > file. > > Quite honestly, I do not like the extra effort. I now have to define > two > signals in any case, so let the user use the right one for what he > intends to do ;) Sorry, I was a bit brief: doing the checksum requires interpretation of the config files, at least as far as the $IncludeConfig statements are concerned. This is costly, as is the whole gathering process to build the checksum. So to be on the efficient side, I need to have a dedicated HUP for "non-reload" in any case. Thus I think there is not much added benefit in doing the checksumming. Rainer From david at ecker-software.de Thu Oct 9 10:53:02 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 10:53:02 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Message-ID: <48EDC66E.1060202@ecker-software.de> Hi Rainer, the assert error seemed to be fixed in HEAD. Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to work a lot better. /etc/rsyslog.conf ---------------- $WorkDirectory /rsyslog/ $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 100m $ActionQueueSize 10000000 $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 ----------------- I found out that the queue was limited to 1000 elements if you do not define ActionQueueSize inside the config. One difference was that version 3.18.3 actually created one file per msg if I did set ActionQueueCheckpointInterval to 1. Right now only one file in addition to the .qi file is created containing all messages. I haven't waited to see what will happen if I reach 100MB on the data file. If I shutdown rsyslog normally some status information seems to be written to the .qi file (508 bytes -> 1024 bytes). After restarting the qi file actually shrinks to 508 bytes again. If I kill rsyslogd (SIGKILL) the qi file is not updated with this information (as expected). But it looks like the queue is still working correctly after restarting rsyslogd at least it doesn't invalidate the queue or loosing messages. Turning the system off immediatly seems to be working most of the times. After 11 tries the last one failed. It actually looked like that I turned the system off during a write. bye David Ecker Rainer Gerhards schrieb: > Please use the version from git. I didn't realize that the bug affects > normal operations, but obviously it does. This is fixed and I'll see > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > today. > > Gitweb available at http://git.adiscon.com > > Rainer > From rgerhards at hq.adiscon.com Thu Oct 9 13:56:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 09 Oct 2008 13:56:14 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDC66E.1060202@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> Message-ID: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Hi David, On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > Hi Rainer, > > the assert error seemed to be fixed in HEAD. It is good to hear this. I begun to have some doubts when I reviewed the code. I would really appreciate if you could download and test this version here: http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz I will probably release that tomorrow and so some indication if the problem is actually gone would be very good. The final 3.21.6 will see one more patch, but nothing that affects the assert in question. > > Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to > work a lot better. > > /etc/rsyslog.conf > ---------------- > $WorkDirectory /rsyslog/ > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 100m > $ActionQueueSize 10000000 > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > ----------------- > > I found out that the queue was limited to 1000 elements if you do not > define ActionQueueSize inside the config. Yes, that's the default for *action* queues (the main message queue default is different, I think 10,000). > One difference was that version 3.18.3 actually created one file per msg > if I did set ActionQueueCheckpointInterval to 1. Right now only one > file in addition to the .qi file is created containing all messages. I > haven't waited to see what will happen if I reach 100MB on the data file. That's probably a result of a bug fixed in the repo but not yet released. But I don't check if it is gone now. 3.18.x will only see one more release, then we move on to 3.20.x. > If I shutdown rsyslog normally some status information seems to be > written to the .qi file (508 bytes -> 1024 bytes). After restarting the > qi file actually shrinks to 508 bytes again. That's right. > > If I kill rsyslogd (SIGKILL) the qi file is not updated with this > information (as expected). But it looks like the queue is still working > correctly after restarting rsyslogd at least it doesn't invalidate the > queue or loosing messages. I'd say it depends on when exactly it is doing when it is being killed. > > Turning the system off immediatly seems to be working most of the times. > After 11 tries the last one failed. It actually looked like that I > turned the system off during a write. This is also within what I expect. If you hit it during a write, things are really bad. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Please use the version from git. I didn't realize that the bug affects > > normal operations, but obviously it does. This is fixed and I'll see > > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > > today. > > > > Gitweb available at http://git.adiscon.com > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Thu Oct 9 14:43:30 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 14:43:30 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EDFC72.2070404@ecker-software.de> Hi Rainer, with the same config it behaves similar to the HEAD version I tested in the last few days. The assert message did not appear. bye David Ecker Rainer Gerhards schrieb: > Hi David, > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > >> Hi Rainer, >> >> the assert error seemed to be fixed in HEAD. >> > > It is good to hear this. I begun to have some doubts when I reviewed the > code. I would really appreciate if you could download and test this > version here: > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > I will probably release that tomorrow and so some indication if the > problem is actually gone would be very good. The final 3.21.6 will see > one more patch, but nothing that affects the assert in question. > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to >> work a lot better. >> >> /etc/rsyslog.conf >> ---------------- >> $WorkDirectory /rsyslog/ >> $ActionQueueFileName buffer >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionQueueType Disk >> $ActionQueueMaxFileSize 100m >> $ActionQueueSize 10000000 >> $ActionQueueCheckpointInterval 1 >> $ActionResumeRetryCount -1 >> *.* @@10.8.0.1:514 >> ----------------- >> >> I found out that the queue was limited to 1000 elements if you do not >> define ActionQueueSize inside the config. >> > > Yes, that's the default for *action* queues (the main message queue > default is different, I think 10,000). > > >> One difference was that version 3.18.3 actually created one file per msg >> if I did set ActionQueueCheckpointInterval to 1. Right now only one >> file in addition to the .qi file is created containing all messages. I >> haven't waited to see what will happen if I reach 100MB on the data file. >> > > That's probably a result of a bug fixed in the repo but not yet > released. But I don't check if it is gone now. 3.18.x will only see one > more release, then we move on to 3.20.x. > > >> If I shutdown rsyslog normally some status information seems to be >> written to the .qi file (508 bytes -> 1024 bytes). After restarting the >> qi file actually shrinks to 508 bytes again. >> > > That's right. > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this >> information (as expected). But it looks like the queue is still working >> correctly after restarting rsyslogd at least it doesn't invalidate the >> queue or loosing messages. >> > > I'd say it depends on when exactly it is doing when it is being killed. > > >> Turning the system off immediatly seems to be working most of the times. >> After 11 tries the last one failed. It actually looked like that I >> turned the system off during a write. >> > > This is also within what I expect. If you hit it during a write, things > are really bad. > > >> bye >> David Ecker >> >> Rainer Gerhards schrieb: >> >>> Please use the version from git. I didn't realize that the bug affects >>> normal operations, but obviously it does. This is fixed and I'll see >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do this >>> today. >>> >>> Gitweb available at http://git.adiscon.com >>> >>> Rainer >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 9 15:31:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Oct 2008 15:31:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDFC72.2070404@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de><1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> <48EDFC72.2070404@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2F3@grfint2.intern.adiscon.com> Hi David, thanks for the quick feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Thursday, October 09, 2008 2:44 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi Rainer, > > with the same config it behaves similar to the HEAD version I tested in > the last few days. The assert message did not appear. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Hi David, > > > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > > > >> Hi Rainer, > >> > >> the assert error seemed to be fixed in HEAD. > >> > > > > It is good to hear this. I begun to have some doubts when I reviewed > the > > code. I would really appreciate if you could download and test this > > version here: > > > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > > > I will probably release that tomorrow and so some indication if the > > problem is actually gone would be very good. The final 3.21.6 will > see > > one more patch, but nothing that affects the assert in question. > > > > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed > to > >> work a lot better. > >> > >> /etc/rsyslog.conf > >> ---------------- > >> $WorkDirectory /rsyslog/ > >> $ActionQueueFileName buffer > >> $ActionQueueMaxDiskSpace 1g > >> $ActionQueueSaveOnShutdown on > >> $ActionQueueType Disk > >> $ActionQueueMaxFileSize 100m > >> $ActionQueueSize 10000000 > >> $ActionQueueCheckpointInterval 1 > >> $ActionResumeRetryCount -1 > >> *.* @@10.8.0.1:514 > >> ----------------- > >> > >> I found out that the queue was limited to 1000 elements if you do > not > >> define ActionQueueSize inside the config. > >> > > > > Yes, that's the default for *action* queues (the main message queue > > default is different, I think 10,000). > > > > > >> One difference was that version 3.18.3 actually created one file per > msg > >> if I did set ActionQueueCheckpointInterval to 1. Right now only one > >> file in addition to the .qi file is created containing all messages. > I > >> haven't waited to see what will happen if I reach 100MB on the data > file. > >> > > > > That's probably a result of a bug fixed in the repo but not yet > > released. But I don't check if it is gone now. 3.18.x will only see > one > > more release, then we move on to 3.20.x. > > > > > >> If I shutdown rsyslog normally some status information seems to be > >> written to the .qi file (508 bytes -> 1024 bytes). After restarting > the > >> qi file actually shrinks to 508 bytes again. > >> > > > > That's right. > > > > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this > >> information (as expected). But it looks like the queue is still > working > >> correctly after restarting rsyslogd at least it doesn't invalidate > the > >> queue or loosing messages. > >> > > > > I'd say it depends on when exactly it is doing when it is being > killed. > > > > > >> Turning the system off immediatly seems to be working most of the > times. > >> After 11 tries the last one failed. It actually looked like that I > >> turned the system off during a write. > >> > > > > This is also within what I expect. If you hit it during a write, > things > > are really bad. > > > > > >> bye > >> David Ecker > >> > >> Rainer Gerhards schrieb: > >> > >>> Please use the version from git. I didn't realize that the bug > affects > >>> normal operations, but obviously it does. This is fixed and I'll > see > >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do > this > >>> today. > >>> > >>> Gitweb available at http://git.adiscon.com > >>> > >>> Rainer > >>> > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From friedl at hq.adiscon.com Thu Oct 9 17:32:51 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 9 Oct 2008 17:32:51 +0200 Subject: [rsyslog] rsyslog 3.18.5 (v3-stable) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2FD@grfint2.intern.adiscon.com> Hi all, we have just released rsyslog 3.18.5, a member of the v3-stable branch. This is most probably the last 3.18.x version, the current beta will soon become the new stable. Rsyslog 3.18.5 is a bug fixing release with some enhanced documentation. The most important bug fixes addresses some issues during HUP, which could lead to a segfault. This is a recommended update for all users of the v3-stable branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-133.phtml Changelog: http://www.rsyslog.com/Article281.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From kyuhwanzz at gmail.com Wed Oct 15 14:03:05 2008 From: kyuhwanzz at gmail.com (Kyu-Hwan Yun) Date: Wed, 15 Oct 2008 21:03:05 +0900 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? Message-ID: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Hi, all. My question is, How cound rsyslog recognize that remote server is alive? port open/close, network interface up/down, or etc... Which method is the answer? From rgerhards at hq.adiscon.com Wed Oct 15 15:08:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Oct 2008 15:08:43 +0200 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? In-Reply-To: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> References: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F378@grfint2.intern.adiscon.com> The simple answer is that we try a send() call and if it succeeds, that's a good indication the other side is alive. If it fails, the retry logic tries to restore the connection. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Kyu-Hwan Yun > Sent: Wednesday, October 15, 2008 2:03 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How can the rsyslog recognize remote server is > alive? > > Hi, all. > > My question is, > > How cound rsyslog recognize that remote server is alive? > > port open/close, network interface up/down, or etc... > > Which method is the answer? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Thu Oct 16 15:11:36 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 16 Oct 2008 15:11:36 +0200 Subject: [rsyslog] rsyslog 3.19.12 (beta) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F39F@grfint2.intern.adiscon.com> Hi all, rsyslog 3.12.19 has been released, a member of the beta branch. It is mainly a refresh of the current beta which provides all those bug-fixes that have been made to the stable releases. Other than that, it includes some minor documentation updates and a bugfix for a beta-specific issue. This is scheduled to become the new v3-stable soon. It is a recommended update for all beta branch users. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-134.phtml Changelog: http://www.rsyslog.com/Article283.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Oct 18 06:40:22 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 17 Oct 2008 21:40:22 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > So let me try to sum up where we are now: > > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) > > So I conclude: > > - SIGHUP, as ugly as it is, must stay with existing semantics > - a new signal can be defined to just do file closure etc > > Unfortunately, this means that most systems will still use the terribly > expensive during e.g. logrotation. However, this is considered > acceptable because a) it always was this way, b) a higher demand > environment then has options to avoid that. Over time, package > maintainers my get maintainers of logrotate involved to change the HUP > to the new signal. > > Am I grasping this right? I learned this week that on a HUP rsyslog will loose any messages that it has in it's memory queue (due to the fact that it basicly does a full shutdown and restart). I suspect that many other people would be surprised at this as well. It looks like we have two options. 1. HUP will notice config file changes, but can loose queued logs 2. HUP will not notice config file changes, but will not loose queued logs I think that there would be less surprise in the second case. all the scripts that use HUP to roll logs don't really care about the config files I guess there is a third case, make a HUP not drop the queue, but if the config file options for the queue change significantly, this can be a very difficult thing to do David Lang From rgerhards at hq.adiscon.com Sat Oct 18 10:32:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 18 Oct 2008 10:32:21 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > So let me try to sum up where we are now: > > > > - it is considered useful to have a full daemon restart be done via HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > - there are situations where it is useful to have the ability to > > just close files, clear caches etc > > - people do not like existing things be used in new ways > > (least surprise principle) > > > > So I conclude: > > > > - SIGHUP, as ugly as it is, must stay with existing semantics > > - a new signal can be defined to just do file closure etc > > > > Unfortunately, this means that most systems will still use the terribly > > expensive during e.g. logrotation. However, this is considered > > acceptable because a) it always was this way, b) a higher demand > > environment then has options to avoid that. Over time, package > > maintainers my get maintainers of logrotate involved to change the HUP > > to the new signal. > > > > Am I grasping this right? > > I learned this week that on a HUP rsyslog will loose any messages that it > has in it's memory queue (due to the fact that it basicly does a full > shutdown and restart). I suspect that many other people would be surprised > at this as well. Well, actually it depends. In the default configuration, this is true. But the spirit of the default configuration (suitable for most simple cases) is "I don't care about message loss". If you care about message loss, queues need to be configured so that no already-gathered messages are lost between sessions. If done so, messages are written to files before a shutdown and read from it upon restart. Of course, messages are still lost while the system is doing the restart (new ones). For a HUP and a very busy system, this can mean it takes a couple of seconds, even minutes, to do a full restart. After I noticed that the "message loss on HUP if not configured to avoid it" is not obvious, I added some explicit doc to the man pages so that people are warned they do a full restart. > > It looks like we have two options. > > 1. HUP will notice config file changes, but can loose queued logs > > 2. HUP will not notice config file changes, but will not loose queued logs > > I think that there would be less surprise in the second case. see comments above > > all the scripts that use HUP to roll logs don't really care about the > config files that was my initial observation. I will define another signal to just close the file. I hope I can get this done next week. Unfortunately, the financial crisis begins to hit the real world and this causes me to currently spent less time on new additions to rsyslog and more on profit-generating projects, but that should be doable in a day or two (the output module interface needs to be extend, a new administrative linked list implemented and the signal handler be done - changes to outputs are also necessary). > > I guess there is a third case, make a HUP not drop the queue, but if the > config file options for the queue change significantly, this can be a very > difficult thing to do This is extremely difficult, but of course doable. I think, however, this is only needed for a very limited number of cases. So this is probably something that I will only work on if a sponsor for this work shows up. Rainer > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sat Oct 18 11:32:32 2008 From: david at lang.hm (david at lang.hm) Date: Sat, 18 Oct 2008 02:32:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: On Sat, 18 Oct 2008, Rainer Gerhards wrote: > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> >>> So let me try to sum up where we are now: >>> >>> - it is considered useful to have a full daemon restart be done via HUP >>> - we can not sufficiently simply detect whether the configuration >>> has changed or not >>> - there are situations where it is useful to have the ability to >>> just close files, clear caches etc >>> - people do not like existing things be used in new ways >>> (least surprise principle) >>> >>> So I conclude: >>> >>> - SIGHUP, as ugly as it is, must stay with existing semantics >>> - a new signal can be defined to just do file closure etc >>> >>> Unfortunately, this means that most systems will still use the terribly >>> expensive during e.g. logrotation. However, this is considered >>> acceptable because a) it always was this way, b) a higher demand >>> environment then has options to avoid that. Over time, package >>> maintainers my get maintainers of logrotate involved to change the HUP >>> to the new signal. >>> >>> Am I grasping this right? >> >> I learned this week that on a HUP rsyslog will loose any messages that it >> has in it's memory queue (due to the fact that it basicly does a full >> shutdown and restart). I suspect that many other people would be surprised >> at this as well. > > Well, actually it depends. In the default configuration, this is true. > But the spirit of the default configuration (suitable for most simple > cases) is "I don't care about message loss". there's a big difference between "I don't care about message loss if a system crashed" and "I don't care about message loss at all" admins should evaluate the probability of the different message loss scenerios and decide on the appropriate cost/benifit for their situation. it's very possible that someone may not care about loosing logs when a system crashes (becouse it's a fairly rare event), but isn't willing to loose logs just to rotate log files. > If you care about message > loss, queues need to be configured so that no already-gathered messages > are lost between sessions. If done so, messages are written to files > before a shutdown and read from it upon restart. Of course, messages are > still lost while the system is doing the restart (new ones). > > For a HUP and a very busy system, this can mean it takes a couple of > seconds, even minutes, to do a full restart. and unfortunantly, the process of writing the pending queue to disk and then reading it back after a restart can greatly lengthen the restart, causing incoming logs to be missed (depending on how able the log sender is to queue the log messages for your) > After I noticed that the "message loss on HUP if not configured to avoid > it" is not obvious, I added some explicit doc to the man pages so that > people are warned they do a full restart. thanks. >> >> It looks like we have two options. >> >> 1. HUP will notice config file changes, but can loose queued logs >> >> 2. HUP will not notice config file changes, but will not loose queued logs >> >> I think that there would be less surprise in the second case. > > see comments above the purpose of this message was mostly to try and prompt the other people who commented in this thread to think about the issue and comment on their expectations. >> >> all the scripts that use HUP to roll logs don't really care about the >> config files > > that was my initial observation. I will define another signal to just > close the file. I hope I can get this done next week. Unfortunately, the > financial crisis begins to hit the real world and this causes me to > currently spent less time on new additions to rsyslog and more on > profit-generating projects, but that should be doable in a day or two > (the output module interface needs to be extend, a new administrative > linked list implemented and the signal handler be done - changes to > outputs are also necessary). I am not trying to preasure you into rushing on this, just to get opinions on which is the 'more expected' behavior for HUP >> >> I guess there is a third case, make a HUP not drop the queue, but if the >> config file options for the queue change significantly, this can be a very >> difficult thing to do > > This is extremely difficult, but of course doable. I think, however, > this is only needed for a very limited number of cases. So this is > probably something that I will only work on if a sponsor for this work > shows up. I would definantly not recommend this one becouse of the difficulty involved, but in the interest of completeness I thought it needed to be mentioned. David Lang From rgerhards at hq.adiscon.com Mon Oct 20 09:41:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Oct 2008 09:41:24 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: > On Sat, 18 Oct 2008, Rainer Gerhards wrote: > > > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >> > >>> So let me try to sum up where we are now: > >>> > >>> - it is considered useful to have a full daemon restart be done via HUP > >>> - we can not sufficiently simply detect whether the configuration > >>> has changed or not > >>> - there are situations where it is useful to have the ability to > >>> just close files, clear caches etc > >>> - people do not like existing things be used in new ways > >>> (least surprise principle) > >>> > >>> So I conclude: > >>> > >>> - SIGHUP, as ugly as it is, must stay with existing semantics > >>> - a new signal can be defined to just do file closure etc > >>> > >>> Unfortunately, this means that most systems will still use the terribly > >>> expensive during e.g. logrotation. However, this is considered > >>> acceptable because a) it always was this way, b) a higher demand > >>> environment then has options to avoid that. Over time, package > >>> maintainers my get maintainers of logrotate involved to change the HUP > >>> to the new signal. > >>> > >>> Am I grasping this right? > >> > >> I learned this week that on a HUP rsyslog will loose any messages that it > >> has in it's memory queue (due to the fact that it basicly does a full > >> shutdown and restart). I suspect that many other people would be surprised > >> at this as well. > > > > Well, actually it depends. In the default configuration, this is true. > > But the spirit of the default configuration (suitable for most simple > > cases) is "I don't care about message loss". > > there's a big difference between "I don't care about message loss if a > system crashed" and "I don't care about message loss at all" That's right. But I was not talking about a crash condition. I was talking about a shutdown (or a restart). With the default settings, rsyslog does what sysklogd does, that is (mostly) not care about message loss. If you shut down the daemon, it tries to drain the queue for a short timeout. If all messages can be processed during that time, we are fine and terminate rsyslogd without message loss. However, if there are too many messages queued, the daemon terminates after the timeout has expired and discards whatever messages are left unprocessed. This is done in an effort to keep compatible with sysklogd. Also, in most scenarios it is highly unlikely that message loss will occur, because the queue will not hold extended data for an extended period of time. If, however, this is anticipated, there are ways to configure rsyslog not to loose messages. For example, the timeout can be increased and/or the queue can be made disk-assisted and data be written to disk if they can not be processed during the timeout. During queue drain, no new messages are accepted (otherwise, how should we finally shut down?). Given a busy system, there is unavoidable message loss. Either you use old messages you already had in the queue, but have a quick restart and loose few new messages. Or you save the already existing messages but experience loss of new messages. This is another reason for the default, which I consider the best compromise under the circumstances given. This is also a good indication why you should not unnecessarily restart rsyslgod (or any other syslogd, because this is not a project-specific problem but rather a general one). With fully reliable protocols, you can of course make the sender block while you are shutting down. But all of this requires very careful planning. I assume folks who want to do that either have the necessary knowledge themselvs or hire someone who does. So there are three scenarios: a) hup b) regular shutdown c) system abort I think you did not take into account b), which is a quite common case. Other than that, I fully agree to you post. Rainer From david at lang.hm Mon Oct 20 19:07:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 10:07:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 20 Oct 2008, Rainer Gerhards wrote: > On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: >> On Sat, 18 Oct 2008, Rainer Gerhards wrote: >> >>> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >>>> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> So let me try to sum up where we are now: >>>>> >>>>> - it is considered useful to have a full daemon restart be done via HUP >>>>> - we can not sufficiently simply detect whether the configuration >>>>> has changed or not >>>>> - there are situations where it is useful to have the ability to >>>>> just close files, clear caches etc >>>>> - people do not like existing things be used in new ways >>>>> (least surprise principle) >>>>> >>>>> So I conclude: >>>>> >>>>> - SIGHUP, as ugly as it is, must stay with existing semantics >>>>> - a new signal can be defined to just do file closure etc >>>>> >>>>> Unfortunately, this means that most systems will still use the terribly >>>>> expensive during e.g. logrotation. However, this is considered >>>>> acceptable because a) it always was this way, b) a higher demand >>>>> environment then has options to avoid that. Over time, package >>>>> maintainers my get maintainers of logrotate involved to change the HUP >>>>> to the new signal. >>>>> >>>>> Am I grasping this right? >>>> >>>> I learned this week that on a HUP rsyslog will loose any messages that it >>>> has in it's memory queue (due to the fact that it basicly does a full >>>> shutdown and restart). I suspect that many other people would be surprised >>>> at this as well. >>> >>> Well, actually it depends. In the default configuration, this is true. >>> But the spirit of the default configuration (suitable for most simple >>> cases) is "I don't care about message loss". >> >> there's a big difference between "I don't care about message loss if a >> system crashed" and "I don't care about message loss at all" > > That's right. But I was not talking about a crash condition. I was > talking about a shutdown (or a restart). With the default settings, > rsyslog does what sysklogd does, that is (mostly) not care about message > loss. If you shut down the daemon, it tries to drain the queue for a > short timeout. If all messages can be processed during that time, we are > fine and terminate rsyslogd without message loss. the big difference is that rsysklog does not maintain a queue, so when it shuts down it doesn't loose any messages that it has processed (even with async writes it has passed the logs to the OS and so those logs are not affected by the sysklog shutdown/HUP) > However, if there are too many messages queued, the daemon terminates > after the timeout has expired and discards whatever messages are left > unprocessed. This is done in an effort to keep compatible with sysklogd. > Also, in most scenarios it is highly unlikely that message loss will > occur, because the queue will not hold extended data for an extended > period of time. If, however, this is anticipated, there are ways to > configure rsyslog not to loose messages. For example, the timeout can be > increased and/or the queue can be made disk-assisted and data be written > to disk if they can not be processed during the timeout. > > During queue drain, no new messages are accepted (otherwise, how should > we finally shut down?). Given a busy system, there is unavoidable > message loss. Either you use old messages you already had in the queue, > but have a quick restart and loose few new messages. Or you save the > already existing messages but experience loss of new messages. This is > another reason for the default, which I consider the best compromise > under the circumstances given. This is also a good indication why you > should not unnecessarily restart rsyslgod (or any other syslogd, because > this is not a project-specific problem but rather a general one). this conflicts with the need to rotate the log files. while we have discussed the possibility of using a different signal to allow the logs to be rotated, I was bringing the subject up again becouse I think the HUP behavior is unexpected enough that it may be less surprise if a HUP doesn't re-read config files but does the close/open needed for log rotation than the alturnative of having HUP remain as-in with a different signal for log rotation. I'm not suggesting more work, I'm suggesting that it may make sense to just use HUP for the close/open and if someone needs the full config reload they do a stop/start instead. > With fully reliable protocols, you can of course make the sender block > while you are shutting down. But all of this requires very careful > planning. I assume folks who want to do that either have the necessary > knowledge themselvs or hire someone who does. > > So there are three scenarios: > > a) hup > b) regular shutdown > c) system abort > > I think you did not take into account b), which is a quite common case. there is case d) log rotation when someone does a stop/start (case b) they expect logs in a queue to be lost (you are fully stopping the process after all), I don't think that's what people would expect from a HUP. David Lang > Other than that, I fully agree to you post. > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From pheller at me.com Mon Oct 20 19:32:51 2008 From: pheller at me.com (Phillip Heller) Date: Mon, 20 Oct 2008 13:32:51 -0400 Subject: [rsyslog] Expression based filters not working? Message-ID: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil From mikel at irontec.com Mon Oct 20 21:54:42 2008 From: mikel at irontec.com (Mikel Jimenez) Date: Mon, 20 Oct 2008 21:54:42 +0200 Subject: [rsyslog] db template how to Message-ID: <48FCE202.1070100@irontec.com> Hello How can I make a template of sql in rsyslog.conf that makes that? : hostname1 >database.hostname1 hostname2 >database.hostname2 hostanem3 >database.hostname3 ..... and hostnameaa hostnameab hostnamedd hostnamede hostnamea*>database.a hostnamed*>database.d Thanks From david at lang.hm Mon Oct 20 22:44:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 13:44:32 -0700 (PDT) Subject: [rsyslog] db template how to In-Reply-To: <48FCE202.1070100@irontec.com> References: <48FCE202.1070100@irontec.com> Message-ID: On Mon, 20 Oct 2008, Mikel Jimenez wrote: > Date: Mon, 20 Oct 2008 21:54:42 +0200 > From: Mikel Jimenez > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] db template how to > > Hello > How can I make a template of sql in rsyslog.conf that makes that? : > > hostname1 >database.hostname1 > hostname2 >database.hostname2 > hostanem3 >database.hostname3 > ..... first a disclaimer, I haven't used the database output yet, so I could be wrong or missing something easier that what I'm suggesting. putting them into different databases would be difficult and expensive, but possible. if you are putting them into different tables inside one database you could change from something like "insert into syslog(message) values ('%msg%')" to something like "insert into syslog||'.'||%hostname% (message) values ('%msg%')" since you are issuing SQL commands, you could do something along the lines of the following to do different databases "connect database||'.'||%hostname% ; insert into syslog(message) values ('%msg%'): to switch the the appropriate databases. (note that this is usually _very_ inefficiant to do) I don't know if you can do multiple database output modules going to different systems, but if you can you may be able to do something nicer by defining a different output module for each hostname and use filters to seperate the traffic (this will obviously only work for a limited number of hostnames, and they all have to be known ahead of time) > > and > > hostnameaa > hostnameab > hostnamedd > hostnamede > > hostnamea*>database.a > hostnamed*>database.d this could be similar, but you would have to do string manipulation of the hostname value to extract out just the character you need. David Lang From aoz.syn at gmail.com Mon Oct 20 23:12:23 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 20 Oct 2008 15:12:23 -0600 Subject: [rsyslog] db template how to In-Reply-To: References: <48FCE202.1070100@irontec.com> Message-ID: <4255c2570810201412n2f8addbbqf94b6d78b1d7ccd8@mail.gmail.com> On Mon, Oct 20, 2008 at 14:44, wrote: > putting them into different databases would be difficult and expensive, > but possible. Hopefully we're not talking different databases, per se, maybe different tables? My suggestion wouldn't be to put any of that logic at all in your rsyslog configuration - the application & configurations are complex enough. Rather (if you have a proper database engine) use table partitioning, even constraint-based partitioning to do what you need. Much less horrifying on the client side and any competent DBA will know what you're doing and be able to handle it. Off the top of my head I know Oracle, PostgreSQL, and MySQL support partitioning and I'd be willing to bet MSSQL does as well. Perhaps a more appropriate question would be why? Other than your mental organization and maybe some access-control concerns, why would you want to divide your database by host? Not only does it make global correlation more complex, it's practically guaranteed to have an uneven distribution. When doing log tables, I always arrange the MERGE/partition constraint by date or the like so it's easier and often faster for limiting searches as well as to age data out (DROP TABLE versus DELETE FROM). From DGillies at fairfaxdigital.com.au Tue Oct 21 01:26:20 2008 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 21 Oct 2008 10:26:20 +1100 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Message-ID: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Hi Phil, I found that ! didn't work for me in my expression style configs. I had to use 'not' instead: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then -?DailyLogs David Gillies Systems Engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller Sent: Tuesday, 21 October 2008 4:33 AM To: rsyslog at lists.adiscon.com Subject: [rsyslog] Expression based filters not working? Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From rgerhards at hq.adiscon.com Tue Oct 21 08:32:57 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 21 Oct 2008 08:32:57 +0200 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F3E8@grfint2.intern.adiscon.com> That's right ! is NOT part of the language. You need to use the word "not". Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Gillies > Sent: Tuesday, October 21, 2008 1:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Expression based filters not working? > > Hi Phil, > > I found that ! didn't work for me in my expression style configs. I had > to use 'not' instead: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat > if $msg contains 'VOIPAAA' then -?DailyCDR > if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then > -?DailyLogs > > David Gillies > Systems Engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller > Sent: Tuesday, 21 October 2008 4:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Expression based filters not working? > > Hello, > > Running 3.19.9 with the following old style config bits: > > :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, > "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs > > This works just fine, though the messages from the ASA are output to > the > DailyLogs template twice. I have not seen an example how one might use > a logical AND with the old style config. > > So, I've tried the new expression style config: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains > 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg > !contains '%ASA-' then -?DailyLogs > > However, this style seems to not work at all. That is, no messages are > recorded to DailyLogs or DailyCDR. > > Any suggestions? > > Regards, > > Phil > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended recipient, > any use, dissemination, reliance, forwarding, printing or copying of > this e-mail or any attached files is unauthorised. This e-mail is > subject to copyright. No part of it should be reproduced, adapted or > communicated without the written consent of the copyright owner. If you > have received this e-mail in error please advise the sender immediately > by return e-mail or telephone and delete all copies. Fairfax does not > guarantee the accuracy or completeness of any information contained in > this e-mail or attached files. Internet communications are not secure, > therefore Fairfax does not accept legal responsibility for the contents > of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 22 10:13:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 10:13:23 +0200 Subject: [rsyslog] rsyslog 3.21.6 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F414@grfint2.intern.adiscon.com> Hi all, rsyslog 3.21.6, a member of the development branch, has just been released. Most importantly, it fixes a segfault condition which could occur during processing. It also contains some other bugfixes, many imported from the stable and beta branches. Feature-wise, performance has been improved, permitting rsyslogd to handle an even higher number of messages per second. This is a recommended update for all devel branch users. This release is scheduled to become the new beta soon. Changelog: http://www.rsyslog.com/Article292.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-135.phtml As always, feedback is appreciated. Rainer From epiphani at gmail.com Wed Oct 22 18:05:06 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 12:05:06 -0400 Subject: [rsyslog] output compression... Message-ID: I've been looking for some method to automatically compress output files from a syslog service - rsyslogd seems to be the most flexible in terms of base architecture. Has anyone written a zlib-based (or some gzip implementation) output module for writing files to disk already compressed? Would it be something difficult to add to the roadmap? :) Thanks, -Aaron From aoz.syn at gmail.com Wed Oct 22 18:30:38 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 10:30:38 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <4255c2570810220930h3b63c09awbe6d993da0a5270b@mail.gmail.com> On Wed, Oct 22, 2008 at 10:05, Aaron Wiebe wrote: > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) It's on my personal roadmap, but fell behind circular logs (FBSD clog-compatible) and my screwing around with Windows .evt formats. Don't know if anyone else has looked at it. Even though it should be a well-received addition, the problem is that most users do well enough with logrotate or its ilk to manage log compression and don't count it too high on their priorities. From rgerhards at hq.adiscon.com Wed Oct 22 18:37:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 18:37:03 +0200 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Hi, I have not much to add to RB's remark. I agree to his analysis. I'd like to do it, but always higher priority things come in between... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 22, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] output compression... > > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) > > Thanks, > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Wed Oct 22 20:55:03 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 14:55:03 -0400 Subject: [rsyslog] output compression... In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: Understood - thanks for the response. Can you point me at some docs for your modules API? I may see what I can do on my own here... Thanks, -Aaron On Wed, Oct 22, 2008 at 12:37 PM, Rainer Gerhards wrote: > Hi, > > I have not much to add to RB's remark. I agree to his analysis. I'd like > to do it, but always higher priority things come in between... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Wednesday, October 22, 2008 6:05 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] output compression... >> >> I've been looking for some method to automatically compress output >> files from a syslog service - rsyslogd seems to be the most flexible >> in terms of base architecture. Has anyone written a zlib-based (or >> some gzip implementation) output module for writing files to disk >> already compressed? >> >> Would it be something difficult to add to the roadmap? :) >> >> Thanks, >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 22 21:40:05 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 13:40:05 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> > Can you point me at some docs for your modules API? I may see what I > can do on my own here... I would suggest starting with plugins/omtesting, plugins/imtemplate and tools/omfile.* - those helped me jump in PDQ. The following URLs were also helpful: http://www.rsyslog.com/doc-generic_design.html http://www.rsyslog.com/doc-modules.html From rgerhards at hq.adiscon.com Thu Oct 23 13:23:10 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 23 Oct 2008 13:23:10 +0200 Subject: [rsyslog] output compression... In-Reply-To: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> Message-ID: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-22 at 13:40 -0600, RB wrote: > > Can you point me at some docs for your modules API? I may see what I > > can do on my own here... > > I would suggest starting with plugins/omtesting, plugins/imtemplate > and tools/omfile.* - those helped me jump in PDQ. The following URLs > were also helpful: > > http://www.rsyslog.com/doc-generic_design.html > http://www.rsyslog.com/doc-modules.html That's a good suggestion. In this case, I would also tend to modifying omfile.c directly. This, too me, sounds much like just adding the ability to zip before doing the write() call. If you intend to contribute back the patch, I can lend you a helping hand (as much as time permits, unfortunately) during the process. Rainer From epiphani at gmail.com Thu Oct 23 17:12:49 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Thu, 23 Oct 2008 11:12:49 -0400 Subject: [rsyslog] output compression... In-Reply-To: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> Message-ID: On Thu, Oct 23, 2008 at 7:23 AM, Rainer Gerhards wrote: > > That's a good suggestion. In this case, I would also tend to modifying > omfile.c directly. This, too me, sounds much like just adding the > ability to zip before doing the write() call. Yes, I've read omfile.c - I will likely be following this route. > If you intend to contribute back the patch, I can lend you a helping > hand (as much as time permits, unfortunately) during the process. Unfortunately that is at the whim of my employer. I have a process started to try to clear such things, but for now I will have to work alone. Thanks, -Aaron From david at lang.hm Fri Oct 24 23:47:54 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 24 Oct 2008 14:47:54 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? Message-ID: I'm constructing custom output formats and it looks like things are not gettng parsed as I would expect. the incoming logs look like 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 143 E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 plug-gw[13051]: disconnect host= /192.168.242.12 destination=179.50.100.52/14872 in=1069 out=71 duration=1 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 135 E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p plug-gw[10883]: disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 out=720 duration=1 what is unexpected is that tag is the hostname/IP and the plug-gw is part of the message the hostname field is getting populated with what I would expect to be in fromhost (the relay box that sent the message to me) the syslog daemons sending me the logs have been modified, so there is a possibility that I messed up on them and the format that's being sent isn't right, but if so I'm not seeing anything wrong with it. I am using the nextmaster branch. David Lang From rgerhards at hq.adiscon.com Sun Oct 26 20:41:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 26 Oct 2008 20:41:12 +0100 Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Could you get me the debug output while such a message is being processed? That would probably be useful... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 24, 2008 11:48 PM > To: rsyslog-users > Subject: [rsyslog] parsing error with imudp? > > I'm constructing custom output formats and it looks like > things are not > gettng parsed as I would expect. > > the incoming logs look like > > 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 143 > E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 > plug-gw[13051]: disconnect host= /192.168.242.12 > destination=179.50.100.52/14872 in=1069 out=71 duration=1 > > 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 135 > E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p > plug-gw[10883]: > disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 > out=720 duration=1 > > > what is unexpected is that tag is the hostname/IP and the > plug-gw is part > of the message > > the hostname field is getting populated with what I would > expect to be in > fromhost (the relay box that sent the message to me) > > the syslog daemons sending me the logs have been modified, so > there is a > possibility that I messed up on them and the format that's being sent > isn't right, but if so I'm not seeing anything wrong with it. > > I am using the nextmaster branch. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 27 01:38:44 2008 From: david at lang.hm (david at lang.hm) Date: Sun, 26 Oct 2008 17:38:44 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, Rainer Gerhards wrote: > Could you get me the debug output while such a message is being > processed? That would probably be useful... I'll have to recompile to get a build with all the debug fetures enabled. I will do this tomorrow. I did check rsyslog->rsyslog relay and saw the same thing (actually it was sysklogd -> rsyslog -> rsyslog. the first rsyslog shows the correct thing, the second rsyslog shows the hostname of the first, with the hostname of the sysklog box in the message) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 24, 2008 11:48 PM >> To: rsyslog-users >> Subject: [rsyslog] parsing error with imudp? >> >> I'm constructing custom output formats and it looks like >> things are not >> gettng parsed as I would expect. >> >> the incoming logs look like >> >> 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 143 >> E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 >> plug-gw[13051]: disconnect host= /192.168.242.12 >> destination=179.50.100.52/14872 in=1069 out=71 duration=1 >> >> 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 135 >> E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p >> plug-gw[10883]: >> disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 >> out=720 duration=1 >> >> >> what is unexpected is that tag is the hostname/IP and the >> plug-gw is part >> of the message >> >> the hostname field is getting populated with what I would >> expect to be in >> fromhost (the relay box that sent the message to me) >> >> the syslog daemons sending me the logs have been modified, so >> there is a >> possibility that I messed up on them and the format that's being sent >> isn't right, but if so I'm not seeing anything wrong with it. >> >> I am using the nextmaster branch. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From luigi.perroti at googlemail.com Tue Oct 28 10:12:36 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 10:12:36 +0100 Subject: [rsyslog] New HUP directive effects on ommysql Message-ID: Hi, maybe the developers would like to add this finding to the ommysql module documentation. I found out that using '$HUPisRestart off' with 3.21.6 has an effect on the ommysql module too. More precisely I noted that changes made to $ActionOmmysqlServerPort become effective. This might be useful if it's needed to change the database port without doing a full configuration reload. Please correct me if you think I got this wrong, thanks. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 14:00:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 14:00:03 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Hi Luigi, could you please double-check your observation? This should not happen, as ommysql does not have any HUP handler. If that happens, we may have a new track to what is the bug in your environment. I will concentrate on a lab with MySQL enabled after I hear back from you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Tuesday, October 28, 2008 10:13 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] New HUP directive effects on ommysql > > Hi, > > maybe the developers would like to add this finding to the ommysql > module documentation. > > I found out that using '$HUPisRestart off' with 3.21.6 has an effect > on the ommysql module too. > More precisely I noted that changes made to $ActionOmmysqlServerPort > become effective. This might be useful if it's needed to change the > database port without doing a full configuration reload. > > Please correct me if you think I got this wrong, thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Tue Oct 28 14:59:01 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 14:59:01 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: Hi, I looked into it more carefully and I get this by checking the configuration file: /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 rsyslogd: version 3.21.6, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 rsyslogd: End of config validation run. Bye. Indeed, line 31 is where I have the $HUPisRestart directive. Even if 3.21.6 is the latest released development version it occurred to me that this feature might only be in the repository at this time. If that's the case then I'm sorry about the noise. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 15:12:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:12:25 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: <1225203145.23821.131.camel@rgf9dev.intern.adiscon.com> Oh, sorry, I think we got the trouble spot ;) It is currently available as part of the "nextmaster" branch. I am right now in the process of switching stable/beta/master versions and nextmaster is what will be the new master when this is done. Rainer On Tue, 2008-10-28 at 14:59 +0100, Luigi Perroti wrote: > Hi, > > I looked into it more carefully and I get this by checking the > configuration file: > > /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 > > rsyslogd: version 3.21.6, config validation run (level 1), master > config /etc/rsyslog.conf > > rsyslogd: invalid or yet-unknown config file command - have you > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] > > rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 > rsyslogd: End of config validation run. Bye. > > Indeed, line 31 is where I have the $HUPisRestart directive. > > Even if 3.21.6 is the latest released development version it occurred > to me that this feature might only be in the repository at this time. > If that's the case then I'm sorry about the noise. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 28 15:55:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:55:15 +0100 Subject: [rsyslog] new v3-stable release candidate Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Hi all, I am about to switch the v3-stable release to 3.20, which means it will be based on the 3.19.x beta branch. I have created a test tarball, available at: http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz I would appreciate if some could try it out and report any potential issues they see. I intend to release this tomorrow or Thursday if nothing bad happens. Rainer From david at lang.hm Tue Oct 28 18:26:14 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 10:26:14 -0700 (PDT) Subject: [rsyslog] new v3-stable release candidate In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: On Tue, 28 Oct 2008, Rainer Gerhards wrote: > Hi all, > > I am about to switch the v3-stable release to 3.20, which means it will > be based on the 3.19.x beta branch. I have created a test tarball, > available at: > > http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz > > I would appreciate if some could try it out and report any potential > issues they see. I intend to release this tomorrow or Thursday if > nothing bad happens. one problem I have been seeing in the nextmaster branch is that with imudp on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the box rsyslog goes into a loop (eating 100% cpu, but not doing anything) when it recieves a IPv4 syslog packet. Rainer has not been able to duplicate the problem, could someone else test this? I've been seeing it on Debian amd64 boxes. David Lang From rgerhards at hq.adiscon.com Tue Oct 28 18:29:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 18:29:38 +0100 Subject: [rsyslog] new v3-stable release candidate In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BF@grfint2.intern.adiscon.com> > one problem I have been seeing in the nextmaster branch is that with > imudp > on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the > box > rsyslog goes into a loop (eating 100% cpu, but not doing anything) when > it > recieves a IPv4 syslog packet. > > Rainer has not been able to duplicate the problem, could someone else > test > this? That would be really good. But please note that nextmaster will become 3.23.x, not 3.20. The upcoming 3.20 stable release is based on the current beta. I just wanted to make sure we are all talking about the right versions :) Rainer From david at lang.hm Wed Oct 29 02:44:04 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 18:44:04 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, david at lang.hm wrote: > On Sun, 26 Oct 2008, Rainer Gerhards wrote: > >> Could you get me the debug output while such a message is being >> processed? That would probably be useful... > > I'll have to recompile to get a build with all the debug fetures enabled. > I will do this tomorrow. 1264.971559438:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1264.971582626:{6000}imudp.c: main queue: entry added, size now 1 entries 1264.971595478:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1264.971611682:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1264.971627886:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1264.971649678:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1264.971664206:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1264.971684042:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1264.971701084:{6000}imudp.c: 1264.971717009:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000049 1264.971747461:{5997}main queue:Reg/w0: Message has legacy syslog format. 1264.971767577:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1264.971784898:{5997}main queue:Reg/w0: (/var/log/messages) 1264.971798867:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1265.971497411:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1265.971509424:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1265.971527863:{6000}imudp.c: main queue: entry added, size now 1 entries 1265.971538480:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1265.971554125:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1265.971569491:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1265.971587372:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1265.971603017:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000050 1265.971617824:{5997}main queue:Reg/w0: Message has legacy syslog format. 1265.971633749:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1265.971647159:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1265.971664760:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1265.971681523:{6000}imudp.c: 1265.971696610:{5997}main queue:Reg/w0: (/var/log/messages) 1265.971712814:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1266.971494055:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1266.971506068:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 From rgerhards at hq.adiscon.com Wed Oct 29 15:23:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:23:46 +0100 Subject: [rsyslog] regex help wanted Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Hi folks, I am tracking down a problem with a user here in this forum thread: http://kb.monitorware.com/regex-match-for-port-t8764.html#p14411 He claims that rsyslog does not correctly handle POSIX ERE regular expressions, but I now have written a simple program (tester) that works in the same way. I don't think I have set any options wrong. However, I am not at all an regex expert. Maybe some of you could shed some light on this? I would really appreciate some help, as the situation sounds sub-optimal, at least until we have an explanation for it... Thanks, Rainer From aoz.syn at gmail.com Wed Oct 29 15:47:23 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 08:47:23 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> > He claims that rsyslog does not correctly handle POSIX ERE regular > expressions, but I now have written a simple program (tester) that works > in the same way. I don't think I have set any options wrong. However, I > am not at all an regex expert. I've not carefully read the portion of the source where you do your matching; can you point me in a general direction? What library are you using? RB From rgerhards at hq.adiscon.com Wed Oct 29 15:50:20 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:50:20 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 3:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > He claims that rsyslog does not correctly handle POSIX ERE regular > > expressions, but I now have written a simple program (tester) that > works > > in the same way. I don't think I have set any options wrong. However, > I > > am not at all an regex expert. > > I've not carefully read the portion of the source where you do your > matching; can you point me in a general direction? What library are > you using? Sure. There is a very simple sample program inside the forum thread. This is a direct link: http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 Well... I'll also include it after my sig. I use the plain old clib regex library. The problem can fully be reproduced with the minimalistic sample, so I think it is better to look at it than at the actual rsyslog source (but most of the regex functionality is in msg.c). Thanks, Rainer #include #include #include #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 (198.39.187.236/54751)" int main() { regex_t preg; size_t nmatch = 10; regmatch_t pmatch[10]; char str[] = STR; int i; i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); printf("regcomp returns %d\n", i); i = regexec(&preg, str, nmatch, pmatch, 0); printf("regexec returns %d\n", i); if(i == REG_NOMATCH) { printf("found no match!\n"); return 1; } printf("returned substrings:\n"); for(i = 0 ; i < 10 ; i++) { printf("%d: so %d, eo %d", i, pmatch[i].rm_so, pmatch[i].rm_eo); if(pmatch[i].rm_so != -1) { int j; printf(", text: '"); for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) putchar(str[j]); putchar('\''); } putchar('\n'); } return 0; } From hks.private at gmail.com Wed Oct 29 16:07:18 2008 From: hks.private at gmail.com ((private) HKS) Date: Wed, 29 Oct 2008 11:07:18 -0400 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: The lazy quantifier (?) is not being respected, so rather than returning the first match of "stuff, a forward slash, more stuff, space", it returns the last one. I know zero C, so I'm afraid I can't be much help in figuring out why. You could try the universal regexp trick: if in doubt, escape it! -HKS On Wed, Oct 29, 2008 at 10:50 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, October 29, 2008 3:47 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] regex help wanted >> >> > He claims that rsyslog does not correctly handle POSIX ERE regular >> > expressions, but I now have written a simple program (tester) that >> works >> > in the same way. I don't think I have set any options wrong. > However, >> I >> > am not at all an regex expert. >> >> I've not carefully read the portion of the source where you do your >> matching; can you point me in a general direction? What library are >> you using? > > Sure. There is a very simple sample program inside the forum thread. > This is a direct link: > > http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 > > Well... I'll also include it after my sig. I use the plain old clib > regex library. The problem can fully be reproduced with the minimalistic > sample, so I think it is better to look at it than at the actual rsyslog > source (but most of the regex functionality is in msg.c). > > Thanks, > Rainer > > #include > #include > #include > > #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 > for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 > (198.39.187.236/54751)" > int main() > { > regex_t preg; > size_t nmatch = 10; > regmatch_t pmatch[10]; > char str[] = STR; > int i; > > i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); > printf("regcomp returns %d\n", i); > i = regexec(&preg, str, nmatch, pmatch, 0); > printf("regexec returns %d\n", i); > if(i == REG_NOMATCH) { > printf("found no match!\n"); > return 1; > } > > printf("returned substrings:\n"); > for(i = 0 ; i < 10 ; i++) { > printf("%d: so %d, eo %d", i, pmatch[i].rm_so, > pmatch[i].rm_eo); > if(pmatch[i].rm_so != -1) { > int j; > printf(", text: '"); > for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) > putchar(str[j]); > putchar('\''); > } > putchar('\n'); > } > return 0; > } > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 29 16:32:39 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 09:32:39 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> > The lazy quantifier (?) is not being respected, so rather than > returning the first match of "stuff, a forward slash, more stuff, > space", it returns the last one. Correct, because the lazy quantifier is not POSIX and therefore does not work when using POSIX extended REs. I was wondering why the extra '?', but had forgotten about lazy notation, preferring to make my regexes a bit more strict. RB From rgerhards at hq.adiscon.com Wed Oct 29 16:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 16:34:52 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com><4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4DA@grfint2.intern.adiscon.com> Ah, thanks a lot to both of you. This greatly helps and makes me feel much better. Of course, we can add additional regex libs, but it has been proven that the current implementation works OK. Thanks again, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 4:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > The lazy quantifier (?) is not being respected, so rather than > > returning the first match of "stuff, a forward slash, more stuff, > > space", it returns the last one. > > Correct, because the lazy quantifier is not POSIX and therefore does > not work when using POSIX extended REs. I was wondering why the extra > '?', but had forgotten about lazy notation, preferring to make my > regexes a bit more strict. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 02:13:54 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:13:54 -0700 (PDT) Subject: [rsyslog] git url? Message-ID: what is the url for the rsyslog git repository? David Lang From mbiebl at gmail.com Wed Oct 1 02:19:05 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 1 Oct 2008 02:19:05 +0200 Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: 2008/10/1 : > what is the url for the rsyslog git repository? gitweb: http://git.adiscon.com/?p=rsyslog.git;a=summary clonable urls: git://git.adiscon.com/git/rsyslog.git http://git.adiscon.com/git/rsyslog.git -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 1 02:38:46 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:38:46 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Wed, 1 Oct 2008, Michael Biebl wrote: > 2008/10/1 : >> what is the url for the rsyslog git repository? > > gitweb: > http://git.adiscon.com/?p=rsyslog.git;a=summary > > clonable urls: > git://git.adiscon.com/git/rsyslog.git > http://git.adiscon.com/git/rsyslog.git thanks, much easier then guessing. now that I have his main branch cloned, what do I need to do to have it fetch all the other branches? David Lang From david at lang.hm Wed Oct 1 02:54:01 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:54:01 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Tue, 30 Sep 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Michael Biebl wrote: > >> 2008/10/1 : >>> what is the url for the rsyslog git repository? >> >> gitweb: >> http://git.adiscon.com/?p=rsyslog.git;a=summary >> >> clonable urls: >> git://git.adiscon.com/git/rsyslog.git >> http://git.adiscon.com/git/rsyslog.git > > thanks, much easier then guessing. > > now that I have his main branch cloned, what do I need to do to have it > fetch all the other branches? found it, git branch -r David Lang From luigi.perroti at googlemail.com Wed Oct 1 10:57:41 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 10:57:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated Message-ID: Hi, I'd like to ask if it's normal that %timegenerated:::date-subseconds% returns a value lower than %timereported:::date-subseconds%. Intuitively I'd say it is since this makes sense by taking in consideration the property names. Still, since the documentation says that timegenerated is the timestamp when the message was received I would have expected a value higher than the one returned by timereported. I'm afraid this is a dumb question but this behaviour got me confused, if anyone could shed some light on this I'd be grateful. Thanks. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 11:24:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 11:24:08 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> It is just returning the subsecond *part* of the timestamp. So while 9:13:14.123 is "higher" than 8:13:14.567, the subsecond part of the later (567) is higher than the subsecond part of the earlier (123). It is important that you are extracting a partial timestamp, which also implies that this is no longer monotonically increasing. Actually, it no longer has any sense of "timing" at all. This is NOT the timestamp converted to nanoseconds (or whatever). I hope this clarifies, if not, keep posting ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 10:58 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Subseconds in timereported and timegenerated > > Hi, > > I'd like to ask if it's normal that %timegenerated:::date-subseconds% > returns a value lower than %timereported:::date-subseconds%. > > Intuitively I'd say it is since this makes sense by taking in > consideration the property names. > Still, since the documentation says that timegenerated is the > timestamp when the message was received I would have expected a value > higher than the one returned by timereported. > > I'm afraid this is a dumb question but this behaviour got me confused, > if anyone could shed some light on this I'd be grateful. Thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 12:00:37 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:00:37 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: Thanks for the thorough explanation but I'm afraid that I still don't understand why I'm seeing this: http://imagebin.ca/view/2EknmP0.html This is how the columns are populated: sent <- %timereported:::date-mysql% sent_subseconds <- %timereported:::date-subseconds% received_by_rsyslog <- %timegenerated:::date-mysql% received_subseconds <- %timegenerated:::date-subseconds% Most probably I didn't get the difference between timereported and timegenerated. Meaning that I should just switch the sent* and received* columns and that nothing is wrong with Rsyslog. I'd be great if you could could confirm this scenario, thanks for your time! Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 12:03:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:03:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Just a quick note while working on a bigger thing: Is the sent timestamp form an instance that received data via the network (data that contained a high-precision timestamp). Also, which version are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks for the thorough explanation but I'm afraid that I still don't > understand why I'm seeing this: > > http://imagebin.ca/view/2EknmP0.html > > This is how the columns are populated: > sent <- %timereported:::date-mysql% > sent_subseconds <- %timereported:::date-subseconds% > received_by_rsyslog <- %timegenerated:::date-mysql% > received_subseconds <- %timegenerated:::date-subseconds% > > Most probably I didn't get the difference between timereported and > timegenerated. > Meaning that I should just switch the sent* and received* columns and > that nothing is wrong with Rsyslog. > > I'd be great if you could could confirm this scenario, thanks for your > time! > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 12:00:47 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 12:00:47 +0200 Subject: [rsyslog] Development of failsafe disk based queue Message-ID: <48E34A4F.308@ecker-software.de> Hi, I am looking for a failsafe solution to store syslog messages localy until they could be send later. I already looked at the disk based memory queue and the disk based queue. Both queue's don't work if you just power down the system immediatly actually loosing the whole queue. I already looked at queue.c and it seemed to me that both queues were not designed for that kind of failure, but I could be wrong there. Since an immediate power down of the system is the major failure which will occure pretty often I need to create a soltution there. Did you already start to develop something addressing that problem? Could you help me extend rsyslog (3.18.4) so that I can develop a new queue myself? I would contribute the code to the rsyslog project if you would like afterwards. bye David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:22:19 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:22:19 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: Here's a better screenshot: http://imagebin.ca/view/8p8hSnT.html Everything is happening on the same machine. I have these modules loaded: imfile, imuxsock, imklog, ommysql. The php-fpm entries are read from a log file by imfile. Right now I'm trying out 3.21.5 but I think I had already seen this behaviour on 3.21.3 but I'm not too sure about the version number. Luigi On 10/1/08, Rainer Gerhards wrote: > Just a quick note while working on a bigger thing: Is the sent timestamp > form an instance that received data via the network (data that contained > a high-precision timestamp). Also, which version are you using? > > Rainer > > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti >> Sent: Wednesday, October 01, 2008 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated >> >> Thanks for the thorough explanation but I'm afraid that I still don't >> understand why I'm seeing this: >> >> http://imagebin.ca/view/2EknmP0.html >> >> This is how the columns are populated: >> sent <- %timereported:::date-mysql% >> sent_subseconds <- %timereported:::date-subseconds% >> received_by_rsyslog <- %timegenerated:::date-mysql% >> received_subseconds <- %timegenerated:::date-subseconds% >> >> Most probably I didn't get the difference between timereported and >> timegenerated. >> Meaning that I should just switch the sent* and received* columns and >> that nothing is wrong with Rsyslog. >> >> I'd be great if you could could confirm this scenario, thanks for your >> time! >> >> Regards, >> Luigi >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 12:28:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:28:09 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Please post your config, that makes it easier for me ;). I think I know where this stems back from - we had two time calls inside rsyslog done, which has been removed in the experimental "perf" git branch. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Here's a better screenshot: > http://imagebin.ca/view/8p8hSnT.html > > Everything is happening on the same machine. > I have these modules loaded: imfile, imuxsock, imklog, ommysql. > The php-fpm entries are read from a log file by imfile. > > Right now I'm trying out 3.21.5 but I think I had already seen this > behaviour on 3.21.3 but I'm not too sure about the version number. > > Luigi > > On 10/1/08, Rainer Gerhards wrote: > > Just a quick note while working on a bigger thing: Is the sent > timestamp > > form an instance that received data via the network (data that > contained > > a high-precision timestamp). Also, which version are you using? > > > > Rainer > > > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > >> Sent: Wednesday, October 01, 2008 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > >> > >> Thanks for the thorough explanation but I'm afraid that I still > don't > >> understand why I'm seeing this: > >> > >> http://imagebin.ca/view/2EknmP0.html > >> > >> This is how the columns are populated: > >> sent <- %timereported:::date-mysql% > >> sent_subseconds <- %timereported:::date-subseconds% > >> received_by_rsyslog <- %timegenerated:::date-mysql% > >> received_subseconds <- %timegenerated:::date-subseconds% > >> > >> Most probably I didn't get the difference between timereported and > >> timegenerated. > >> Meaning that I should just switch the sent* and received* columns > and > >> that nothing is wrong with Rsyslog. > >> > >> I'd be great if you could could confirm this scenario, thanks for > your > >> time! > >> > >> Regards, > >> Luigi > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:32:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:32:26 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. > Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. I doubt there is a software soution against this (one that does not depend on a transactional file system, of course). What prevents you from using a UPS? I'd say that a sudden power-loss is by far the least probable error cause for a system that is configured to do any serious work. Please elaborate why you (or others ;)) consider this case important. > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:37:04 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:37:04 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: Here it is: $ModLoad imfile $ModLoad imuxsock $ModLoad imklog $ModLoad ommysql $ActionOmmysqlServerPort 1234 $MaxMessageSize 8k $EscapeControlCharactersOnReceive off $FileOwner root $FileGroup root $FileCreateMode 0600 $InputFileName /path/to/log/php-fpm.log $InputFileTag php-fpm $InputFileStateFile php-fpm.state $InputFileSeverity notice $InputFileFacility local5 $InputRunFileMonitor $MainMsgQueueType LinkedList $MainMsgQueueSize 20000 $MainMsgQueueFileName main-spool $MainMsgQueueMaxDiskSpace 200M $MainMsgQueueWorkerThreadMinimumMessages 2500 $MainMsgQueueWorkerThreads 1 $MainMsgQueueWorkerTimeoutThreadShutdown 10000 $MainMsgQueueSaveOnShutdown on $template my-rsyslog-schema,"insert into messages(hostname, sent, sent_subseconds, received_by_rsyslog, received_subseconds, process, facility, severity, message) values ('%hostname%', '%timereported:::date-mysql%', %timereported:::date-subseconds%, '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', '%msg%')",sql *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema From rgerhards at hq.adiscon.com Wed Oct 1 12:46:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:46:42 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Thanks, this is useful and helps understanding what is going on. I think what you see is an artifact of the two time() calls I mentioned. To verify, I'd appreciate if you could try out the latest from the perf branch, which can be found here: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=1531c541b294b6c029e5de9a01d4d8499f7e4718;sf=tgz Note that you need to setup the build environment (quite easy, but you need to know). Details at: http://www.rsyslog.com/doc-build_from_repo.html These two timestamps, for internally generated sources, should be the same in that version. Note, however, that, for externally generated messages, timereceived is whatever is recorded inside the message. There are various reasons for this not being correct, clock skew being a highly probably cause (which is also often found). HTH Rainer On Wed, 2008-10-01 at 12:37 +0200, Luigi Perroti wrote: > Here it is: > > $ModLoad imfile > $ModLoad imuxsock > $ModLoad imklog > $ModLoad ommysql > $ActionOmmysqlServerPort 1234 > > $MaxMessageSize 8k > > $EscapeControlCharactersOnReceive off > > $FileOwner root > $FileGroup root > $FileCreateMode 0600 > > $InputFileName /path/to/log/php-fpm.log > $InputFileTag php-fpm > $InputFileStateFile php-fpm.state > $InputFileSeverity notice > $InputFileFacility local5 > $InputRunFileMonitor > > $MainMsgQueueType LinkedList > $MainMsgQueueSize 20000 > $MainMsgQueueFileName main-spool > $MainMsgQueueMaxDiskSpace 200M > $MainMsgQueueWorkerThreadMinimumMessages 2500 > $MainMsgQueueWorkerThreads 1 > $MainMsgQueueWorkerTimeoutThreadShutdown 10000 > $MainMsgQueueSaveOnShutdown on > > $template my-rsyslog-schema,"insert into messages(hostname, sent, > sent_subseconds, received_by_rsyslog, received_subseconds, process, > facility, severity, message) values ('%hostname%', > '%timereported:::date-mysql%', %timereported:::date-subseconds%, > '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, > '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', > '%msg%')",sql > > *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:50:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:50:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> One thing I forgot to mention: a pure disk queue (not a disk-assisted one) gets you as close to your goal as possible (well, mostly - we could, at a considerable performance expense, require synced writing). With that case, all data is immediately stored on disk. You can configure it to also write the meta data out immediately (and again with sync, not yet supported). However, you still have a window of exposure, for example if the power loss happens right in the middle of when the disk actually writes data to the disk sector. I still wonder why this scenario would be useful to address... Rainer On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. > > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 13:02:50 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:02:50 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> Message-ID: <48E358DA.5080104@ecker-software.de> Rainer Gerhards schrieb: >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole >> > queue. > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. >> Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> > > I doubt there is a software soution against this (one that does not > depend on a transactional file system, of course). What prevents you > from using a UPS? I'd say that a sudden power-loss is by far the least > probable error cause for a system that is configured to do any serious > work. > > Please elaborate why you (or others ;)) consider this case important. > > The client systems (about 200 of them planned) are stationed in public places around the world connected to centralized servers through vpn connections over an unreliable network connection. Since space and look requirement is important a UPS won't fit there. There is actually no space for an UPS. The main problem is that customers are actually pulling the plug to restart the system, to charge their laptops or mobile phones or just for the fun of it. The client base image is a read-only system (Knoppix Like) with an extra hard disk for swap and other informations like syslog messages. Since there are no administrators close to the client system the client itself needs to have the capability to send all the missing log information between a network failure and an immediate power down to the central server for error analysis since those are usualy the most important once to pinpoint the cause of the inital error. My approach would be to use a block device directly since a file system if fault-prone if you shut down the system immediatly. Each entry including the header information guarded by a checksum value. It would be actually something like a fixed array based queue just that it would store the information in a block device. But this is just an inital thought. >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if >> > you > >> would like afterwards. >> >> bye >> David Ecker >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 13:15:22 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:15:22 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E35BCA.7030808@ecker-software.de> Rainer Gerhards schrieb: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > If that would work it would be perfect. For testing I could actually pass the correct fctl flag inside of queue.c hardcoded if that is all that is needed. > I still wonder why this scenario would be useful to address... > > > David > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 13:48:32 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:48:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E35BCA.7030808@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <48E35BCA.7030808@ecker-software.de> Message-ID: <1222861712.2682.450.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 13:15 +0200, David Ecker wrote: > Rainer Gerhards schrieb: > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > > If that would work it would be perfect. For testing I could actually > pass the correct fctl flag inside of queue.c hardcoded if that is all > that is needed. that works ;) You just need to fine-tune the queue params. If you find problems with that, I am more than happy to help. > > I still wonder why this scenario would be useful to address... very interesting scenario. Never thought about such one :) If you need to enhance the queue (for block devices), I can provide a some hints. But, given the other priorities which are required by a much broader user base, involving me more than a hint here or there would probably require a consulting contract. Sorry for being so bluntly, but I think it is always good to set the right expectation level. Rainer From david at lang.hm Wed Oct 1 13:55:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 04:55:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. are you sure about the disk based queue? per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue can be set to do a commit of the metadata after each message. Disk Queues Disk queues use disk drives for buffering. The important fact is that the always use the disk and do not buffer anything in memory. Thus, the queue is ultra-reliable, but by far the slowest mode. For regular use cases, this queue mode is not recommended. It is useful if log data is so important that it must not be lost, even in extreme cases. When a disk queue is written, it is done in chunks. Each chunk receives its individual file. Files are named with a prefix (set via the "$QueueFilename" config directive) and followed by a 7-digit number (starting at one and incremented for each file). Chunks are 10mb by default, a different size can be set via the"$QueueMaxFileSize" config directive. Note that the size limit is not a sharp one: rsyslog always writes one complete queue entry, even if it violates the size limit. So chunks are actually a little but (usually less than 1k) larger then the configured size. Each chunk also has a different size for the same reason. If you observe different chunk sizes, you can relax: this is not a problem. Writing in chunks is used so that processed data can quickly be deleted and is free for other uses - while at the same time keeping no artificial upper limit on disk space used. If a disk quota is set (instructions further below), be sure that the quota/chunk size allows at least two chunks to be written. Rsyslog currently does not check that and will fail miserably if a single chunk is over the quota. Creating new chunks costs performance but provides quicker ability to free disk space. The 10mb default is considered a good compromise between these two. However, it may make sense to adapt these settings to local policies. For example, if a disk queue is written on a dedicated 200gb disk, it may make sense to use a 2gb (or even larger) chunk size. Please note, however, that the disk queue by default does not update its housekeeping structures every time it writes to disk. This is for performance reasons. In the event of failure, data will still be lost (except when manually is mangled with the file structures). However, disk queues can be set to write bookkeeping information on checkpoints (every n records), so that this can be made ultra-reliable, too. If the checkpoint interval is set to one, no data can be lost, but the queue is exceptionally slow. Each queue can be placed on a different disk for best performance and/or isolation. This is currently selected by specifying different $WorkDirectory config directives before the queue creation statement. To create a disk queue, use the "$QueueType Disk" config directive. Checkpoint intervals can be specified via "$QueueCheckpointInterval", with 0 meaning no checkpoints. you also need to specificly enable syncing (from http://www.rsyslog.com/doc-v3compatibility.html ) Output File Syncing Rsyslogd tries to keep as compatible to stock syslogd as possible. As such, it retained stock syslogd's default of syncing every file write if not specified otherwise (by placing a dash in front of the output file name). While this was a useful feature in past days where hardware was much less reliable and UPS seldom, this no longer is useful in today's worl. Instead, the syncing is a high performace hit. With it, rsyslogd writes files around 50 *times* slower than without it. It also affects overall system performance due to the high IO activity. In rsyslog v3, syncing has been turned off by default. This is done via a specific configuration directive "$ActionFileEnableSync on/off" which is off by default. So even if rsyslogd finds sync selector lines, it ignores them by default. In order to enable file syncing, the administrator must specify "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that syncing only happens in some installations where the administrator actually wanted that (performance-intense) feature. In the fast majority of cases (if not all), this dramatically increases rsyslogd performance without any negative effects. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. with checkpoint interval set to 1 and syncing enabled the data should be in on the disk safely (assuming you have hardware that supports this) and a power-off won't affect it. David Lang > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 13:57:55 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:57:55 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> David, the file syncing mentioned in the compatibility doc applies to the output action, only. The queue does never do synchronous writes - I always assumed that a critical system would have a UPS and could never think (so far) about a valid reason for not having it. So the queue would need to have an extra option to do sync writes. Obviously, that's not a big deal. Performance, of course, will be extremely terrible with such a setup... Rainer On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Hi, > > > > I am looking for a failsafe solution to store syslog messages localy > > until they could be send later. I already looked at the disk based > > memory queue and the disk based queue. Both queue's don't work if you > > just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that the > always use the disk and do not buffer anything in memory. Thus, the queue > is ultra-reliable, but by far the slowest mode. For regular use cases, > this queue mode is not recommended. It is useful if log data is so > important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk receives > its individual file. Files are named with a prefix (set via the > "$QueueFilename" config directive) and followed by a 7-digit > number (starting at one and incremented for each file). Chunks are 10mb by > default, a different size can be set via the"$QueueMaxFileSize" > config directive. Note that the size limit is not a sharp one: rsyslog > always writes one complete queue entry, even if it violates the size > limit. So chunks are actually a little but (usually less than 1k) larger > then the configured size. Each chunk also has a different size for the > same reason. If you observe different chunk sizes, you can relax: this is > not a problem. > > Writing in chunks is used so that processed data can quickly be deleted > and is free for other uses - while at the same time keeping no artificial > upper limit on disk space used. If a disk quota is set (instructions > further below), be sure that the quota/chunk size allows at least two > chunks to be written. Rsyslog currently does not check that and will fail > miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to free > disk space. The 10mb default is considered a good compromise between these > two. However, it may make sense to adapt these settings to local policies. > For example, if a disk queue is written on a dedicated 200gb disk, it may > make sense to use a 2gb (or even larger) chunk size. > > Please note, however, that the disk queue by default does not update its > housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, disk > queues can be set to write bookkeeping information on checkpoints (every n > records), so that this can be made ultra-reliable, too. If the checkpoint > interval is set to one, no data can be lost, but the queue is > exceptionally slow. > > Each queue can be placed on a different disk for best performance and/or > isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write if > not specified otherwise (by placing a dash in front of the output file > name). While this was a useful feature in past days where hardware was > much less reliable and UPS seldom, this no longer is useful in today's > worl. Instead, the syncing is a high performace hit. With it, rsyslogd > writes files around 50 *times* slower than without it. It also affects > overall system performance due to the high IO activity. In rsyslog v3, > syncing has been turned off by default. This is done via a specific > configuration directive "$ActionFileEnableSync on/off" which is off by > default. So even if rsyslogd finds sync selector lines, it ignores them by > default. In order to enable file syncing, the administrator must specify > "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > syncing only happens in some installations where the administrator > actually wanted that (performance-intense) feature. In the fast majority > of cases (if not all), this dramatically increases rsyslogd performance > without any negative effects. > > > > > I already looked at queue.c and it seemed to me that both queues were > > not designed for that kind of failure, but I could be wrong there. Since > > an immediate power down of the system is the major failure which will > > occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should be > in on the disk safely (assuming you have hardware that supports this) and > a power-off won't affect it. > > David Lang > > > > > Did you already start to develop something addressing that problem? > > Could you help me extend rsyslog (3.18.4) so that I can develop a new > > queue myself? I would contribute the code to the rsyslog project if you > > would like afterwards. > > > > bye > > David Ecker > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:02:37 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:02:37 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E358DA.5080104@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> <48E358DA.5080104@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Rainer Gerhards schrieb: >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole >>> >> queue. >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. >>> Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> >> I doubt there is a software soution against this (one that does not >> depend on a transactional file system, of course). What prevents you >> from using a UPS? I'd say that a sudden power-loss is by far the least >> probable error cause for a system that is configured to do any serious >> work. >> >> Please elaborate why you (or others ;)) consider this case important. >> >> > The client systems (about 200 of them planned) are stationed in public > places around the world connected to centralized servers through vpn > connections over an unreliable network connection. Since space and look > requirement is important a UPS won't fit there. There is actually no > space for an UPS. The main problem is that customers are actually > pulling the plug to restart the system, to charge their laptops or > mobile phones or just for the fun of it. you can get UPS systems that are PCI cards, completly internal. they still may not fit, but you at least have a chance. > The client base image is a read-only system (Knoppix Like) with an extra > hard disk for swap and other informations like syslog messages. Since > there are no administrators close to the client system the client itself > needs to have the capability to send all the missing log information > between a network failure and an immediate power down to the central > server for error analysis since those are usualy the most important once > to pinpoint the cause of the inital error. > > My approach would be to use a block device directly since a file system > if fault-prone if you shut down the system immediatly. Each entry > including the header information guarded by a checksum value. It would > be actually something like a fixed array based queue just that it would > store the information in a block device. But this is just an inital thought. you are inventing a new filesystem here. it's not that easy to be reliable becouse the disk can lie to you. unless you are doing interesting things at the ATA/SCSI command level the disk may re-order your writes and may cache them in memory on the drive for an unknown time before actually writing them if you need reliable writes at anything close to a reasonable speed you need to have a battery backed cache or solid state drive in your machine (and the solid state drives are not all fast to write to) David Lang >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if >>> >> you >> >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 14:05:42 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:05:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E36796.9060509@ecker-software.de> Hi, should I use 3.18.4 (latest stable, I would preferr that one) or do I need the latest development version? I would actually alter queue.c directly changing the fctl flags in the disk based queue (O_DIRECT,O_SYNC,.O_NOATIME). Performance is not really an issue. There will be only 1000 to 2000 Messages per hour in peak times bye David Ecker Rainer Gerhards schrieb: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. > > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. > > Performance, of course, will be extremely terrible with such a setup... > > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >>> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:07:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:07:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > I still wonder why this scenario would be useful to address... not all uses of rsyslog are for simple system logs. it's a good general purpose log tool, and there are some cases where you want to be as sure as you possibly can be that once a message has been acknowledged it has no chance of being lost. useing some form of solid-state reliable storage (battery backed ram on a raid controller, a battery backed ram disk, a flash disk) it is possible (but not nessasarily cheap) to get the ability to do tens to hundreds of thousands of writes + syncs per second David Lang > Rainer > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:07:32 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:07:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <48E36804.3030107@ecker-software.de> Hi David, the actuall problem was that the qi file was not correct after an immediate restart. All messages were actually correctly written except the qi file. bye David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based > queue can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that > the always use the disk and do not buffer anything in memory. Thus, > the queue is ultra-reliable, but by far the slowest mode. For regular > use cases, this queue mode is not recommended. It is useful if log > data is so important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk > receives its individual file. Files are named with a prefix (set via > the "$QueueFilename" config directive) and followed by a > 7-digit number (starting at one and incremented for each file). Chunks > are 10mb by default, a different size can be set via > the"$QueueMaxFileSize" config directive. Note that the size > limit is not a sharp one: rsyslog always writes one complete queue > entry, even if it violates the size limit. So chunks are actually a > little but (usually less than 1k) larger then the configured size. > Each chunk also has a different size for the same reason. If you > observe different chunk sizes, you can relax: this is not a problem. > > Writing in chunks is used so that processed data can quickly be > deleted and is free for other uses - while at the same time keeping no > artificial upper limit on disk space used. If a disk quota is set > (instructions further below), be sure that the quota/chunk size allows > at least two chunks to be written. Rsyslog currently does not check > that and will fail miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to > free disk space. The 10mb default is considered a good compromise > between these two. However, it may make sense to adapt these settings > to local policies. For example, if a disk queue is written on a > dedicated 200gb disk, it may make sense to use a 2gb (or even larger) > chunk size. > > Please note, however, that the disk queue by default does not update > its housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, > disk queues can be set to write bookkeeping information on checkpoints > (every n records), so that this can be made ultra-reliable, too. If > the checkpoint interval is set to one, no data can be lost, but the > queue is exceptionally slow. > > Each queue can be placed on a different disk for best performance > and/or isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write > if not specified otherwise (by placing a dash in front of the output > file name). While this was a useful feature in past days where > hardware was much less reliable and UPS seldom, this no longer is > useful in today's worl. Instead, the syncing is a high performace hit. > With it, rsyslogd writes files around 50 *times* slower than without > it. It also affects overall system performance due to the high IO > activity. In rsyslog v3, syncing has been turned off by default. This > is done via a specific configuration directive "$ActionFileEnableSync > on/off" which is off by default. So even if rsyslogd finds sync > selector lines, it ignores them by default. In order to enable file > syncing, the administrator must specify "$ActionFileEnableSync on" at > the top of rsyslog.conf. This ensures that syncing only happens in > some installations where the administrator actually wanted that > (performance-intense) feature. In the fast majority of cases (if not > all), this dramatically increases rsyslogd performance without any > negative effects. > > > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should > be in on the disk safely (assuming you have hardware that supports > this) and a power-off won't affect it. > > David Lang > > > >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:09:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:09:33 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E36796.9060509@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <48E36796.9060509@ecker-software.de> Message-ID: <1222862973.2682.456.camel@rgf9dev.intern.adiscon.com> I am currently working on the queue engine in the devel branch. In that light, it could make sense to start with that branch. Also, I won't make any non-bug related changes in the stable version, so if you run into anything I can quickly change, that will happen in devel. Rainer On Wed, 2008-10-01 at 14:05 +0200, David Ecker wrote: > Hi, > > should I use 3.18.4 (latest stable, I would preferr that one) or do I > need the latest development version? I would actually alter queue.c > directly changing the fctl flags in the disk based queue > (O_DIRECT,O_SYNC,.O_NOATIME). > > Performance is not really an issue. There will be only 1000 to 2000 > Messages per hour in peak times > > bye > David Ecker > > Rainer Gerhards schrieb: > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > > > Performance, of course, will be extremely terrible with such a setup... > > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >>> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >>> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 14:09:53 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 14:09:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: I installed the snapshot but apparently the issue persists. If there's something else that I can do, now or when you have more time, let me know. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 14:12:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:12:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > I still wonder why this scenario would be useful to address... > > not all uses of rsyslog are for simple system logs. it's a good general > purpose log tool, and there are some cases where you want to be as sure as > you possibly can be that once a message has been acknowledged it has no > chance of being lost. I designed the engine for audit-class reliability. However, I assumed that the rest of the system is also playing in that class. Doing everything with a potential power failure in mind creates a lot of extra demands. And I have never heard of anybody doing serious datacenter work without a proper UPS. Is this *really* an issue? Rainer > > useing some form of solid-state reliable storage (battery backed ram on a > raid controller, a battery backed ram disk, a flash disk) it is possible > (but not nessasarily cheap) to get the ability to do tens to hundreds of > thousands of writes + syncs per second > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, > >> > >> I am looking for a failsafe solution to store syslog messages localy > >> until they could be send later. I already looked at the disk based > >> memory queue and the disk based queue. Both queue's don't work if you > >> just power down the system immediatly actually loosing the whole queue. > >> I already looked at queue.c and it seemed to me that both queues were > >> not designed for that kind of failure, but I could be wrong there. Since > >> an immediate power down of the system is the major failure which will > >> occure pretty often I need to create a soltution there. > >> > >> Did you already start to develop something addressing that problem? > >> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >> queue myself? I would contribute the code to the rsyslog project if you > >> would like afterwards. > >> > >> bye > >> David Ecker > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 14:15:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:15:14 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863314.2682.463.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:09 +0200, Luigi Perroti wrote: > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. That's interesting. Could you please get me a debug log (you can mail it privately). Here is info about how to do that: http://www.rsyslog.com/doc-troubleshoot.html Rainer From david at lang.hm Wed Oct 1 14:25:18 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:25:18 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. ouch. > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. good > Performance, of course, will be extremely terrible with such a setup... only if you have to wait for a spinning disk to do the write. this is the same problem that databases have. they need to guarentee that once the database tells the writing program that the data is written it will be there even if the system looses power immediatly. if you run a database on standard desktop hardware (and it doesn't have this safety disabled) you cannot do more then about 80 writes/second. If you upgrade to the super speedy 15K rpm drives you can do ~160 writes/second. given that you need to write the data + metadata it gets even uglier, so what the databases do (and some journaling filesystems) is to write a log that says what they are going to do, sync that, and then later write the data to the actual files (updating the journal when they complete the write) it sounds like you order your write correctly for a disk-based queue, but you would need the option of issuing the syncs (probably when you do the checkpoints) if you do this on the wrong hardware (say a laptop 5200 rpm drive or the wrong flash drive), the fact that you need to do four writes per log entry (data to queue, metadata to queue, data to output, update metadata for queue) could drop you to below 15 logs/sec (60/4 but then you loose time to seeking as well) however, with the correct drive to write to (say a $2,400 80G fusion-io flash card that can do ~100k IO ops/sec) you should be able to sustain 20,000 logs/sec. realisticly very few people need the sustained write capacity that you would get from such a setup. but if you go with a $500-$700 raid card with a battery-backed cache you get very similar performance, but with some possibility that you can't sustain it forever. David Lang > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, David Ecker wrote: >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:29:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:29:03 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > ouch. > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > good > > > Performance, of course, will be extremely terrible with such a setup... > > only if you have to wait for a spinning disk to do the write. I agree to the rest of your argument below. But the question raised here was in regard to a system without any battery backup. So I would need to wait. Even then, in the worst case, I think it would be possible that the disk does only a partial write. I am not sure if that's really the case with today's disk drives (which I think have capacitors to prevent this scenario), but with past drives this could happen (I know all too well - a few years ago that cost me a weekend ;)). Rainer > > this is the same problem that databases have. they need to guarentee that > once the database tells the writing program that the data is written it > will be there even if the system looses power immediatly. > > if you run a database on standard desktop hardware (and it doesn't have > this safety disabled) you cannot do more then about 80 writes/second. If > you upgrade to the super speedy 15K rpm drives you can do ~160 > writes/second. > > given that you need to write the data + metadata it gets even uglier, so > what the databases do (and some journaling filesystems) is to write a log > that says what they are going to do, sync that, and then later write the > data to the actual files (updating the journal when they complete the > write) > > it sounds like you order your write correctly for a disk-based queue, but > you would need the option of issuing the syncs (probably when you do the > checkpoints) > > if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > wrong flash drive), the fact that you need to do four writes per log entry > (data to queue, metadata to queue, data to output, update metadata for > queue) could drop you to below 15 logs/sec (60/4 but then you loose time > to seeking as well) > > however, with the correct drive to write to (say a $2,400 80G fusion-io > flash card that can do ~100k IO ops/sec) you should be able to sustain > 20,000 logs/sec. > > realisticly very few people need the sustained write capacity that you > would get from such a setup. but if you go with a $500-$700 raid card with > a battery-backed cache you get very similar performance, but with some > possibility that you can't sustain it forever. > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:35:17 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:35:17 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> One thing I forgot to mention: a pure disk queue (not a disk-assisted >>> one) gets you as close to your goal as possible (well, mostly - we >>> could, at a considerable performance expense, require synced writing). >>> With that case, all data is immediately stored on disk. You can >>> configure it to also write the meta data out immediately (and again with >>> sync, not yet supported). However, you still have a window of exposure, >>> for example if the power loss happens right in the middle of when the >>> disk actually writes data to the disk sector. >>> >>> I still wonder why this scenario would be useful to address... >> >> not all uses of rsyslog are for simple system logs. it's a good general >> purpose log tool, and there are some cases where you want to be as sure as >> you possibly can be that once a message has been acknowledged it has no >> chance of being lost. > > I designed the engine for audit-class reliability. However, I assumed > that the rest of the system is also playing in that class. Doing > everything with a potential power failure in mind creates a lot of extra > demands. And I have never heard of anybody doing serious datacenter work > without a proper UPS. Is this *really* an issue? Yes. UPSs fail. generators fail power cords come loose. power cords get unplugged by someone who thinks they are unplugging a different system people bump power switches on power strips. power supplies are defective I had one production outage where a visiting tech pulled a power cord from an overhead plug and dropped it on the ground, where it happened to hit the power switch on a power strip. I've had high-end systems with redundant power supplies go down becouse of faulty hardware that decided to disble both power supplies at once (it turned out that there was a defect in the whole batch of servers, but it took IBM several weeks to figure out what was going on) I've had UPS systems blow up (literally) I've had a datacenter go down becouse the it was running on generator power (due to other issues), and the refueling guy filled the tank incorrectly and got air bubbles into the fuel system, a few min later the 500Kw diesel generator couldn't maintain constant speed and the safety triggers kicked in and disabled it. it's amazing the things that happen in real-life David Lang > Rainer >> >> useing some form of solid-state reliable storage (battery backed ram on a >> raid controller, a battery backed ram disk, a flash disk) it is possible >> (but not nessasarily cheap) to get the ability to do tens to hundreds of >> thousands of writes + syncs per second >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >>>> Hi, >>>> >>>> I am looking for a failsafe solution to store syslog messages localy >>>> until they could be send later. I already looked at the disk based >>>> memory queue and the disk based queue. Both queue's don't work if you >>>> just power down the system immediatly actually loosing the whole queue. >>>> I already looked at queue.c and it seemed to me that both queues were >>>> not designed for that kind of failure, but I could be wrong there. Since >>>> an immediate power down of the system is the major failure which will >>>> occure pretty often I need to create a soltution there. >>>> >>>> Did you already start to develop something addressing that problem? >>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>> queue myself? I would contribute the code to the rsyslog project if you >>>> would like afterwards. >>>> >>>> bye >>>> David Ecker >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:39:54 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:39:54 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >> >> ouch. >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >> >> good >> >>> Performance, of course, will be extremely terrible with such a setup... >> >> only if you have to wait for a spinning disk to do the write. > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. no UPS is not nessasarily the same as no battey backup. you could use a compact flash drive and probably get better performance/reliability than spinning disks with no battery at all. > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). current disks do not have capacitors to prevent partial writes or to flush their caches. but options like the linux ext3 data-journaled make it so that you have your data in the journal safely, and the various solid-state options solve that problem. David Lang > Rainer > >> >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:45:06 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:45:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E370D2.7080901@ecker-software.de> Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >>> >> ouch. >> >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >>> >> good >> >> >>> Performance, of course, will be extremely terrible with such a setup... >>> >> only if you have to wait for a spinning disk to do the write. >> > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. > > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). > > Rainer > Hi, as long as you do sector based writes (512 byte per sector, usual) you can be sure that the write wasn"t partial.. Writing more than one sector or not starting at a correct offset (n*512,n=0,1,2,...x) might result in a partial write. I'll already tested that with my devel client here. So fencing each sector with a crc32 value would help detecting errors during a write operation. This is actually only a problem if you are writing directly to a block device like any filesystem does and yes, reordering is definitly a problem. So validating the content written to the disk afterwards is important. If writing through a filesystem reserving space in the destination file beforehand actually minimizes errors since the file system table doesn't have to be updated (you should also use the Flag O_NOATIME for that case). See for example VMWare ESX VMDK file handling. David > >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>> >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:55:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:55:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E370D2.7080901@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> Message-ID: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: [snip] > as long as you do sector based writes (512 byte per sector, usual) you > can be sure that the write wasn"t partial.. Writing more than one sector > or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > a partial write. I'll already tested that with my devel client here. So > fencing each sector with a crc32 value would help detecting errors > during a write operation. This is actually only a problem if you are > writing directly to a block device like any filesystem does and yes, > reordering is definitly a problem. So validating the content written to > the disk afterwards is important. > > If writing through a filesystem reserving space in the destination file > beforehand actually minimizes errors since the file system table doesn't > have to be updated (you should also use the Flag O_NOATIME for that > case). See for example VMWare ESX VMDK file handling. Well, first of all let me re-iterate that I do not intend to do a block device driver for rsyslog (but I definitely do not object getting one contributed ;)). Still thinking about the case and thinking about non-solid-state, non-internal-battery-backed-up disk, I can't see how you can be sure the data will be written. David just told me there are no capacitors. So if power fails, it fails rather quickly. So how can you be sure the disk will be able to finish writing that sector? Let's say the drive has begun to write the sector and been able to write the first 5 bytes. Now power fails. No capacitors, no battery-backup, so why should there be enough power to drive the disk write head for another 507 bytes? It the drives assures it can do that, it needs capacitors - doesn't it? Am I overlooking something obvious? Rainer From rgerhards at hq.adiscon.com Wed Oct 1 14:57:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:57:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> David, going back to the higher layer: do you say that immediate power failure is a case that you consider needed to be addressed in an enterprise logging system? Anybody else with an opinion? Rainer On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> > >>> David, > >>> > >>> the file syncing mentioned in the compatibility doc applies to the > >>> output action, only. > >> > >> ouch. > >> > >>> The queue does never do synchronous writes - I always assumed that a > >>> critical system would have a UPS and could never think (so far) about a > >>> valid reason for not having it. So the queue would need to have an extra > >>> option to do sync writes. Obviously, that's not a big deal. > >> > >> good > >> > >>> Performance, of course, will be extremely terrible with such a setup... > >> > >> only if you have to wait for a spinning disk to do the write. > > > > I agree to the rest of your argument below. But the question raised here > > was in regard to a system without any battery backup. So I would need to > > wait. > > no UPS is not nessasarily the same as no battey backup. > > you could use a compact flash drive and probably get better > performance/reliability than spinning disks with no battery at all. > > > Even then, in the worst case, I think it would be possible that the disk > > does only a partial write. I am not sure if that's really the case with > > today's disk drives (which I think have capacitors to prevent this > > scenario), but with past drives this could happen (I know all too well - > > a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. > > David Lang > > > Rainer > > > >> > >> this is the same problem that databases have. they need to guarentee that > >> once the database tells the writing program that the data is written it > >> will be there even if the system looses power immediatly. > >> > >> if you run a database on standard desktop hardware (and it doesn't have > >> this safety disabled) you cannot do more then about 80 writes/second. If > >> you upgrade to the super speedy 15K rpm drives you can do ~160 > >> writes/second. > >> > >> given that you need to write the data + metadata it gets even uglier, so > >> what the databases do (and some journaling filesystems) is to write a log > >> that says what they are going to do, sync that, and then later write the > >> data to the actual files (updating the journal when they complete the > >> write) > >> > >> it sounds like you order your write correctly for a disk-based queue, but > >> you would need the option of issuing the syncs (probably when you do the > >> checkpoints) > >> > >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > >> wrong flash drive), the fact that you need to do four writes per log entry > >> (data to queue, metadata to queue, data to output, update metadata for > >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time > >> to seeking as well) > >> > >> however, with the correct drive to write to (say a $2,400 80G fusion-io > >> flash card that can do ~100k IO ops/sec) you should be able to sustain > >> 20,000 logs/sec. > >> > >> realisticly very few people need the sustained write capacity that you > >> would get from such a setup. but if you go with a $500-$700 raid card with > >> a battery-backed cache you get very similar performance, but with some > >> possibility that you can't sustain it forever. > >> > >> David Lang > >> > >>> Rainer > >>> > >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >>>> On Wed, 1 Oct 2008, David Ecker wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I am looking for a failsafe solution to store syslog messages localy > >>>>> until they could be send later. I already looked at the disk based > >>>>> memory queue and the disk based queue. Both queue's don't work if you > >>>>> just power down the system immediatly actually loosing the whole queue. > >>>> > >>>> are you sure about the disk based queue? > >>>> > >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >>>> can be set to do a commit of the metadata after each message. > >>>> > >>>> Disk Queues > >>>> > >>>> Disk queues use disk drives for buffering. The important fact is that the > >>>> always use the disk and do not buffer anything in memory. Thus, the queue > >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, > >>>> this queue mode is not recommended. It is useful if log data is so > >>>> important that it must not be lost, even in extreme cases. > >>>> > >>>> When a disk queue is written, it is done in chunks. Each chunk receives > >>>> its individual file. Files are named with a prefix (set via the > >>>> "$QueueFilename" config directive) and followed by a 7-digit > >>>> number (starting at one and incremented for each file). Chunks are 10mb by > >>>> default, a different size can be set via the"$QueueMaxFileSize" > >>>> config directive. Note that the size limit is not a sharp one: rsyslog > >>>> always writes one complete queue entry, even if it violates the size > >>>> limit. So chunks are actually a little but (usually less than 1k) larger > >>>> then the configured size. Each chunk also has a different size for the > >>>> same reason. If you observe different chunk sizes, you can relax: this is > >>>> not a problem. > >>>> > >>>> Writing in chunks is used so that processed data can quickly be deleted > >>>> and is free for other uses - while at the same time keeping no artificial > >>>> upper limit on disk space used. If a disk quota is set (instructions > >>>> further below), be sure that the quota/chunk size allows at least two > >>>> chunks to be written. Rsyslog currently does not check that and will fail > >>>> miserably if a single chunk is over the quota. > >>>> > >>>> Creating new chunks costs performance but provides quicker ability to free > >>>> disk space. The 10mb default is considered a good compromise between these > >>>> two. However, it may make sense to adapt these settings to local policies. > >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may > >>>> make sense to use a 2gb (or even larger) chunk size. > >>>> > >>>> Please note, however, that the disk queue by default does not update its > >>>> housekeeping structures every time it writes to disk. This is for > >>>> performance reasons. In the event of failure, data will still be lost > >>>> (except when manually is mangled with the file structures). However, disk > >>>> queues can be set to write bookkeeping information on checkpoints (every n > >>>> records), so that this can be made ultra-reliable, too. If the checkpoint > >>>> interval is set to one, no data can be lost, but the queue is > >>>> exceptionally slow. > >>>> > >>>> Each queue can be placed on a different disk for best performance and/or > >>>> isolation. This is currently selected by specifying different > >>>> $WorkDirectory config directives before the queue creation statement. > >>>> > >>>> To create a disk queue, use the "$QueueType Disk" config > >>>> directive. Checkpoint intervals can be specified via > >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> you also need to specificly enable syncing (from > >>>> http://www.rsyslog.com/doc-v3compatibility.html ) > >>>> > >>>> Output File Syncing > >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >>>> such, it retained stock syslogd's default of syncing every file write if > >>>> not specified otherwise (by placing a dash in front of the output file > >>>> name). While this was a useful feature in past days where hardware was > >>>> much less reliable and UPS seldom, this no longer is useful in today's > >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >>>> writes files around 50 *times* slower than without it. It also affects > >>>> overall system performance due to the high IO activity. In rsyslog v3, > >>>> syncing has been turned off by default. This is done via a specific > >>>> configuration directive "$ActionFileEnableSync on/off" which is off by > >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by > >>>> default. In order to enable file syncing, the administrator must specify > >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >>>> syncing only happens in some installations where the administrator > >>>> actually wanted that (performance-intense) feature. In the fast majority > >>>> of cases (if not all), this dramatically increases rsyslogd performance > >>>> without any negative effects. > >>>> > >>>> > >>>> > >>>>> I already looked at queue.c and it seemed to me that both queues were > >>>>> not designed for that kind of failure, but I could be wrong there. Since > >>>>> an immediate power down of the system is the major failure which will > >>>>> occure pretty often I need to create a soltution there. > >>>> > >>>> with checkpoint interval set to 1 and syncing enabled the data should be > >>>> in on the disk safely (assuming you have hardware that supports this) and > >>>> a power-off won't affect it. > >>>> > >>>> David Lang > >>>> > >>>> > >>>> > >>>>> Did you already start to develop something addressing that problem? > >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>>>> queue myself? I would contribute the code to the rsyslog project if you > >>>>> would like afterwards. > >>>>> > >>>>> bye > >>>>> David Ecker > >>>>> > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 15:09:08 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:09:08 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37674.9090408@ecker-software.de> Hi, [quote] Depending on the media and the block device driver design, individual sector writes may not be atomic. A physical sector in typical devices is 512 bytes. In most cases, physical sector writes are atomic (either completely written, or not modified at all). A truly reliable file system, however, cannot count on this. [/quote] In most cases it works but some way of validating the data is needed if you want ultra reliability, which I don't need. If the last few messages a few seconds before an immediate shutdown are lost but all other messages are send correctly afterwards then that would be OK in my case. I'll just test version 2.21.5 with the altered open behauvior. The disk based queue-array developed by myself is just a fallback solution if the disk-based queue doesn't work with an immediate shutdown. David Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. >> > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:11:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:11:09 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Sorry, I overlooked this mail in the big bunch of messages. That's good reasoning. To cover these scenarios, we need to do everything with syncing. This also means that you can not use any of the disk-assisted modes, because in these modes we always try to keep things in memory in order to save writes. So while you have convinced me things can go wrong, I'd still say that is is very unusual (at least very costly) to care for all these things. But, of course, there are situations where it is needed. I'll probably see that I provide a facility to open files in "always sync" mode, but that for sure will not be the default setting ;) But even with the fast solid state disks (and similar methods) you mention, I think there will be a severe impact on performance because everything now needs to go through two write (data+metadata) and two read (again, data+metadata) OS call where we currently simply update an in-memory structure. Just out of curiosity: do you expect the majority of you rollouts to be using such methods? Rainer On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > > ... And I have never heard of anybody doing serious datacenter work > > without a proper UPS. Is this *really* an issue? > > Yes. > > UPSs fail. > generators fail > power cords come loose. > power cords get unplugged by someone who thinks they are unplugging a > different system > people bump power switches on power strips. > power supplies are defective > > I had one production outage where a visiting tech pulled a power cord from > an overhead plug and dropped it on the ground, where it happened to hit > the power switch on a power strip. > > I've had high-end systems with redundant power supplies go down becouse of > faulty hardware that decided to disble both power supplies at once (it > turned out that there was a defect in the whole batch of servers, but it > took IBM several weeks to figure out what was going on) > > I've had UPS systems blow up (literally) > > I've had a datacenter go down becouse the it was running on generator > power (due to other issues), and the refueling guy filled the tank > incorrectly and got air bubbles into the fuel system, a few min later the > 500Kw diesel generator couldn't maintain constant speed and the safety > triggers kicked in and disabled it. > > it's amazing the things that happen in real-life From david at lang.hm Wed Oct 1 15:17:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:17:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. I'm replying out of order as I see things anyway > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. I think you are saying that we must use the disk-only mode which is correct. > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. absolutly!! > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) thanks. > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. given the performance gains that we have seen by eliminating syscalls, it will hurt to add these back in, even with solid-state disks. that being said, it looks like the output module is nowhere close to being the limit (when I could get a good, stable reading on it, it looked like it was eating ~15% cpu compared to the input module at 100%) so it may not make much of a difference. > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? absolutly not. I have one case I am considering (the one I am talking to you about more efficiant database writes) that would be this paranoid, but the rest of it will be optimized for speed (battery-backed disk caches on the final server, but everything else can just use ram) David Lang > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:17:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37674.9090408@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <48E37674.9090408@ecker-software.de> Message-ID: <1222867062.2682.492.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:09 +0200, David Ecker wrote: > Hi, > > [quote] > Depending on the media and the block device driver design, individual > sector writes may > not be atomic. A physical sector in typical devices is 512 bytes. In > most cases, physical > sector writes are atomic (either completely written, or not modified at > all). A truly > reliable file system, however, cannot count on this. > [/quote] > > In most cases it works exactly - in most cases. That means it does not work always. > but some way of validating the data how can you validate if there is no power and the machine is off? > is needed if > you want ultra reliability, which I don't need. If the last few messages > a few seconds before an immediate shutdown are lost but all other > messages are send correctly afterwards then that would be OK in my case. but we can not guarantee that, at least not in all cases. Let's assume the disk died in the middle of the write access. Chances are good you'll never be able to read that sector again. Using a journaling file system will help, but without it, you may just have destroyed the sector that contained the .qi file. So on next startup the .qi is either not readable at all or not pointing at the correct information. The end result can be total loss of information. This scenario is probably acceptable in your case, because it is really, really highly unlikely. But it still exists. > I'll just test version 2.21.5 with the altered open behauvior. The disk > based queue-array developed by myself is just a fallback solution if the > disk-based queue doesn't work with an immediate shutdown. If it does not work under the constraints described here, this would point to a problem in the queue implementation (I have to admit the reason to provide a capability to write periodic qi file updates was related to a scenario like this, though not thought in this extreme ;)). Rainer > > David > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > > > [snip] > > > > > >> as long as you do sector based writes (512 byte per sector, usual) you > >> can be sure that the write wasn"t partial.. Writing more than one sector > >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > >> a partial write. I'll already tested that with my devel client here. So > >> fencing each sector with a crc32 value would help detecting errors > >> during a write operation. This is actually only a problem if you are > >> writing directly to a block device like any filesystem does and yes, > >> reordering is definitly a problem. So validating the content written to > >> the disk afterwards is important. > >> > >> If writing through a filesystem reserving space in the destination file > >> beforehand actually minimizes errors since the file system table doesn't > >> have to be updated (you should also use the Flag O_NOATIME for that > >> case). See for example VMWare ESX VMDK file handling. > >> > > > > Well, first of all let me re-iterate that I do not intend to do a block > > device driver for rsyslog (but I definitely do not object getting one > > contributed ;)). > > > > Still thinking about the case and thinking about non-solid-state, > > non-internal-battery-backed-up disk, I can't see how you can be sure the > > data will be written. David just told me there are no capacitors. So if > > power fails, it fails rather quickly. So how can you be sure the disk > > will be able to finish writing that sector? Let's say the drive has > > begun to write the sector and been able to write the first 5 bytes. Now > > power fails. No capacitors, no battery-backup, so why should there be > > enough power to drive the disk write head for another 507 bytes? It the > > drives assures it can do that, it needs capacitors - doesn't it? > > > > Am I overlooking something obvious? > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 15:18:55 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:18:55 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? in an audit-grade logging system yes, in an enterprise-grade system probably not. David Lang > Anybody else with an opinion? > Rainer > > On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> David, >>>>> >>>>> the file syncing mentioned in the compatibility doc applies to the >>>>> output action, only. >>>> >>>> ouch. >>>> >>>>> The queue does never do synchronous writes - I always assumed that a >>>>> critical system would have a UPS and could never think (so far) about a >>>>> valid reason for not having it. So the queue would need to have an extra >>>>> option to do sync writes. Obviously, that's not a big deal. >>>> >>>> good >>>> >>>>> Performance, of course, will be extremely terrible with such a setup... >>>> >>>> only if you have to wait for a spinning disk to do the write. >>> >>> I agree to the rest of your argument below. But the question raised here >>> was in regard to a system without any battery backup. So I would need to >>> wait. >> >> no UPS is not nessasarily the same as no battey backup. >> >> you could use a compact flash drive and probably get better >> performance/reliability than spinning disks with no battery at all. >> >>> Even then, in the worst case, I think it would be possible that the disk >>> does only a partial write. I am not sure if that's really the case with >>> today's disk drives (which I think have capacitors to prevent this >>> scenario), but with past drives this could happen (I know all too well - >>> a few years ago that cost me a weekend ;)). >> >> current disks do not have capacitors to prevent partial writes or to flush >> their caches. but options like the linux ext3 data-journaled make it so >> that you have your data in the journal safely, and the various solid-state >> options solve that problem. >> >> David Lang >> >>> Rainer >>> >>>> >>>> this is the same problem that databases have. they need to guarentee that >>>> once the database tells the writing program that the data is written it >>>> will be there even if the system looses power immediatly. >>>> >>>> if you run a database on standard desktop hardware (and it doesn't have >>>> this safety disabled) you cannot do more then about 80 writes/second. If >>>> you upgrade to the super speedy 15K rpm drives you can do ~160 >>>> writes/second. >>>> >>>> given that you need to write the data + metadata it gets even uglier, so >>>> what the databases do (and some journaling filesystems) is to write a log >>>> that says what they are going to do, sync that, and then later write the >>>> data to the actual files (updating the journal when they complete the >>>> write) >>>> >>>> it sounds like you order your write correctly for a disk-based queue, but >>>> you would need the option of issuing the syncs (probably when you do the >>>> checkpoints) >>>> >>>> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >>>> wrong flash drive), the fact that you need to do four writes per log entry >>>> (data to queue, metadata to queue, data to output, update metadata for >>>> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >>>> to seeking as well) >>>> >>>> however, with the correct drive to write to (say a $2,400 80G fusion-io >>>> flash card that can do ~100k IO ops/sec) you should be able to sustain >>>> 20,000 logs/sec. >>>> >>>> realisticly very few people need the sustained write capacity that you >>>> would get from such a setup. but if you go with a $500-$700 raid card with >>>> a battery-backed cache you get very similar performance, but with some >>>> possibility that you can't sustain it forever. >>>> >>>> David Lang >>>> >>>>> Rainer >>>>> >>>>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am looking for a failsafe solution to store syslog messages localy >>>>>>> until they could be send later. I already looked at the disk based >>>>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>>>> just power down the system immediatly actually loosing the whole queue. >>>>>> >>>>>> are you sure about the disk based queue? >>>>>> >>>>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>>>> can be set to do a commit of the metadata after each message. >>>>>> >>>>>> Disk Queues >>>>>> >>>>>> Disk queues use disk drives for buffering. The important fact is that the >>>>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>>>> this queue mode is not recommended. It is useful if log data is so >>>>>> important that it must not be lost, even in extreme cases. >>>>>> >>>>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>>>> its individual file. Files are named with a prefix (set via the >>>>>> "$QueueFilename" config directive) and followed by a 7-digit >>>>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>>>> default, a different size can be set via the"$QueueMaxFileSize" >>>>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>>>> always writes one complete queue entry, even if it violates the size >>>>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>>>> then the configured size. Each chunk also has a different size for the >>>>>> same reason. If you observe different chunk sizes, you can relax: this is >>>>>> not a problem. >>>>>> >>>>>> Writing in chunks is used so that processed data can quickly be deleted >>>>>> and is free for other uses - while at the same time keeping no artificial >>>>>> upper limit on disk space used. If a disk quota is set (instructions >>>>>> further below), be sure that the quota/chunk size allows at least two >>>>>> chunks to be written. Rsyslog currently does not check that and will fail >>>>>> miserably if a single chunk is over the quota. >>>>>> >>>>>> Creating new chunks costs performance but provides quicker ability to free >>>>>> disk space. The 10mb default is considered a good compromise between these >>>>>> two. However, it may make sense to adapt these settings to local policies. >>>>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>>>> make sense to use a 2gb (or even larger) chunk size. >>>>>> >>>>>> Please note, however, that the disk queue by default does not update its >>>>>> housekeeping structures every time it writes to disk. This is for >>>>>> performance reasons. In the event of failure, data will still be lost >>>>>> (except when manually is mangled with the file structures). However, disk >>>>>> queues can be set to write bookkeeping information on checkpoints (every n >>>>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>>>> interval is set to one, no data can be lost, but the queue is >>>>>> exceptionally slow. >>>>>> >>>>>> Each queue can be placed on a different disk for best performance and/or >>>>>> isolation. This is currently selected by specifying different >>>>>> $WorkDirectory config directives before the queue creation statement. >>>>>> >>>>>> To create a disk queue, use the "$QueueType Disk" config >>>>>> directive. Checkpoint intervals can be specified via >>>>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> you also need to specificly enable syncing (from >>>>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>>>> >>>>>> Output File Syncing >>>>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>>>> such, it retained stock syslogd's default of syncing every file write if >>>>>> not specified otherwise (by placing a dash in front of the output file >>>>>> name). While this was a useful feature in past days where hardware was >>>>>> much less reliable and UPS seldom, this no longer is useful in today's >>>>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>>>> writes files around 50 *times* slower than without it. It also affects >>>>>> overall system performance due to the high IO activity. In rsyslog v3, >>>>>> syncing has been turned off by default. This is done via a specific >>>>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>>>> default. In order to enable file syncing, the administrator must specify >>>>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>>>> syncing only happens in some installations where the administrator >>>>>> actually wanted that (performance-intense) feature. In the fast majority >>>>>> of cases (if not all), this dramatically increases rsyslogd performance >>>>>> without any negative effects. >>>>>> >>>>>> >>>>>> >>>>>>> I already looked at queue.c and it seemed to me that both queues were >>>>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>>>> an immediate power down of the system is the major failure which will >>>>>>> occure pretty often I need to create a soltution there. >>>>>> >>>>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>>>> in on the disk safely (assuming you have hardware that supports this) and >>>>>> a power-off won't affect it. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> >>>>>>> Did you already start to develop something addressing that problem? >>>>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>>>> would like afterwards. >>>>>>> >>>>>>> bye >>>>>>> David Ecker >>>>>>> >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 15:20:29 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:20:29 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E3791D.3050304@ecker-software.de> Hi, this is already the second version of this systems we develop. Not being able to do error analysis beacause of missing log data was one of the big problems including filesystem crashes. Having all logs and being able to proof the cause also helps a lot afterwards not only for creating a workaround for the incident but also to proof that the Service Level Agreement wasn't violated. bye David Ecker Rainer Gerhards schrieb: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. > > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. > > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) > > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. > > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? > > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >>> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:24:33 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:24:33 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? one possible thing is tht if the write has not completed then the system sending you the logs has not received confirmation that you have the log yet, so they are the ones responsible for it. it's only after you acknowledge the message (via relp or equivalent) that you are required to not loose the log message. with ext3 and data=journaled you can do this. the writes to the journal are done by the filesystem in such a way that they can be considered atomic (either they happen and are reliable, or they can be treated as if they never happened), once that write (or rather the sync on that write) has completed the data is safe. the filesystem will later modify the actual sector on disk, but it jumps through hoops to make that safe from power outages. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:24:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:24:24 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. mhhh... This sounds like you still may be in trouble if the file system crashes (as the queue files are stored inside that system). Rainer From david at lang.hm Wed Oct 1 15:31:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:31:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). pick the right filesystem and you should still be safe ext3+data=journaled is safe databases do the journaling themselves and like to store their journal files on ext2. you can use a log-structured filesystem that never overwrites data. it writes the new sector to a new place on disk then modifies pointers to tell the filesystem that the data is in the new place instead of the old place (the results in horrible fragmentation in many cases, but it's _very_ safe) some of this can be done without doing a sync for every write with barriers, which can guarentee that a write before the barrier takes place before any writes after the barrier. this lets you do things like modify the queue and then (after you know the data is safe) modify the metadata to indicate that the new data is there. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:31:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:31:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > one possible thing is tht if the write has not completed then the system > sending you the logs has not received confirmation that you have the log > yet, so they are the ones responsible for it. I concur. But as I understood the scenario here, the log messages are emited from a process *inside* the failing machine. So that process fails, too, and we do not have any interim that has a copy of the data. So if it is not persisted to disk, it is lost. Anyhow, this requirement has been relaxed in later posting. Also note that I was just thinking about the physical layer, considering a single physical write - far away from rsyslog. > it's only after you acknowledge the message (via relp or equivalent) that > you are required to not loose the log message. I concur and this is how RELP handles this. Well, actually I think there currently is a very slight (but still existing) window of exposure. I think RELP acks when the message is submited to the queue engine. That does not necessarily mean the message is already present on disk. Also, I think some mild duplication of messages may happen with RELP in a power fail scenario. It is not doing a two-phase commit, thought it tries very hard to get a perfect understanding of what is written and what not. I could check this with spec/code, but I think this is not justified at this point in time ;) > > with ext3 and data=journaled you can do this. the writes to the journal > are done by the filesystem in such a way that they can be considered > atomic (either they happen and are reliable, or they can be treated as if > they never happened), once that write (or rather the sync on that write) > has completed the data is safe. the filesystem will later modify the > actual sector on disk, but it jumps through hoops to make that safe from > power outages. > sure, definitely. And I assume a sync'ed write will return only after all this has happened. So journaled ext3 should be able to solve the problem described here. Rainer From david at lang.hm Wed Oct 1 15:35:01 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:35:01 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> Even then, in the worst case, I think it would be possible that the disk >> does only a partial write. I am not sure if that's really the case with >> today's disk drives (which I think have capacitors to prevent this >> scenario), but with past drives this could happen (I know all too well - >> a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. actually, I need to correct my answer. I know that disks do not have capacitors large enough to write their buffer. I'm pretty sure that they don't have capacitors large enough to write an entire track. but they may have capacitors large enough to finish the sector they are on before stopping (considering the number of sectors in a track nowdays this is a _very_ sort time) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:34:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:34:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: > some of this can be done without doing a sync for every write with > barriers, which can guarentee that a write before the barrier takes place > before any writes after the barrier. this lets you do things like modify > the queue and then (after you know the data is safe) modify the metadata > to indicate that the new data is there. That sounds interesting - can you point me to some API documentation? A good place to do that would be inside the queue after the qi file has been persisted. That could reduce performance toll of synchronous writes. Especially if loss of a few messages is acceptable (as David Ecker just said). Rainer From david at ecker-software.de Wed Oct 1 15:35:18 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:35:18 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37C96.3080904@ecker-software.de> I know. I used reiserfs before. The client systems were turned off like 4 times a day (average). At that point we had only 60 clients running. At least once a day one client needed to be reimaged since the filesystem was broke. The system itself wrote too much to the local disk during operation. So the failure rate was actually below 0,5% but it keept a constand buzz for the support center. I just was lucky that I implemented a failback image so that the systems could reimage themselfs without anybody going there. In this since the I would do it like this: Since the filesystem would only store the queue files of the syslog and nothing else I would check the filesystem at startup. If it's broken I would reformat the fs automtically. So that would leave me with a loss of 0,5% of all logs. Thats a lot better than nothing. I just have to make sure, that a rollback of the fs leaves the syslog data queue intact. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. >> > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:40:30 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:40:30 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > >> one possible thing is tht if the write has not completed then the system >> sending you the logs has not received confirmation that you have the log >> yet, so they are the ones responsible for it. > > I concur. But as I understood the scenario here, the log messages are > emited from a process *inside* the failing machine. So that process > fails, too, and we do not have any interim that has a copy of the data. > So if it is not persisted to disk, it is lost. Anyhow, this requirement > has been relaxed in later posting. > > Also note that I was just thinking about the physical layer, considering > a single physical write - far away from rsyslog. > >> it's only after you acknowledge the message (via relp or equivalent) that >> you are required to not loose the log message. > > I concur and this is how RELP handles this. Well, actually I think there > currently is a very slight (but still existing) window of exposure. I > think RELP acks when the message is submited to the queue engine. That > does not necessarily mean the message is already present on disk. right now it doesn't make a difference (since the queue doesn't sync the data), but if/when this is added checking that the call to queue the data doesn't return until after that point should be done. > Also, > I think some mild duplication of messages may happen with RELP in a > power fail scenario. It is not doing a two-phase commit, thought it > tries very hard to get a perfect understanding of what is written and > what not. I could check this with spec/code, but I think this is not > justified at this point in time ;) no matter what checks you do, it's always possible for things to fail after succeeding on doing the work and before the acknowlegement gets back to the sender (with relp, if the receiving machine sends the acknowledgement, but the sending machine crashes before fully processing it, the receiving machine has no way of detecting this) on the other hand, this is fairly easy to deal with by making messages include a sequence number to guarentee that they are all unique, and then have something filter out duplicates later. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:39:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:39:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868396.2682.511.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:35 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, david at lang.hm wrote: > > > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > >> Even then, in the worst case, I think it would be possible that the disk > >> does only a partial write. I am not sure if that's really the case with > >> today's disk drives (which I think have capacitors to prevent this > >> scenario), but with past drives this could happen (I know all too well - > >> a few years ago that cost me a weekend ;)). > > > > current disks do not have capacitors to prevent partial writes or to flush > > their caches. but options like the linux ext3 data-journaled make it so > > that you have your data in the journal safely, and the various solid-state > > options solve that problem. > > actually, I need to correct my answer. > > I know that disks do not have capacitors large enough to write their > buffer. absolutely > > I'm pretty sure that they don't have capacitors large enough to write an > entire track. > but they may have capacitors large enough to finish the sector they are on > before stopping (considering the number of sectors in a track nowdays this > is a _very_ sort time) *That* I had expected. Especially if you think about the low-level sync marks which otherwise could be affected. I think loss of disk could otherwise be the extreme result of a power failure (or at least loss of disk track). But you never know what a failing write arm does... So I would think (but have no evidence) that current disk drives have capacitors large enough to finish the write AND shut down the write mechanism in an orderly manner. As you say, there are "not many electrons" required to ensure this. Rainer From david at lang.hm Wed Oct 1 15:42:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:42:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: >> some of this can be done without doing a sync for every write with >> barriers, which can guarentee that a write before the barrier takes place >> before any writes after the barrier. this lets you do things like modify >> the queue and then (after you know the data is safe) modify the metadata >> to indicate that the new data is there. > > That sounds interesting - can you point me to some API documentation? A > good place to do that would be inside the queue after the qi file has > been persisted. That could reduce performance toll of synchronous > writes. Especially if loss of a few messages is acceptable (as David > Ecker just said). I'll try to find it (I've seen it discussed on the linux-kernel mailing list by the filesystem developers, but I don't know how it's exposed to userspace) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:46:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:46:52 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868812.2682.515.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:40 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > > > >> one possible thing is tht if the write has not completed then the system > >> sending you the logs has not received confirmation that you have the log > >> yet, so they are the ones responsible for it. > > > > I concur. But as I understood the scenario here, the log messages are > > emited from a process *inside* the failing machine. So that process > > fails, too, and we do not have any interim that has a copy of the data. > > So if it is not persisted to disk, it is lost. Anyhow, this requirement > > has been relaxed in later posting. > > > > Also note that I was just thinking about the physical layer, considering > > a single physical write - far away from rsyslog. > > > >> it's only after you acknowledge the message (via relp or equivalent) that > >> you are required to not loose the log message. > > > > I concur and this is how RELP handles this. Well, actually I think there > > currently is a very slight (but still existing) window of exposure. I > > think RELP acks when the message is submited to the queue engine. That > > does not necessarily mean the message is already present on disk. > > right now it doesn't make a difference (since the queue doesn't sync the > data), but if/when this is added checking that the call to queue the data > doesn't return until after that point should be done. I just checked the code. Actually, librelp already ensures this. It acks only after the callback has successfully completed. > > > Also, > > I think some mild duplication of messages may happen with RELP in a > > power fail scenario. It is not doing a two-phase commit, thought it > > tries very hard to get a perfect understanding of what is written and > > what not. I could check this with spec/code, but I think this is not > > justified at this point in time ;) > > no matter what checks you do, it's always possible for things to fail > after succeeding on doing the work and before the acknowlegement gets back > to the sender (with relp, if the receiving machine sends the > acknowledgement, but the sending machine crashes before fully processing > it, the receiving machine has no way of detecting this) RELP has a "Window of uncertainty", in which peers negotiate where to restart. But power failure was not on my spec list, so this only works if the shutdown was relatively clean. > > on the other hand, this is fairly easy to deal with by making messages > include a sequence number to guarentee that they are all unique, and then > have something filter out duplicates later. well said - among others one reason why RELP currently is not receiving more attention that just letting it mature... Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 15:59:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:59:34 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37C96.3080904@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> Message-ID: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> David (Ecker ;)), I'd appreciate if you let me know your results after you have tested with $..QueueCheckpointInterval 1 Rainer From david at lang.hm Wed Oct 1 16:10:38 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 07:10:38 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David (Ecker ;)), > > I'd appreciate if you let me know your results after you have tested > with $..QueueCheckpointInterval 1 also, try mounting the filesystem -o sync this should make everything on it a sync write without Rainer having to change his code (won't work for other situations, but will definantly work for a test) David Lang From david at ecker-software.de Wed Oct 1 16:19:45 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 16:19:45 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E38701.80707@ecker-software.de> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My guess is, that the O_DIRECT in combination with the O_SYNC flag (turning of cache) will have an impact. I won't be able to test it tomorrow and since Friday is a vaction in germany I'll be able to tell you on monday the results of my tests. Thanks for your help. I was actually positivly suprised by the constructiveness of our discussion. David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > >> David (Ecker ;)), >> >> I'd appreciate if you let me know your results after you have tested >> with $..QueueCheckpointInterval 1 >> > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 1 16:32:28 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:32:28 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? > > Anybody else with an opinion? Yes, I have an opinion: no. Enterprise-grade applications need to make the best effort they can to be fault-tolerant and reliable, but we're following an exponential curve - at some point, you _have_ to give over to the hardware and OS engineers to do their job properly. Don't reinvent the hammer & chisel to reinvent the wheel - software can _never_ account for all hardware failures and should never be expected to do so. RB From rgerhards at hq.adiscon.com Wed Oct 1 16:39:36 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 16:39:36 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> Message-ID: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Hi RB, I guess you've seen in the discussion (well, if you read all these mails, I just noticed how many they were...) I am coming from the same camp. But after all the dust settles, it looks like a very minor change to rsyslog to allow the OS to do what it is expected to do. The queue logic already cares about proper sequence (at least it should), the only thing that is missing is to tell the OS to sync when we need it. The spot to do so is also well defined, that is when the qi file is persisted (because that completes the "queue transaction"). I will verify with the code, but my current guess is that no more than 10 lines of code will be needed to support this functionality. If so, I think it is worth it. More complex questions may arise (I have some on my mind) and that will probably go beyond a general-purpose solution. Rainer On Wed, 2008-10-01 at 08:32 -0600, RB wrote: > On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > > going back to the higher layer: do you say that immediate power failure > > is a case that you consider needed to be addressed in an enterprise > > logging system? > > > > Anybody else with an opinion? > > Yes, I have an opinion: no. Enterprise-grade applications need to > make the best effort they can to be fault-tolerant and reliable, but > we're following an exponential curve - at some point, you _have_ to > give over to the hardware and OS engineers to do their job properly. > Don't reinvent the hammer & chisel to reinvent the wheel - software > can _never_ account for all hardware failures and should never be > expected to do so. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Wed Oct 1 16:40:21 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:40:21 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. Having all logs and being > able to proof the cause also helps a lot afterwards not only for > creating a workaround for the incident but also to proof that the > Service Level Agreement wasn't violated. Somewhat randomly selecting one to respond to - you guys generated a lot of chatter. My view as both a software and hardware engineer on this is simple: never try to solve a hardware problem with a software solution. You will never win. It is possible to account for many edge cases, but no matter how paranoid you may be, hardware always trumps software - it has the last word, regardless of what you may try to do. RB From aoz.syn at gmail.com Wed Oct 1 16:50:24 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:50:24 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> > I guess you've seen in the discussion (well, if you read all these > mails, I just noticed how many they were...) ;-) I was wondering why my hip was rattling during the entire commute today. > persisted (because that completes the "queue transaction"). I will > verify with the code, but my current guess is that no more than 10 lines > of code will be needed to support this functionality. If so, I think it > is worth it. If it's something you can solve without worrying about block-level writes and whether the underlying drive (if indeed there is even a "drive") has a battery-backed cache or sufficient capacitor charge to write your data, I'm all for it. Even better if it's POSIX. Although a block driver and other filesystem-bypassing solutions may be interesting in limited cases, I'd rather not see anyone stab their wife over it. From rgerhards at hq.adiscon.com Wed Oct 1 17:18:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:18:11 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> David, On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David (Ecker ;)), > > > > I'd appreciate if you let me know your results after you have tested > > with $..QueueCheckpointInterval 1 > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) I after all this discussion I will take a few minutes to see if I can capture results in code. There are two things and I guess you have an opinion on that. First of all, the doc I have on fsync() is a bit cryptic. It says: ---- fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)). ---- so it looks like fsync() is sufficient to flush both the data as well as the size information (so that data is not only present but can actually be read ;). But in the next paragraph, the man page says: ---- Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed. ---- which sound much like the opposite. I know there are many flavours of *nix and even though this is a POSIX call, things may be different in different OS... Do you have an opinion of we can assume a call to fsync() on the descriptor in questions solves the issue? This would probably avoid the need to open with always synced writes and thus could offer a somewhat better performance. Also, the thought about O_NOATIME looks interesting. For internal queue files, I think the last access time is of very limited interest, so it would probably be useful to always specify this option (at least by default). Feedback appreciated. Rainer From david at lang.hm Wed Oct 1 17:41:11 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:41:11 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) > > I after all this discussion I will take a few minutes to see if I can > capture results in code. There are two things and I guess you have an > opinion on that. > > First of all, the doc I have on fsync() is a bit cryptic. It says: > > ---- > fsync() transfers ("flushes") all modified in-core data of (i.e., > modified buffer cache pages for) the file referred to by the file > descriptor fd to the disk device (or other permanent storage device) > where that file resides. The call blocks until the device reports that > the transfer has completed. It also flushes metadata information > associated with the file (see stat(2)). > ---- > > so it looks like fsync() is sufficient to flush both the data as well as > the size information (so that data is not only present but can actually > be read ;). But in the next paragraph, the man page says: > > ---- > Calling fsync() does not necessarily ensure that the entry in the > directory containing the file has also reached disk. For that an > explicit fsync() on a file descriptor for the directory is also needed. > ---- > > which sound much like the opposite. I know there are many flavours of > *nix and even though this is a POSIX call, things may be different in > different OS... > > Do you have an opinion of we can assume a call to fsync() on the > descriptor in questions solves the issue? This would probably avoid the > need to open with always synced writes and thus could offer a somewhat > better performance. my understanding is that you may need to call both. if you have pre-allocated the file you would not need to call fsync on the directory, if you are dynamicly extending it you will need to. > Also, the thought about O_NOATIME looks interesting. For internal queue > files, I think the last access time is of very limited interest, so it > would probably be useful to always specify this option (at least by > default). in this case noatime won't make much difference. normally you want it so that when you read a file it doesn't have to update the 'last accessed' time in the directory, but since we are writing to these just about every time we access them we are having to update the 'last modified' time there anyway. I am assuming that even though the data is being written to the disk queue file, it's still available in memory so you aren't having to do a disk read to read the message from the queue. If this is not the case then noatime could make a difference if the file is opened O_SYNC becouse the read would then require the directory update to be synced before you could do anything else. If you are dong the fsync calls yourself it shouldn't matter much becouse the atime update will (probably) not get pushed out to disk before you are going back and modifying the queue to indicate that you wrote the message to the output (when the 'last modified' times will get updated) David Lang > Feedback appreciated. > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 17:43:52 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:43:52 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E38701.80707@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > of cache) will have an impact. O_DIRECT is doing very different things. I don't think you need to worry about those things, having the data not go into the OS cache is a drawback not an advantage becouse it means that when you go to pull the data back out of the file it will need to actually touch disk. it also imposes significant alignment issues on the application that I don't think you want to have to desl with. David Lang > I won't be able to test it tomorrow and since Friday is a vaction in > germany I'll be able to tell you on monday the results of my tests. > > Thanks for your help. I was actually positivly suprised by the > constructiveness of our discussion. > David Ecker > > david at lang.hm schrieb: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >>> >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 17:44:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:44:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > > of cache) will have an impact. > > O_DIRECT is doing very different things. I don't think you need to worry > about those things, having the data not go into the OS cache is a drawback > not an advantage becouse it means that when you go to pull the data back > out of the file it will need to actually touch disk. it also imposes > significant alignment issues on the application that I don't think you > want to have to desl with. plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... Rainer From david at lang.hm Wed Oct 1 17:45:16 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:45:16 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. Having all logs and being >> able to proof the cause also helps a lot afterwards not only for >> creating a workaround for the incident but also to proof that the >> Service Level Agreement wasn't violated. > > Somewhat randomly selecting one to respond to - you guys generated a > lot of chatter. > > My view as both a software and hardware engineer on this is simple: > never try to solve a hardware problem with a software solution. You > will never win. It is possible to account for many edge cases, but no > matter how paranoid you may be, hardware always trumps software - it > has the last word, regardless of what you may try to do. I'm not understanding the point you are trying to make. are you saying that it's a bad idea to try and have an option to do the syncing we are talking about for the queue? David Lang From david at lang.hm Wed Oct 1 17:48:57 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:48:57 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I guess you've seen in the discussion (well, if you read all these >> mails, I just noticed how many they were...) > > ;-) I was wondering why my hip was rattling during the entire commute today. > >> persisted (because that completes the "queue transaction"). I will >> verify with the code, but my current guess is that no more than 10 lines >> of code will be needed to support this functionality. If so, I think it >> is worth it. > > If it's something you can solve without worrying about block-level > writes and whether the underlying drive (if indeed there is even a > "drive") has a battery-backed cache or sufficient capacitor charge to > write your data, I'm all for it. Even better if it's POSIX. > > Although a block driver and other filesystem-bypassing solutions may > be interesting in limited cases, I'd rather not see anyone stab their > wife over it. I agree that trying to bypass the filesystem is highly questionable, and not something for a core change (as always a contributed vrsion can be tested to see if it makes a difference) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 17:58:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:58:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: <1222876701.2682.541.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:48 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, RB wrote: > > >> I guess you've seen in the discussion (well, if you read all these > >> mails, I just noticed how many they were...) > > > > ;-) I was wondering why my hip was rattling during the entire commute today. > > > >> persisted (because that completes the "queue transaction"). I will > >> verify with the code, but my current guess is that no more than 10 lines > >> of code will be needed to support this functionality. If so, I think it > >> is worth it. > > > > If it's something you can solve without worrying about block-level > > writes and whether the underlying drive (if indeed there is even a > > "drive") has a battery-backed cache or sufficient capacitor charge to > > write your data, I'm all for it. Even better if it's POSIX. > > > > Although a block driver and other filesystem-bypassing solutions may > > be interesting in limited cases, I'd rather not see anyone stab their > > wife over it. > > I agree that trying to bypass the filesystem is highly questionable, and > not something for a core change (as always a contributed vrsion can be > tested to see if it makes a difference) Just FYI: preliminary analysis indicates that is probably around the 10 lines of code that need to be added in stream.c, maybe a few more. The idea is that I can set a flag similar to O_SYNC on stream creation but then sync when the "atomic" writes are done. This may save some few cycles over an O_SYNC open(). However, there need to be a few config settings, which in turn need to be passed down to the queue and stream class. That adds more code, maybe around 100 lines (the config interface needs to be redone, thus the many LOC required, it's on the todo list...). An alternative is to use a simple global $AllWritesSync on/off option, which would probably be sufficient and cut down changes required. For a robust implementation, some more analysis is required (including thinking about the implications of fsync()). So it is not totally trivial, but well doable. I am just not sure if I'll do it immediately, there are many things in the work queue. Testing effort is probably much bigger than implementation effort, there are so many cases to check out... Rainer From aoz.syn at gmail.com Wed Oct 1 18:00:10 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 10:00:10 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> > I'm not understanding the point you are trying to make. > > are you saying that it's a bad idea to try and have an option to do the > syncing we are talking about for the queue? Sync: not as long as it's optional. Worrying about sector writes and capacitors/battery-backed cache on the underlying drives: yes. Unless rsyslog starts writing to raw devices, it is my opinion that once it reasonably hands the data off to the filesystem, it becomes a kernel and/or hardware problem and unnecessarily complex for a userspace application to govern. I'm all for anything rsyslog can do to encourage proper behavior (like calling sync()) without getting into the kernel/hardware space. RB From david at lang.hm Wed Oct 1 18:07:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 09:07:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I'm not understanding the point you are trying to make. >> >> are you saying that it's a bad idea to try and have an option to do the >> syncing we are talking about for the queue? > > Sync: not as long as it's optional. Worrying about sector writes and > capacitors/battery-backed cache on the underlying drives: yes. Unless > rsyslog starts writing to raw devices, it is my opinion that once it > reasonably hands the data off to the filesystem, it becomes a kernel > and/or hardware problem and unnecessarily complex for a userspace > application to govern. I'm all for anything rsyslog can do to > encourage proper behavior (like calling sync()) without getting into > the kernel/hardware space. Ok, we are on the same page then. in case I confused anyone, the reason I went into detail on the hardware side of things was to explain how proper hardware selection could result in good performance while poor hardware selection would result is dismal performance (and to give people who aren't familiar with the options some hints as to what they could do) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 18:51:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 18:51:52 +0200 Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> [going back to the list with Lorenzo's permission] Hi Lorenzo, finally... I did not realize that the queue is running in direct mode, but now I saw it and can reproduce a failure. That should make things much better. Anyhow, it is very unusual that the main message queue runs in direct mode, so I guess it is a side-effect of a config file. In those files you sent me, I don't see it, but I see a .d directory is included. May there be something? In any case, I hope I will be able to fix the bug now that I can reproduce it. Looks like a productive day today :) Thanks for your persistence, Rainer > -----Original Message----- > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > Sent: Wednesday, October 01, 2008 1:46 PM > To: Rainer Gerhards > Subject: Re: [rsyslog] abort case > > Sorry for the delay, I've just run helgrind's head, bau as you can see, > it > crashed almost as soon as I started. > > The log is enclosed; I'm now checking-out head and will retry. > > Yours, > > lorenzo > > RG> I have created a new version with one slight change, to be found in > RG> the helgrind branch. > RG> > RG> There is also a new valgrind tool called drd inside the valgrind > RG> development tree. I think you already downloaded that tree. If so, > RG> could you please replace > RG> > RG> Valgrind --tool=helgrind .. rsyslogd ... > RG> > RG> with > RG> > RG> Valgrind --tool=drd .. rsyslogd ... > RG> > RG> Drd does an even better job than helgrind. I also changed the > source > RG> to remove some debug-system related warning, which otherwise would > RG> clutter up the error message. But unfortunately, in my lab I did > not > RG> find any more problems, except for the small change I mentioned. > But > RG> that one affects program termination (and in a very subtle way), so > it > RG> should not change anything for you. I'd still be interested in a > new > RG> run, including debug info, from you. > RG> > > > +-------------------------+-------------------------------------------- > --+ > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > | > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > Vergata" | > | | Via O. Raimondo 18 ** I-00173 ROMA ** > ITALY | > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > | > +-------------------------+-------------------------------------------- > --+ From rgerhards at hq.adiscon.com Wed Oct 1 19:02:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 19:02:15 +0200 Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 01, 2008 6:52 PM > To: Lorenzo M. Catucci > Cc: rsyslog-users > Subject: Re: [rsyslog] abort case > > [going back to the list with Lorenzo's permission] > > Hi Lorenzo, > > finally... I did not realize that the queue is running in direct mode, > but now I saw it and can reproduce a failure. That should make things > much better. Anyhow, it is very unusual that the main message queue > runs in direct mode, so I guess it is a side-effect of a config file. > In those files you sent me, I don't see it, but I see a .d directory is > included. May there be something? In any case, I hope I will be able to > fix the bug now that I can reproduce it. > > Looks like a productive day today :) > > Thanks for your persistence, > Rainer > > > -----Original Message----- > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > > Sent: Wednesday, October 01, 2008 1:46 PM > > To: Rainer Gerhards > > Subject: Re: [rsyslog] abort case > > > > Sorry for the delay, I've just run helgrind's head, bau as you can > see, > > it > > crashed almost as soon as I started. > > > > The log is enclosed; I'm now checking-out head and will retry. > > > > Yours, > > > > lorenzo > > > > RG> I have created a new version with one slight change, to be found > in > > RG> the helgrind branch. > > RG> > > RG> There is also a new valgrind tool called drd inside the valgrind > > RG> development tree. I think you already downloaded that tree. If > so, > > RG> could you please replace > > RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... > > RG> > > RG> with > > RG> > > RG> Valgrind --tool=drd .. rsyslogd ... > > RG> > > RG> Drd does an even better job than helgrind. I also changed the > > source > > RG> to remove some debug-system related warning, which otherwise > would > > RG> clutter up the error message. But unfortunately, in my lab I did > > not > > RG> find any more problems, except for the small change I mentioned. > > But > > RG> that one affects program termination (and in a very subtle way), > so > > it > > RG> should not change anything for you. I'd still be interested in a > > new > > RG> run, including debug info, from you. > > RG> > > > > > > +-------------------------+------------------------------------------ > -- > > --+ > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > > | > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > > Vergata" | > > | | Via O. Raimondo 18 ** I-00173 ROMA ** > > ITALY | > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > > | > > +-------------------------+------------------------------------------ > -- > > --+ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:05:31 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:05:31 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> [going back to the list with Lorenzo's permission] RG> [...] RG> RG> finally... I did not realize that the queue is running in direct mode, RG> but now I saw it and can reproduce a failure. That should make things RG> much better. Anyhow, it is very unusual that the main message queue RG> runs in direct mode, so I guess it is a side-effect of a config file. RG> No, they are really the only config files present on the server (rsyslog.conf is in /etc, the other one is the only thing inside /etc/rsyslog.d/) Really, I wouldn't go back to a saner configuration until we zeroed on the bug... still, nowhere I touched the MainQueue options! Hope to hear from you soon! Yours, lorenzo RG> RG> In those files you sent me, I don't see it, but I see a .d directory RG> is included. May there be something? In any case, I hope I will be RG> able to fix the bug now that I can reproduce it. RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:16:23 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:16:23 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: It just crashed under drd (sending both direct and through the list, it should get to you!). I've restarted under helgrind, just for the sake of not wasting the upcoming night... Hear you tomorrow! Yours, lorenzo On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) RG> RG> Rainer RG> RG> > -----Original Message----- RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards RG> > Sent: Wednesday, October 01, 2008 6:52 PM RG> > To: Lorenzo M. Catucci RG> > Cc: rsyslog-users RG> > Subject: Re: [rsyslog] abort case RG> > RG> > [going back to the list with Lorenzo's permission] RG> > RG> > Hi Lorenzo, RG> > RG> > finally... I did not realize that the queue is running in direct mode, RG> > but now I saw it and can reproduce a failure. That should make things RG> > much better. Anyhow, it is very unusual that the main message queue RG> > runs in direct mode, so I guess it is a side-effect of a config file. RG> > In those files you sent me, I don't see it, but I see a .d directory is RG> > included. May there be something? In any case, I hope I will be able to RG> > fix the bug now that I can reproduce it. RG> > RG> > Looks like a productive day today :) RG> > RG> > Thanks for your persistence, RG> > Rainer RG> > RG> > > -----Original Message----- RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] RG> > > Sent: Wednesday, October 01, 2008 1:46 PM RG> > > To: Rainer Gerhards RG> > > Subject: Re: [rsyslog] abort case RG> > > RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can RG> > see, RG> > > it RG> > > crashed almost as soon as I started. RG> > > RG> > > The log is enclosed; I'm now checking-out head and will retry. RG> > > RG> > > Yours, RG> > > RG> > > lorenzo RG> > > RG> > > RG> I have created a new version with one slight change, to be found RG> > in RG> > > RG> the helgrind branch. RG> > > RG> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind RG> > > RG> development tree. I think you already downloaded that tree. If RG> > so, RG> > > RG> could you please replace RG> > > RG> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... RG> > > RG> RG> > > RG> with RG> > > RG> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... RG> > > RG> RG> > > RG> Drd does an even better job than helgrind. I also changed the RG> > > source RG> > > RG> to remove some debug-system related warning, which otherwise RG> > would RG> > > RG> clutter up the error message. But unfortunately, in my lab I did RG> > > not RG> > > RG> find any more problems, except for the small change I mentioned. RG> > > But RG> > > RG> that one affects program termination (and in a very subtle way), RG> > so RG> > > it RG> > > RG> should not change anything for you. I'd still be interested in a RG> > > new RG> > > RG> run, including debug info, from you. RG> > > RG> RG> > > RG> > > RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione RG> > > | RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor RG> > > Vergata" | RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** RG> > > ITALY | RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 RG> > > | RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > _______________________________________________ RG> > rsyslog mailing list RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog RG> > http://www.rsyslog.com RG> _______________________________________________ RG> rsyslog mailing list RG> http://lists.adiscon.net/mailman/listinfo/rsyslog RG> http://www.rsyslog.com RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:20:48 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:20:48 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: This time I attach the file! Sorry! On Wed, 1 Oct 2008, Lorenzo M. Catucci wrote: LMC> It just crashed under drd (sending both direct and through the list, it LMC> should get to you!). I've restarted under helgrind, just for the sake of LMC> not wasting the upcoming night... LMC> LMC> Hear you tomorrow! Yours, LMC> LMC> lorenzo LMC> LMC> On Wed, 1 Oct 2008, Rainer Gerhards wrote: LMC> LMC> RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) LMC> RG> LMC> RG> Rainer LMC> RG> LMC> RG> > -----Original Message----- LMC> RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- LMC> RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards LMC> RG> > Sent: Wednesday, October 01, 2008 6:52 PM LMC> RG> > To: Lorenzo M. Catucci LMC> RG> > Cc: rsyslog-users LMC> RG> > Subject: Re: [rsyslog] abort case LMC> RG> > LMC> RG> > [going back to the list with Lorenzo's permission] LMC> RG> > LMC> RG> > Hi Lorenzo, LMC> RG> > LMC> RG> > finally... I did not realize that the queue is running in direct mode, LMC> RG> > but now I saw it and can reproduce a failure. That should make things LMC> RG> > much better. Anyhow, it is very unusual that the main message queue LMC> RG> > runs in direct mode, so I guess it is a side-effect of a config file. LMC> RG> > In those files you sent me, I don't see it, but I see a .d directory is LMC> RG> > included. May there be something? In any case, I hope I will be able to LMC> RG> > fix the bug now that I can reproduce it. LMC> RG> > LMC> RG> > Looks like a productive day today :) LMC> RG> > LMC> RG> > Thanks for your persistence, LMC> RG> > Rainer LMC> RG> > LMC> RG> > > -----Original Message----- LMC> RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] LMC> RG> > > Sent: Wednesday, October 01, 2008 1:46 PM LMC> RG> > > To: Rainer Gerhards LMC> RG> > > Subject: Re: [rsyslog] abort case LMC> RG> > > LMC> RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can LMC> RG> > see, LMC> RG> > > it LMC> RG> > > crashed almost as soon as I started. LMC> RG> > > LMC> RG> > > The log is enclosed; I'm now checking-out head and will retry. LMC> RG> > > LMC> RG> > > Yours, LMC> RG> > > LMC> RG> > > lorenzo LMC> RG> > > LMC> RG> > > RG> I have created a new version with one slight change, to be found LMC> RG> > in LMC> RG> > > RG> the helgrind branch. LMC> RG> > > RG> LMC> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind LMC> RG> > > RG> development tree. I think you already downloaded that tree. If LMC> RG> > so, LMC> RG> > > RG> could you please replace LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> with LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> Drd does an even better job than helgrind. I also changed the LMC> RG> > > source LMC> RG> > > RG> to remove some debug-system related warning, which otherwise LMC> RG> > would LMC> RG> > > RG> clutter up the error message. But unfortunately, in my lab I did LMC> RG> > > not LMC> RG> > > RG> find any more problems, except for the small change I mentioned. LMC> RG> > > But LMC> RG> > > RG> that one affects program termination (and in a very subtle way), LMC> RG> > so LMC> RG> > > it LMC> RG> > > RG> should not change anything for you. I'd still be interested in a LMC> RG> > > new LMC> RG> > > RG> run, including debug info, from you. LMC> RG> > > RG> LMC> RG> > > LMC> RG> > > LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione LMC> RG> > > | LMC> RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor LMC> RG> > > Vergata" | LMC> RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** LMC> RG> > > ITALY | LMC> RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 LMC> RG> > > | LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > _______________________________________________ LMC> RG> > rsyslog mailing list LMC> RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> > http://www.rsyslog.com LMC> RG> _______________________________________________ LMC> RG> rsyslog mailing list LMC> RG> http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> http://www.rsyslog.com LMC> RG> LMC> LMC> +-------------------------+----------------------------------------------+ LMC> | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | LMC> | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | LMC> | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | LMC> | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | LMC> +-------------------------+----------------------------------------------+ +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Thu Oct 2 15:14:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:14:34 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Thanks to the debug data provided, I could find out that the timestamps stem back to imfile and the way (sequence) in which it generates the timestamps. I will take the opportunity and fix it in a way that also improves performance. I just thought I let you know, more info follows when I have written the fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 2:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Oct 2 15:54:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:54:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> I (think) I have now fixed it, but not verified all cases. I would appreciate if you could give the new snapshot a try: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=39000a62024510cd62607 200e6100e3cd7c05005;sf=tgz Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, October 02, 2008 3:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks to the debug data provided, I could find out that the timestamps > stem back to imfile and the way (sequence) in which it generates the > timestamps. I will take the opportunity and fix it in a way that also > improves performance. I just thought I let you know, more info follows > when I have written the fix. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > > Sent: Wednesday, October 01, 2008 2:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > > > I installed the snapshot but apparently the issue persists. > > > > If there's something else that I can do, now or when you have more > > time, let me know. > > > > Regards, > > Luigi > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Oct 2 17:29:30 2008 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 2 Oct 2008 17:29:30 +0200 Subject: [rsyslog] could not load TLS module even when it's there! Message-ID: <0B1B9163138571439EAF804646F3F6AA05917715@GIBSVWIN004X.partygaming.local> Hi list, I'm setting up rsyslog to use TLS. The server-side comes up fine and I hope it's expecting encrypted traffic... Anyway, when I restart rsyslog on a client which is supposed to forward its syslogs to the logserver I see the following error message: -- ... rsyslogd: # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so'/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 : No such file or directory -- ...but funny enough: -- # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so -- What am I doing wrong?? TIA, Martin From luigi.perroti at googlemail.com Fri Oct 3 17:00:57 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Fri, 3 Oct 2008 17:00:57 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: Everything's fine here. Now I get the same subsecond part for all the events. This is how it should be since they are being generated on the same host. If it's of relevance I'm using these modules: imfile, imuxsock, imklog, ommysql. I'm also listening on a couple of additional unix sockets. Everything seems to be working fine. Thanks for the quick turnaround. Regards, Luigi From rgerhards at hq.adiscon.com Fri Oct 3 17:47:41 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 3 Oct 2008 17:47:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F28A@grfint2.intern.adiscon.com> Excellent, thanks for the confirmation. The modules are no longer relevant, they all get the time from a consistent call now (not doing this was part of the problem, if not the whole ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Friday, October 03, 2008 5:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Everything's fine here. Now I get the same subsecond part for all the > events. > This is how it should be since they are being generated on the same > host. > > If it's of relevance I'm using these modules: > imfile, imuxsock, imklog, ommysql. > I'm also listening on a couple of additional unix sockets. > Everything seems to be working fine. > > Thanks for the quick turnaround. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Oct 6 11:24:05 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:24:05 +0200 Subject: [rsyslog] HUP action Message-ID: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Hi List, I again have a backwards compatibility question. HUP, in sysklogd, means a full restart of the syslogd. This is done because the configuration may have been changed and to apply the new one, the previous one must be shut down. This is even more so the case in rsyslog, which has multiple dynamically loadable plugins and all that... rsyslog has implemented HUP processing to be compatible to sysklogd. However, in practice (as far as I know) HUP is almost always being used to close the output files after log rotation. An automatted config reaload indication usually does not happen (but operators use HUP after doing a config change). HUP, as it currently is, is an extremely expensive operation. Keep in mind that it is a full daemon restart, so, among other things, it unloads modules and tears down (TCP, RELP, TLS, ...) connections. In most cases, though, this would not be needed. I now consider changing the way HUP works. I would like to model it in a way that closes files and clears caches (where useful), but not do a full restart. Initially, I thought to use a special signal (e.g. USR2) for this purpose. But still logrotate, as a well-known example, would cause a daemon restart where none is needed. So I now think it would be best to change HUP in the way described above. Those that need to apply a config change would then need to do a regular daemon restart - the same thing that is done with other daemons. In order to keep compatible, I would add an $HUPisRestart option, by default off, which could be used to re-enable previous behavior. I would appreciate feedback on this plan. Are there any concerns? Is this a good idea or a bad one? Does it break something I have not yet mentioned? Thanks, Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 11:42:36 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 11:42:36 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> RG> [...] RG> RG> Initially, I thought to use a special signal (e.g. USR2) for this RG> purpose. But still logrotate, as a well-known example, would cause a RG> daemon restart where none is needed. So I now think it would be best to RG> change HUP in the way described above. Those that need to apply a config RG> change would then need to do a regular daemon restart - the same thing RG> that is done with other daemons. In order to keep compatible, I would RG> add an $HUPisRestart option, by default off, which could be used to RG> re-enable previous behavior. RG> Still, I'd like to have a way to force full restart without really tearing down and reloading the server. What about (optionally?) binding current HUP behaviour to USR2 adding a default on option like $USR2isRestart? Cheers, and a nice forthweek to all, yours lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 11:44:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:44:25 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Thanks for the feedback, but please let me ask one question: On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: > Still, I'd like to have a way to force full restart without really tearing > down and reloading the server. What about (optionally?) binding current > HUP behaviour to USR2 adding a default on option like $USR2isRestart? Why is this useful? I am trying to understand the use cases and why a "normal" restart is not desirable. Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 13:33:05 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 13:33:05 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> Thanks for the feedback, but please let me ask one question: RG> RG> On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: RG> RG> > Still, I'd like to have a way to force full restart without really tearing RG> > down and reloading the server. What about (optionally?) binding current RG> > HUP behaviour to USR2 adding a default on option like $USR2isRestart? RG> RG> Why is this useful? I am trying to understand the use cases and why a RG> "normal" restart is not desirable. RG> Habits, pid stability, monitoring systems, one less thing to break... etc. Yours, lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 13:56:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 13:56:50 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Lorenzo, > Habits, pid stability, monitoring systems, one less thing to break... I understand habits and pid stability. What do you mean by "monitoring system" and "one less thing to break"? Anyone else with an opinion? Rainer From david at ecker-software.de Mon Oct 6 14:41:25 2008 From: david at ecker-software.de (David Ecker) Date: Mon, 06 Oct 2008 14:41:25 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EA0775.7050105@ecker-software.de> Hi, 3.21.5 created an assertion error. rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void *)0)' failed. close to the end. Propably short before the abort signal. I mounted /rsyslog (ext3) with the option : sync Here is a copy of my rsyslog.conf: #--------------------------------------- $ModLoad imuxsock.so $ModLoad imklog.so $WorkDirectory /rsyslog $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 1m $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 #--------------------------------------- I attached the output from" rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 Actually only one messagefile was written, no .qi file was created. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My >>> guess is, that the O_DIRECT in combination with the O_SYNC flag (turning >>> of cache) will have an impact. >>> >> O_DIRECT is doing very different things. I don't think you need to worry >> about those things, having the data not go into the OS cache is a drawback >> not an advantage becouse it means that when you go to pull the data back >> out of the file it will need to actually touch disk. it also imposes >> significant alignment issues on the application that I don't think you >> want to have to desl with. >> > > plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: error.txt URL: From rgerhards at hq.adiscon.com Mon Oct 6 14:43:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 14:43:12 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EA0775.7050105@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Please use the version from git. I didn't realize that the bug affects normal operations, but obviously it does. This is fixed and I'll see that I release 3.21.6 ASAP, but I am not sure if I manage to do this today. Gitweb available at http://git.adiscon.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Monday, October 06, 2008 2:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi, > > 3.21.5 created an assertion error. > > rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void > *)0)' failed. > > close to the end. Propably short before the abort signal. > > I mounted /rsyslog (ext3) with the option : sync > > Here is a copy of my rsyslog.conf: > #--------------------------------------- > $ModLoad imuxsock.so > $ModLoad imklog.so > > $WorkDirectory /rsyslog > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 1m > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > #--------------------------------------- > > I attached the output from" > > rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 > > Actually only one messagefile was written, no .qi file was created. > > bye > David Ecker > > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Already did both with 2.18.3 but'll try again with 3.21.5 and > 3.18.4. My > >>> guess is, that the O_DIRECT in combination with the O_SYNC flag > (turning > >>> of cache) will have an impact. > >>> > >> O_DIRECT is doing very different things. I don't think you need to > worry > >> about those things, having the data not go into the OS cache is a > drawback > >> not an advantage becouse it means that when you go to pull the data > back > >> out of the file it will need to actually touch disk. it also imposes > >> significant alignment issues on the application that I don't think > you > >> want to have to desl with. > >> > > > > plus rsyslog does not care about the alignment (at this time), so I > think it is dangerous... > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From hks.private at gmail.com Mon Oct 6 16:49:26 2008 From: hks.private at gmail.com ((private) HKS) Date: Mon, 6 Oct 2008 10:49:26 -0400 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: In general, I understand HUP to be exactly what you're talking - a reload rather than a restart. I see little to lose by enforcing that in rsyslogd. The pid stability argument has no relevance for me or the tools I use since I just pull it out of pidfiles anyway. -HKS On Mon, Oct 6, 2008 at 7:56 AM, Rainer Gerhards wrote: > Lorenzo, > >> Habits, pid stability, monitoring systems, one less thing to break... > > I understand habits and pid stability. What do you mean by "monitoring > system" and "one less thing to break"? > > Anyone else with an opinion? > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 6 17:08:07 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:08:07 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > Hi List, > > I again have a backwards compatibility question. > > HUP, in sysklogd, means a full restart of the syslogd. This is done > because the configuration may have been changed and to apply the new > one, the previous one must be shut down. This is even more so the case > in rsyslog, which has multiple dynamically loadable plugins and all > that... > > rsyslog has implemented HUP processing to be compatible to sysklogd. > > However, in practice (as far as I know) HUP is almost always being used > to close the output files after log rotation. An automatted config > reaload indication usually does not happen (but operators use HUP after > doing a config change). > > HUP, as it currently is, is an extremely expensive operation. Keep in > mind that it is a full daemon restart, so, among other things, it > unloads modules and tears down (TCP, RELP, TLS, ...) connections. In > most cases, though, this would not be needed. > > I now consider changing the way HUP works. I would like to model it in a > way that closes files and clears caches (where useful), but not do a > full restart. > > Initially, I thought to use a special signal (e.g. USR2) for this > purpose. But still logrotate, as a well-known example, would cause a > daemon restart where none is needed. So I now think it would be best to > change HUP in the way described above. Those that need to apply a config > change would then need to do a regular daemon restart - the same thing > that is done with other daemons. In order to keep compatible, I would > add an $HUPisRestart option, by default off, which could be used to > re-enable previous behavior. > > I would appreciate feedback on this plan. Are there any concerns? Is > this a good idea or a bad one? Does it break something I have not yet > mentioned? could you do something along the lines of checking to see if the config file changed, and if so do a full restart, otherwise just the flush/close? if you know when you started you may be able to just check the last-modified time of the config file and do a restart if it's been modified after you started. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:08:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:08:08 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> > could you do something along the lines of checking to see if the config > file changed, and if so do a full restart, otherwise just the > flush/close? > > if you know when you started you may be able to just check the > last-modified time of the config file and do a restart if it's been > modified after you started. That's a very interesting idea. Do you think this criterion is sufficient? Rainer From david at lang.hm Mon Oct 6 17:15:40 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:15:40 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> could you do something along the lines of checking to see if the > config >> file changed, and if so do a full restart, otherwise just the >> flush/close? >> >> if you know when you started you may be able to just check the >> last-modified time of the config file and do a restart if it's been >> modified after you started. > > That's a very interesting idea. Do you think this criterion is > sufficient? I was thinking of other possible items (doing a checksum of the file, doing a syntactic check to see if the new is equivalent to the old, etc) and I think that they don't end up adding noticable value the only way the last-modified check would fail is if your clock goes backwards (and goes backwards far enough that the 'new' file is modified before you last started). two ways to deal with that 1. store the last-modified time of the file when you start up and re-read it if it has changed (still vunerable if the clock moved back to that exact second, but a very small windows) 2. document the issue and tell people that if their clock goes backwards and the HUP doesn't re-read the config file do a 'touch' of the file and try again. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:19:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:19:23 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AC@grfint2.intern.adiscon.com> This sounds very reasonable. Thanks for the good suggestion! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, October 06, 2008 5:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > >> could you do something along the lines of checking to see if the > > config > >> file changed, and if so do a full restart, otherwise just the > >> flush/close? > >> > >> if you know when you started you may be able to just check the > >> last-modified time of the config file and do a restart if it's been > >> modified after you started. > > > > That's a very interesting idea. Do you think this criterion is > > sufficient? > > I was thinking of other possible items (doing a checksum of the file, > doing a syntactic check to see if the new is equivalent to the old, > etc) > and I think that they don't end up adding noticable value > > the only way the last-modified check would fail is if your clock goes > backwards (and goes backwards far enough that the 'new' file is > modified > before you last started). two ways to deal with that > > 1. store the last-modified time of the file when you start up and re- > read > it if it has changed (still vunerable if the clock moved back to that > exact second, but a very small windows) > > 2. document the issue and tell people that if their clock goes > backwards > and the HUP doesn't re-read the config file do a 'touch' of the file > and > try again. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Mon Oct 6 17:45:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 17:45:56 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 : > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >>> could you do something along the lines of checking to see if the >> config >>> file changed, and if so do a full restart, otherwise just the >>> flush/close? >>> >>> if you know when you started you may be able to just check the >>> last-modified time of the config file and do a restart if it's been >>> modified after you started. >> >> That's a very interesting idea. Do you think this criterion is >> sufficient? Don't forget $IncludeConfig Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 17:47:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:47:46 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Args... thanks Michael, this makes life a little less easy ;) So we do not longer have a clear indication. Would it make sense to require that the main config file is touched when something in the includes is changed? This could be documented. And a restart of course pulls everything. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Monday, October 06, 2008 5:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > 2008/10/6 : > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > >>> could you do something along the lines of checking to see if the > >> config > >>> file changed, and if so do a full restart, otherwise just the > >>> flush/close? > >>> > >>> if you know when you started you may be able to just check the > >>> last-modified time of the config file and do a restart if it's been > >>> modified after you started. > >> > >> That's a very interesting idea. Do you think this criterion is > >> sufficient? > > Don't forget $IncludeConfig > > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Oct 6 18:29:03 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 6 Oct 2008 10:29:03 -0600 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> On Mon, Oct 6, 2008 at 08:49, (private) HKS wrote: > In general, I understand HUP to be exactly what you're talking - a > reload rather than a restart. I see little to lose by enforcing that > in rsyslogd. Ditto - IMHO SIGHUP has come to signify 'change what you can without restarting' or a "hot restart". Some daemons are capable of fully re-configuring on the fly, i.e. "warm restart". It would be rather complex but one could mark particular configuration pragmas as requiring a full tear-down and issue a warning if a warm restart was issued and those had changed. The third option (which many poorly-written init scripts do anyway) is a "cold restart", where the process is killed off and manually re-started, allowing full-blown reconfiguration. I don't care what signals are used for what, beyond the [de facto?] standard of 'skip a beat' when given SIGHUP. Beyond the scope of this discussion, I've not looked at your configuration parsing but wonder if you're not parsing it into an internal, static structure and handling bits from there. If you were, it would be trivial to parse your new configuration, see how/whether it differs from the running one, and shift over if able - like failsafe firmware in embedded systems. That would also offer interesting implications for alternative configuration syntaxes. RB From rgerhards at hq.adiscon.com Mon Oct 6 18:34:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:34:12 +0200 Subject: [rsyslog] HUP action In-Reply-To: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> Message-ID: <1223310852.2682.566.camel@rgf9dev.intern.adiscon.com> On Mon, 2008-10-06 at 10:29 -0600, RB wrote: > Beyond the scope of this discussion, I've not looked at your > configuration parsing but wonder if you're not parsing it into an > internal, static structure and handling bits from there. If you were, > it would be trivial to parse your new configuration, see how/whether > it differs from the running one, and shift over if able - like > failsafe firmware in embedded systems. That would also offer > interesting implications for alternative configuration syntaxes. That is quite complex, because the configuration defines what can be configured. With the plug-in architecture, we do not have any fixed memory structures. So, we would need to spawn a new process, load plugins there, check what in-memory structures they create and, if they differ, than use this new configuration instead of the old one. Rainer From mbiebl at gmail.com Mon Oct 6 18:37:31 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 18:37:31 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 Rainer Gerhards : > Args... thanks Michael, this makes life a little less easy ;) So we do > not longer have a clear indication. > > Would it make sense to require that the main config file is touched when > something in the includes is changed? This could be documented. And a > restart of course pulls everything. I don't think that would be such a good idea. It is not very intuitive and people are not used to such a behaviour. They (at least I do) expect "reload" to reload the complete configuration. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 18:46:16 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:46:16 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> So let me try to sum up where we are now: - it is considered useful to have a full daemon restart be done via HUP - we can not sufficiently simply detect whether the configuration has changed or not - there are situations where it is useful to have the ability to just close files, clear caches etc - people do not like existing things be used in new ways (least surprise principle) So I conclude: - SIGHUP, as ugly as it is, must stay with existing semantics - a new signal can be defined to just do file closure etc Unfortunately, this means that most systems will still use the terribly expensive during e.g. logrotation. However, this is considered acceptable because a) it always was this way, b) a higher demand environment then has options to avoid that. Over time, package maintainers my get maintainers of logrotate involved to change the HUP to the new signal. Am I grasping this right? Rainer On Mon, 2008-10-06 at 18:37 +0200, Michael Biebl wrote: > 2008/10/6 Rainer Gerhards : > > Args... thanks Michael, this makes life a little less easy ;) So we do > > not longer have a clear indication. > > > > Would it make sense to require that the main config file is touched when > > something in the includes is changed? This could be documented. And a > > restart of course pulls everything. > > I don't think that would be such a good idea. > > It is not very intuitive and people are not used to such a behaviour. > They (at least I do) expect "reload" to reload the complete configuration. > > > Michael > From linggao at us.ibm.com Mon Oct 6 20:44:40 2008 From: linggao at us.ibm.com (Ling Gao) Date: Mon, 6 Oct 2008 14:44:40 -0400 Subject: [rsyslog] log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From dt.abernathy at gmail.com Tue Oct 7 18:00:24 2008 From: dt.abernathy at gmail.com (Dan Abernathy) Date: Tue, 7 Oct 2008 10:00:24 -0600 Subject: [rsyslog] FreeBSD and rsyslog Message-ID: Hi, I've been working on integrating rsyslog into a FreeBSD environment, and the following issues and questions have come up along the way: 1. The security facility isn't deprecated in FreeBSD, and I've noticed that this is aliased to the auth facility in rsyslog. I have make a quick patch for this on FreeBSD in order to log to the security facility again, and added the ntp facility (also supported). These changes may be more suitable for the FreeBSD ports tree. 2. The other FreeBSD facility in use is console, which logs all writes to /dev/console to the console log facility. I'm not sure how important this feature is, and it's certainly something I can live without, but it doesn't seem supported in rsyslog. 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. I was curious about support for /var/run/logpriv, however I can't find a great deal of information about how and where this socket is used - other than for privileged applications. 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP address is probably at the top of my list right now. Is there any feature to achieve this that I'm missing? Any thoughts, suggestions and discussion welcome. Dan. From hks.private at gmail.com Tue Oct 7 18:38:16 2008 From: hks.private at gmail.com ((private) HKS) Date: Tue, 7 Oct 2008 12:38:16 -0400 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: On Tue, Oct 7, 2008 at 12:00 PM, Dan Abernathy wrote: > Hi, > > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: > > 1. The security facility isn't deprecated in FreeBSD, and I've noticed that > this is aliased to the auth facility in rsyslog. I have make a quick patch > for this on FreeBSD in order to log to the security facility again, and > added the ntp facility (also supported). These changes may be more suitable > for the FreeBSD ports tree. Excellent, that's good news. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. > > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Can't answer either of these. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? I assume you're running rsyslogd v3. If you're using v2, check the man pages (they're still relevant at that point). When you create the UDP/TCP server in the rsyslog.conf file, you give it the address and port you want it to bind to: $UDPServerAddress 10.123.0.19 $UDPServerRun 514 Hope that helps. -HKS From aoz.syn at gmail.com Tue Oct 7 18:53:05 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 7 Oct 2008 10:53:05 -0600 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: <4255c2570810070953j7d489dfcre45b6515d9d68b5b@mail.gmail.com> > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: You're not alone, I've just gotten distracted along the way and not chased down any issues. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. Curious, I'd not carefully looked at that - it's a character device. No idea. > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Should be addressable with imuxsock and permissions; not much to that. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? Configuration pragma $UDPServerAddress From linggao at us.ibm.com Tue Oct 7 19:39:09 2008 From: linggao at us.ibm.com (Ling Gao) Date: Tue, 7 Oct 2008 13:39:09 -0400 Subject: [rsyslog] Fw: log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From rgerhards at hq.adiscon.com Wed Oct 8 11:04:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 11:04:26 +0200 Subject: [rsyslog] my responsiveness Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D4@grfint2.intern.adiscon.com> Hi folks, unfortunately, I am currently a bit unresponsive to mailing list posts. The reason is that I am reviewing the threading model, which is quite complex and does not play well with any distraction. Also, there is some additional (paid) work to be done and this obviously has some priority. Thanks to all who contribute their knowledge. I try my best to catch up with the posts. I think the situation will become much better in a few days at latest. Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:42:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:42:27 +0200 Subject: [rsyslog] Fw: log forwarding in version 3 In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D9@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ling Gao > Sent: Tuesday, October 07, 2008 7:39 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Fw: log forwarding in version 3 > > > > Hi, > We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. > We'd > like to have logs sent from tier3 to tier1 through tier2. (tier3- > >tier2-> > tier1). > > In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to > enable them receiving remote logs. > $ModLoad imudp.so > $UDPServerRun 514 > > In tier2, we have set up the following in etc/rsyslog.conf to send the > logs > to tier1: > *.* @tier1 > > In tier3, we have set up the following in /etc/rsyslog.conf to send the > logs to tier2 : > *.* @tier2 > > Question: > 1. What other setting we need to enable in tier2 so that the logs from > tier3 can be forwarded to tier1 via tier2? (something like -h option > in > lower versions). None (but keep the firewalls in mind!) > 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some > are > Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using > rsyslog > v3, do we need to change all to the compatibility mode? The compatibility mode affects the way the config file and command line options are interpreted. It does not change anything other than those settings. Review the compatibility guide if there are some problems with any of your settings (my guess is "no") and if so, adjust them. It is recommended to run each instance in native mode. Rainer From r.bhatia at ipax.at Wed Oct 8 12:49:05 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 08 Oct 2008 12:49:05 +0200 Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EC9021.1000701@ipax.at> Rainer Gerhards wrote: > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not no knowing the code, i suggest md5/sha1/... checksum the whole file after integrating everything including rsyslog.d/* into one flat file. > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) i, currently, do not care that much about this issue - at least regarding the reload vs. restart issue. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 8 12:56:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:56:11 +0200 Subject: [rsyslog] HUP action In-Reply-To: <48EC9021.1000701@ipax.at> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <48EC9021.1000701@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 08, 2008 12:49 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > Rainer Gerhards wrote: > > - it is considered useful to have a full daemon restart be done via > HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > no knowing the code, i suggest md5/sha1/... checksum the whole file > after integrating everything including rsyslog.d/* into one flat > file. Quite honestly, I do not like the extra effort. I now have to define two signals in any case, so let the user use the right one for what he intends to do ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:58:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:58:26 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com><48EC9021.1000701@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DB@grfint2.intern.adiscon.com> > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > > Sent: Wednesday, October 08, 2008 12:49 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP action > > > > Rainer Gerhards wrote: > > > - it is considered useful to have a full daemon restart be done via > > HUP > > > - we can not sufficiently simply detect whether the configuration > > > has changed or not > > > > no knowing the code, i suggest md5/sha1/... checksum the whole file > > after integrating everything including rsyslog.d/* into one flat > > file. > > Quite honestly, I do not like the extra effort. I now have to define > two > signals in any case, so let the user use the right one for what he > intends to do ;) Sorry, I was a bit brief: doing the checksum requires interpretation of the config files, at least as far as the $IncludeConfig statements are concerned. This is costly, as is the whole gathering process to build the checksum. So to be on the efficient side, I need to have a dedicated HUP for "non-reload" in any case. Thus I think there is not much added benefit in doing the checksumming. Rainer From david at ecker-software.de Thu Oct 9 10:53:02 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 10:53:02 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Message-ID: <48EDC66E.1060202@ecker-software.de> Hi Rainer, the assert error seemed to be fixed in HEAD. Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to work a lot better. /etc/rsyslog.conf ---------------- $WorkDirectory /rsyslog/ $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 100m $ActionQueueSize 10000000 $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 ----------------- I found out that the queue was limited to 1000 elements if you do not define ActionQueueSize inside the config. One difference was that version 3.18.3 actually created one file per msg if I did set ActionQueueCheckpointInterval to 1. Right now only one file in addition to the .qi file is created containing all messages. I haven't waited to see what will happen if I reach 100MB on the data file. If I shutdown rsyslog normally some status information seems to be written to the .qi file (508 bytes -> 1024 bytes). After restarting the qi file actually shrinks to 508 bytes again. If I kill rsyslogd (SIGKILL) the qi file is not updated with this information (as expected). But it looks like the queue is still working correctly after restarting rsyslogd at least it doesn't invalidate the queue or loosing messages. Turning the system off immediatly seems to be working most of the times. After 11 tries the last one failed. It actually looked like that I turned the system off during a write. bye David Ecker Rainer Gerhards schrieb: > Please use the version from git. I didn't realize that the bug affects > normal operations, but obviously it does. This is fixed and I'll see > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > today. > > Gitweb available at http://git.adiscon.com > > Rainer > From rgerhards at hq.adiscon.com Thu Oct 9 13:56:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 09 Oct 2008 13:56:14 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDC66E.1060202@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> Message-ID: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Hi David, On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > Hi Rainer, > > the assert error seemed to be fixed in HEAD. It is good to hear this. I begun to have some doubts when I reviewed the code. I would really appreciate if you could download and test this version here: http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz I will probably release that tomorrow and so some indication if the problem is actually gone would be very good. The final 3.21.6 will see one more patch, but nothing that affects the assert in question. > > Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to > work a lot better. > > /etc/rsyslog.conf > ---------------- > $WorkDirectory /rsyslog/ > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 100m > $ActionQueueSize 10000000 > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > ----------------- > > I found out that the queue was limited to 1000 elements if you do not > define ActionQueueSize inside the config. Yes, that's the default for *action* queues (the main message queue default is different, I think 10,000). > One difference was that version 3.18.3 actually created one file per msg > if I did set ActionQueueCheckpointInterval to 1. Right now only one > file in addition to the .qi file is created containing all messages. I > haven't waited to see what will happen if I reach 100MB on the data file. That's probably a result of a bug fixed in the repo but not yet released. But I don't check if it is gone now. 3.18.x will only see one more release, then we move on to 3.20.x. > If I shutdown rsyslog normally some status information seems to be > written to the .qi file (508 bytes -> 1024 bytes). After restarting the > qi file actually shrinks to 508 bytes again. That's right. > > If I kill rsyslogd (SIGKILL) the qi file is not updated with this > information (as expected). But it looks like the queue is still working > correctly after restarting rsyslogd at least it doesn't invalidate the > queue or loosing messages. I'd say it depends on when exactly it is doing when it is being killed. > > Turning the system off immediatly seems to be working most of the times. > After 11 tries the last one failed. It actually looked like that I > turned the system off during a write. This is also within what I expect. If you hit it during a write, things are really bad. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Please use the version from git. I didn't realize that the bug affects > > normal operations, but obviously it does. This is fixed and I'll see > > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > > today. > > > > Gitweb available at http://git.adiscon.com > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Thu Oct 9 14:43:30 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 14:43:30 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EDFC72.2070404@ecker-software.de> Hi Rainer, with the same config it behaves similar to the HEAD version I tested in the last few days. The assert message did not appear. bye David Ecker Rainer Gerhards schrieb: > Hi David, > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > >> Hi Rainer, >> >> the assert error seemed to be fixed in HEAD. >> > > It is good to hear this. I begun to have some doubts when I reviewed the > code. I would really appreciate if you could download and test this > version here: > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > I will probably release that tomorrow and so some indication if the > problem is actually gone would be very good. The final 3.21.6 will see > one more patch, but nothing that affects the assert in question. > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to >> work a lot better. >> >> /etc/rsyslog.conf >> ---------------- >> $WorkDirectory /rsyslog/ >> $ActionQueueFileName buffer >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionQueueType Disk >> $ActionQueueMaxFileSize 100m >> $ActionQueueSize 10000000 >> $ActionQueueCheckpointInterval 1 >> $ActionResumeRetryCount -1 >> *.* @@10.8.0.1:514 >> ----------------- >> >> I found out that the queue was limited to 1000 elements if you do not >> define ActionQueueSize inside the config. >> > > Yes, that's the default for *action* queues (the main message queue > default is different, I think 10,000). > > >> One difference was that version 3.18.3 actually created one file per msg >> if I did set ActionQueueCheckpointInterval to 1. Right now only one >> file in addition to the .qi file is created containing all messages. I >> haven't waited to see what will happen if I reach 100MB on the data file. >> > > That's probably a result of a bug fixed in the repo but not yet > released. But I don't check if it is gone now. 3.18.x will only see one > more release, then we move on to 3.20.x. > > >> If I shutdown rsyslog normally some status information seems to be >> written to the .qi file (508 bytes -> 1024 bytes). After restarting the >> qi file actually shrinks to 508 bytes again. >> > > That's right. > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this >> information (as expected). But it looks like the queue is still working >> correctly after restarting rsyslogd at least it doesn't invalidate the >> queue or loosing messages. >> > > I'd say it depends on when exactly it is doing when it is being killed. > > >> Turning the system off immediatly seems to be working most of the times. >> After 11 tries the last one failed. It actually looked like that I >> turned the system off during a write. >> > > This is also within what I expect. If you hit it during a write, things > are really bad. > > >> bye >> David Ecker >> >> Rainer Gerhards schrieb: >> >>> Please use the version from git. I didn't realize that the bug affects >>> normal operations, but obviously it does. This is fixed and I'll see >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do this >>> today. >>> >>> Gitweb available at http://git.adiscon.com >>> >>> Rainer >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 9 15:31:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Oct 2008 15:31:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDFC72.2070404@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de><1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> <48EDFC72.2070404@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2F3@grfint2.intern.adiscon.com> Hi David, thanks for the quick feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Thursday, October 09, 2008 2:44 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi Rainer, > > with the same config it behaves similar to the HEAD version I tested in > the last few days. The assert message did not appear. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Hi David, > > > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > > > >> Hi Rainer, > >> > >> the assert error seemed to be fixed in HEAD. > >> > > > > It is good to hear this. I begun to have some doubts when I reviewed > the > > code. I would really appreciate if you could download and test this > > version here: > > > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > > > I will probably release that tomorrow and so some indication if the > > problem is actually gone would be very good. The final 3.21.6 will > see > > one more patch, but nothing that affects the assert in question. > > > > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed > to > >> work a lot better. > >> > >> /etc/rsyslog.conf > >> ---------------- > >> $WorkDirectory /rsyslog/ > >> $ActionQueueFileName buffer > >> $ActionQueueMaxDiskSpace 1g > >> $ActionQueueSaveOnShutdown on > >> $ActionQueueType Disk > >> $ActionQueueMaxFileSize 100m > >> $ActionQueueSize 10000000 > >> $ActionQueueCheckpointInterval 1 > >> $ActionResumeRetryCount -1 > >> *.* @@10.8.0.1:514 > >> ----------------- > >> > >> I found out that the queue was limited to 1000 elements if you do > not > >> define ActionQueueSize inside the config. > >> > > > > Yes, that's the default for *action* queues (the main message queue > > default is different, I think 10,000). > > > > > >> One difference was that version 3.18.3 actually created one file per > msg > >> if I did set ActionQueueCheckpointInterval to 1. Right now only one > >> file in addition to the .qi file is created containing all messages. > I > >> haven't waited to see what will happen if I reach 100MB on the data > file. > >> > > > > That's probably a result of a bug fixed in the repo but not yet > > released. But I don't check if it is gone now. 3.18.x will only see > one > > more release, then we move on to 3.20.x. > > > > > >> If I shutdown rsyslog normally some status information seems to be > >> written to the .qi file (508 bytes -> 1024 bytes). After restarting > the > >> qi file actually shrinks to 508 bytes again. > >> > > > > That's right. > > > > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this > >> information (as expected). But it looks like the queue is still > working > >> correctly after restarting rsyslogd at least it doesn't invalidate > the > >> queue or loosing messages. > >> > > > > I'd say it depends on when exactly it is doing when it is being > killed. > > > > > >> Turning the system off immediatly seems to be working most of the > times. > >> After 11 tries the last one failed. It actually looked like that I > >> turned the system off during a write. > >> > > > > This is also within what I expect. If you hit it during a write, > things > > are really bad. > > > > > >> bye > >> David Ecker > >> > >> Rainer Gerhards schrieb: > >> > >>> Please use the version from git. I didn't realize that the bug > affects > >>> normal operations, but obviously it does. This is fixed and I'll > see > >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do > this > >>> today. > >>> > >>> Gitweb available at http://git.adiscon.com > >>> > >>> Rainer > >>> > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From friedl at hq.adiscon.com Thu Oct 9 17:32:51 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 9 Oct 2008 17:32:51 +0200 Subject: [rsyslog] rsyslog 3.18.5 (v3-stable) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2FD@grfint2.intern.adiscon.com> Hi all, we have just released rsyslog 3.18.5, a member of the v3-stable branch. This is most probably the last 3.18.x version, the current beta will soon become the new stable. Rsyslog 3.18.5 is a bug fixing release with some enhanced documentation. The most important bug fixes addresses some issues during HUP, which could lead to a segfault. This is a recommended update for all users of the v3-stable branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-133.phtml Changelog: http://www.rsyslog.com/Article281.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From kyuhwanzz at gmail.com Wed Oct 15 14:03:05 2008 From: kyuhwanzz at gmail.com (Kyu-Hwan Yun) Date: Wed, 15 Oct 2008 21:03:05 +0900 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? Message-ID: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Hi, all. My question is, How cound rsyslog recognize that remote server is alive? port open/close, network interface up/down, or etc... Which method is the answer? From rgerhards at hq.adiscon.com Wed Oct 15 15:08:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Oct 2008 15:08:43 +0200 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? In-Reply-To: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> References: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F378@grfint2.intern.adiscon.com> The simple answer is that we try a send() call and if it succeeds, that's a good indication the other side is alive. If it fails, the retry logic tries to restore the connection. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Kyu-Hwan Yun > Sent: Wednesday, October 15, 2008 2:03 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How can the rsyslog recognize remote server is > alive? > > Hi, all. > > My question is, > > How cound rsyslog recognize that remote server is alive? > > port open/close, network interface up/down, or etc... > > Which method is the answer? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Thu Oct 16 15:11:36 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 16 Oct 2008 15:11:36 +0200 Subject: [rsyslog] rsyslog 3.19.12 (beta) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F39F@grfint2.intern.adiscon.com> Hi all, rsyslog 3.12.19 has been released, a member of the beta branch. It is mainly a refresh of the current beta which provides all those bug-fixes that have been made to the stable releases. Other than that, it includes some minor documentation updates and a bugfix for a beta-specific issue. This is scheduled to become the new v3-stable soon. It is a recommended update for all beta branch users. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-134.phtml Changelog: http://www.rsyslog.com/Article283.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Oct 18 06:40:22 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 17 Oct 2008 21:40:22 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > So let me try to sum up where we are now: > > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) > > So I conclude: > > - SIGHUP, as ugly as it is, must stay with existing semantics > - a new signal can be defined to just do file closure etc > > Unfortunately, this means that most systems will still use the terribly > expensive during e.g. logrotation. However, this is considered > acceptable because a) it always was this way, b) a higher demand > environment then has options to avoid that. Over time, package > maintainers my get maintainers of logrotate involved to change the HUP > to the new signal. > > Am I grasping this right? I learned this week that on a HUP rsyslog will loose any messages that it has in it's memory queue (due to the fact that it basicly does a full shutdown and restart). I suspect that many other people would be surprised at this as well. It looks like we have two options. 1. HUP will notice config file changes, but can loose queued logs 2. HUP will not notice config file changes, but will not loose queued logs I think that there would be less surprise in the second case. all the scripts that use HUP to roll logs don't really care about the config files I guess there is a third case, make a HUP not drop the queue, but if the config file options for the queue change significantly, this can be a very difficult thing to do David Lang From rgerhards at hq.adiscon.com Sat Oct 18 10:32:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 18 Oct 2008 10:32:21 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > So let me try to sum up where we are now: > > > > - it is considered useful to have a full daemon restart be done via HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > - there are situations where it is useful to have the ability to > > just close files, clear caches etc > > - people do not like existing things be used in new ways > > (least surprise principle) > > > > So I conclude: > > > > - SIGHUP, as ugly as it is, must stay with existing semantics > > - a new signal can be defined to just do file closure etc > > > > Unfortunately, this means that most systems will still use the terribly > > expensive during e.g. logrotation. However, this is considered > > acceptable because a) it always was this way, b) a higher demand > > environment then has options to avoid that. Over time, package > > maintainers my get maintainers of logrotate involved to change the HUP > > to the new signal. > > > > Am I grasping this right? > > I learned this week that on a HUP rsyslog will loose any messages that it > has in it's memory queue (due to the fact that it basicly does a full > shutdown and restart). I suspect that many other people would be surprised > at this as well. Well, actually it depends. In the default configuration, this is true. But the spirit of the default configuration (suitable for most simple cases) is "I don't care about message loss". If you care about message loss, queues need to be configured so that no already-gathered messages are lost between sessions. If done so, messages are written to files before a shutdown and read from it upon restart. Of course, messages are still lost while the system is doing the restart (new ones). For a HUP and a very busy system, this can mean it takes a couple of seconds, even minutes, to do a full restart. After I noticed that the "message loss on HUP if not configured to avoid it" is not obvious, I added some explicit doc to the man pages so that people are warned they do a full restart. > > It looks like we have two options. > > 1. HUP will notice config file changes, but can loose queued logs > > 2. HUP will not notice config file changes, but will not loose queued logs > > I think that there would be less surprise in the second case. see comments above > > all the scripts that use HUP to roll logs don't really care about the > config files that was my initial observation. I will define another signal to just close the file. I hope I can get this done next week. Unfortunately, the financial crisis begins to hit the real world and this causes me to currently spent less time on new additions to rsyslog and more on profit-generating projects, but that should be doable in a day or two (the output module interface needs to be extend, a new administrative linked list implemented and the signal handler be done - changes to outputs are also necessary). > > I guess there is a third case, make a HUP not drop the queue, but if the > config file options for the queue change significantly, this can be a very > difficult thing to do This is extremely difficult, but of course doable. I think, however, this is only needed for a very limited number of cases. So this is probably something that I will only work on if a sponsor for this work shows up. Rainer > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sat Oct 18 11:32:32 2008 From: david at lang.hm (david at lang.hm) Date: Sat, 18 Oct 2008 02:32:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: On Sat, 18 Oct 2008, Rainer Gerhards wrote: > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> >>> So let me try to sum up where we are now: >>> >>> - it is considered useful to have a full daemon restart be done via HUP >>> - we can not sufficiently simply detect whether the configuration >>> has changed or not >>> - there are situations where it is useful to have the ability to >>> just close files, clear caches etc >>> - people do not like existing things be used in new ways >>> (least surprise principle) >>> >>> So I conclude: >>> >>> - SIGHUP, as ugly as it is, must stay with existing semantics >>> - a new signal can be defined to just do file closure etc >>> >>> Unfortunately, this means that most systems will still use the terribly >>> expensive during e.g. logrotation. However, this is considered >>> acceptable because a) it always was this way, b) a higher demand >>> environment then has options to avoid that. Over time, package >>> maintainers my get maintainers of logrotate involved to change the HUP >>> to the new signal. >>> >>> Am I grasping this right? >> >> I learned this week that on a HUP rsyslog will loose any messages that it >> has in it's memory queue (due to the fact that it basicly does a full >> shutdown and restart). I suspect that many other people would be surprised >> at this as well. > > Well, actually it depends. In the default configuration, this is true. > But the spirit of the default configuration (suitable for most simple > cases) is "I don't care about message loss". there's a big difference between "I don't care about message loss if a system crashed" and "I don't care about message loss at all" admins should evaluate the probability of the different message loss scenerios and decide on the appropriate cost/benifit for their situation. it's very possible that someone may not care about loosing logs when a system crashes (becouse it's a fairly rare event), but isn't willing to loose logs just to rotate log files. > If you care about message > loss, queues need to be configured so that no already-gathered messages > are lost between sessions. If done so, messages are written to files > before a shutdown and read from it upon restart. Of course, messages are > still lost while the system is doing the restart (new ones). > > For a HUP and a very busy system, this can mean it takes a couple of > seconds, even minutes, to do a full restart. and unfortunantly, the process of writing the pending queue to disk and then reading it back after a restart can greatly lengthen the restart, causing incoming logs to be missed (depending on how able the log sender is to queue the log messages for your) > After I noticed that the "message loss on HUP if not configured to avoid > it" is not obvious, I added some explicit doc to the man pages so that > people are warned they do a full restart. thanks. >> >> It looks like we have two options. >> >> 1. HUP will notice config file changes, but can loose queued logs >> >> 2. HUP will not notice config file changes, but will not loose queued logs >> >> I think that there would be less surprise in the second case. > > see comments above the purpose of this message was mostly to try and prompt the other people who commented in this thread to think about the issue and comment on their expectations. >> >> all the scripts that use HUP to roll logs don't really care about the >> config files > > that was my initial observation. I will define another signal to just > close the file. I hope I can get this done next week. Unfortunately, the > financial crisis begins to hit the real world and this causes me to > currently spent less time on new additions to rsyslog and more on > profit-generating projects, but that should be doable in a day or two > (the output module interface needs to be extend, a new administrative > linked list implemented and the signal handler be done - changes to > outputs are also necessary). I am not trying to preasure you into rushing on this, just to get opinions on which is the 'more expected' behavior for HUP >> >> I guess there is a third case, make a HUP not drop the queue, but if the >> config file options for the queue change significantly, this can be a very >> difficult thing to do > > This is extremely difficult, but of course doable. I think, however, > this is only needed for a very limited number of cases. So this is > probably something that I will only work on if a sponsor for this work > shows up. I would definantly not recommend this one becouse of the difficulty involved, but in the interest of completeness I thought it needed to be mentioned. David Lang From rgerhards at hq.adiscon.com Mon Oct 20 09:41:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Oct 2008 09:41:24 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: > On Sat, 18 Oct 2008, Rainer Gerhards wrote: > > > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >> > >>> So let me try to sum up where we are now: > >>> > >>> - it is considered useful to have a full daemon restart be done via HUP > >>> - we can not sufficiently simply detect whether the configuration > >>> has changed or not > >>> - there are situations where it is useful to have the ability to > >>> just close files, clear caches etc > >>> - people do not like existing things be used in new ways > >>> (least surprise principle) > >>> > >>> So I conclude: > >>> > >>> - SIGHUP, as ugly as it is, must stay with existing semantics > >>> - a new signal can be defined to just do file closure etc > >>> > >>> Unfortunately, this means that most systems will still use the terribly > >>> expensive during e.g. logrotation. However, this is considered > >>> acceptable because a) it always was this way, b) a higher demand > >>> environment then has options to avoid that. Over time, package > >>> maintainers my get maintainers of logrotate involved to change the HUP > >>> to the new signal. > >>> > >>> Am I grasping this right? > >> > >> I learned this week that on a HUP rsyslog will loose any messages that it > >> has in it's memory queue (due to the fact that it basicly does a full > >> shutdown and restart). I suspect that many other people would be surprised > >> at this as well. > > > > Well, actually it depends. In the default configuration, this is true. > > But the spirit of the default configuration (suitable for most simple > > cases) is "I don't care about message loss". > > there's a big difference between "I don't care about message loss if a > system crashed" and "I don't care about message loss at all" That's right. But I was not talking about a crash condition. I was talking about a shutdown (or a restart). With the default settings, rsyslog does what sysklogd does, that is (mostly) not care about message loss. If you shut down the daemon, it tries to drain the queue for a short timeout. If all messages can be processed during that time, we are fine and terminate rsyslogd without message loss. However, if there are too many messages queued, the daemon terminates after the timeout has expired and discards whatever messages are left unprocessed. This is done in an effort to keep compatible with sysklogd. Also, in most scenarios it is highly unlikely that message loss will occur, because the queue will not hold extended data for an extended period of time. If, however, this is anticipated, there are ways to configure rsyslog not to loose messages. For example, the timeout can be increased and/or the queue can be made disk-assisted and data be written to disk if they can not be processed during the timeout. During queue drain, no new messages are accepted (otherwise, how should we finally shut down?). Given a busy system, there is unavoidable message loss. Either you use old messages you already had in the queue, but have a quick restart and loose few new messages. Or you save the already existing messages but experience loss of new messages. This is another reason for the default, which I consider the best compromise under the circumstances given. This is also a good indication why you should not unnecessarily restart rsyslgod (or any other syslogd, because this is not a project-specific problem but rather a general one). With fully reliable protocols, you can of course make the sender block while you are shutting down. But all of this requires very careful planning. I assume folks who want to do that either have the necessary knowledge themselvs or hire someone who does. So there are three scenarios: a) hup b) regular shutdown c) system abort I think you did not take into account b), which is a quite common case. Other than that, I fully agree to you post. Rainer From david at lang.hm Mon Oct 20 19:07:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 10:07:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 20 Oct 2008, Rainer Gerhards wrote: > On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: >> On Sat, 18 Oct 2008, Rainer Gerhards wrote: >> >>> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >>>> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> So let me try to sum up where we are now: >>>>> >>>>> - it is considered useful to have a full daemon restart be done via HUP >>>>> - we can not sufficiently simply detect whether the configuration >>>>> has changed or not >>>>> - there are situations where it is useful to have the ability to >>>>> just close files, clear caches etc >>>>> - people do not like existing things be used in new ways >>>>> (least surprise principle) >>>>> >>>>> So I conclude: >>>>> >>>>> - SIGHUP, as ugly as it is, must stay with existing semantics >>>>> - a new signal can be defined to just do file closure etc >>>>> >>>>> Unfortunately, this means that most systems will still use the terribly >>>>> expensive during e.g. logrotation. However, this is considered >>>>> acceptable because a) it always was this way, b) a higher demand >>>>> environment then has options to avoid that. Over time, package >>>>> maintainers my get maintainers of logrotate involved to change the HUP >>>>> to the new signal. >>>>> >>>>> Am I grasping this right? >>>> >>>> I learned this week that on a HUP rsyslog will loose any messages that it >>>> has in it's memory queue (due to the fact that it basicly does a full >>>> shutdown and restart). I suspect that many other people would be surprised >>>> at this as well. >>> >>> Well, actually it depends. In the default configuration, this is true. >>> But the spirit of the default configuration (suitable for most simple >>> cases) is "I don't care about message loss". >> >> there's a big difference between "I don't care about message loss if a >> system crashed" and "I don't care about message loss at all" > > That's right. But I was not talking about a crash condition. I was > talking about a shutdown (or a restart). With the default settings, > rsyslog does what sysklogd does, that is (mostly) not care about message > loss. If you shut down the daemon, it tries to drain the queue for a > short timeout. If all messages can be processed during that time, we are > fine and terminate rsyslogd without message loss. the big difference is that rsysklog does not maintain a queue, so when it shuts down it doesn't loose any messages that it has processed (even with async writes it has passed the logs to the OS and so those logs are not affected by the sysklog shutdown/HUP) > However, if there are too many messages queued, the daemon terminates > after the timeout has expired and discards whatever messages are left > unprocessed. This is done in an effort to keep compatible with sysklogd. > Also, in most scenarios it is highly unlikely that message loss will > occur, because the queue will not hold extended data for an extended > period of time. If, however, this is anticipated, there are ways to > configure rsyslog not to loose messages. For example, the timeout can be > increased and/or the queue can be made disk-assisted and data be written > to disk if they can not be processed during the timeout. > > During queue drain, no new messages are accepted (otherwise, how should > we finally shut down?). Given a busy system, there is unavoidable > message loss. Either you use old messages you already had in the queue, > but have a quick restart and loose few new messages. Or you save the > already existing messages but experience loss of new messages. This is > another reason for the default, which I consider the best compromise > under the circumstances given. This is also a good indication why you > should not unnecessarily restart rsyslgod (or any other syslogd, because > this is not a project-specific problem but rather a general one). this conflicts with the need to rotate the log files. while we have discussed the possibility of using a different signal to allow the logs to be rotated, I was bringing the subject up again becouse I think the HUP behavior is unexpected enough that it may be less surprise if a HUP doesn't re-read config files but does the close/open needed for log rotation than the alturnative of having HUP remain as-in with a different signal for log rotation. I'm not suggesting more work, I'm suggesting that it may make sense to just use HUP for the close/open and if someone needs the full config reload they do a stop/start instead. > With fully reliable protocols, you can of course make the sender block > while you are shutting down. But all of this requires very careful > planning. I assume folks who want to do that either have the necessary > knowledge themselvs or hire someone who does. > > So there are three scenarios: > > a) hup > b) regular shutdown > c) system abort > > I think you did not take into account b), which is a quite common case. there is case d) log rotation when someone does a stop/start (case b) they expect logs in a queue to be lost (you are fully stopping the process after all), I don't think that's what people would expect from a HUP. David Lang > Other than that, I fully agree to you post. > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From pheller at me.com Mon Oct 20 19:32:51 2008 From: pheller at me.com (Phillip Heller) Date: Mon, 20 Oct 2008 13:32:51 -0400 Subject: [rsyslog] Expression based filters not working? Message-ID: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil From mikel at irontec.com Mon Oct 20 21:54:42 2008 From: mikel at irontec.com (Mikel Jimenez) Date: Mon, 20 Oct 2008 21:54:42 +0200 Subject: [rsyslog] db template how to Message-ID: <48FCE202.1070100@irontec.com> Hello How can I make a template of sql in rsyslog.conf that makes that? : hostname1 >database.hostname1 hostname2 >database.hostname2 hostanem3 >database.hostname3 ..... and hostnameaa hostnameab hostnamedd hostnamede hostnamea*>database.a hostnamed*>database.d Thanks From david at lang.hm Mon Oct 20 22:44:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 13:44:32 -0700 (PDT) Subject: [rsyslog] db template how to In-Reply-To: <48FCE202.1070100@irontec.com> References: <48FCE202.1070100@irontec.com> Message-ID: On Mon, 20 Oct 2008, Mikel Jimenez wrote: > Date: Mon, 20 Oct 2008 21:54:42 +0200 > From: Mikel Jimenez > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] db template how to > > Hello > How can I make a template of sql in rsyslog.conf that makes that? : > > hostname1 >database.hostname1 > hostname2 >database.hostname2 > hostanem3 >database.hostname3 > ..... first a disclaimer, I haven't used the database output yet, so I could be wrong or missing something easier that what I'm suggesting. putting them into different databases would be difficult and expensive, but possible. if you are putting them into different tables inside one database you could change from something like "insert into syslog(message) values ('%msg%')" to something like "insert into syslog||'.'||%hostname% (message) values ('%msg%')" since you are issuing SQL commands, you could do something along the lines of the following to do different databases "connect database||'.'||%hostname% ; insert into syslog(message) values ('%msg%'): to switch the the appropriate databases. (note that this is usually _very_ inefficiant to do) I don't know if you can do multiple database output modules going to different systems, but if you can you may be able to do something nicer by defining a different output module for each hostname and use filters to seperate the traffic (this will obviously only work for a limited number of hostnames, and they all have to be known ahead of time) > > and > > hostnameaa > hostnameab > hostnamedd > hostnamede > > hostnamea*>database.a > hostnamed*>database.d this could be similar, but you would have to do string manipulation of the hostname value to extract out just the character you need. David Lang From aoz.syn at gmail.com Mon Oct 20 23:12:23 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 20 Oct 2008 15:12:23 -0600 Subject: [rsyslog] db template how to In-Reply-To: References: <48FCE202.1070100@irontec.com> Message-ID: <4255c2570810201412n2f8addbbqf94b6d78b1d7ccd8@mail.gmail.com> On Mon, Oct 20, 2008 at 14:44, wrote: > putting them into different databases would be difficult and expensive, > but possible. Hopefully we're not talking different databases, per se, maybe different tables? My suggestion wouldn't be to put any of that logic at all in your rsyslog configuration - the application & configurations are complex enough. Rather (if you have a proper database engine) use table partitioning, even constraint-based partitioning to do what you need. Much less horrifying on the client side and any competent DBA will know what you're doing and be able to handle it. Off the top of my head I know Oracle, PostgreSQL, and MySQL support partitioning and I'd be willing to bet MSSQL does as well. Perhaps a more appropriate question would be why? Other than your mental organization and maybe some access-control concerns, why would you want to divide your database by host? Not only does it make global correlation more complex, it's practically guaranteed to have an uneven distribution. When doing log tables, I always arrange the MERGE/partition constraint by date or the like so it's easier and often faster for limiting searches as well as to age data out (DROP TABLE versus DELETE FROM). From DGillies at fairfaxdigital.com.au Tue Oct 21 01:26:20 2008 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 21 Oct 2008 10:26:20 +1100 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Message-ID: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Hi Phil, I found that ! didn't work for me in my expression style configs. I had to use 'not' instead: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then -?DailyLogs David Gillies Systems Engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller Sent: Tuesday, 21 October 2008 4:33 AM To: rsyslog at lists.adiscon.com Subject: [rsyslog] Expression based filters not working? Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From rgerhards at hq.adiscon.com Tue Oct 21 08:32:57 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 21 Oct 2008 08:32:57 +0200 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F3E8@grfint2.intern.adiscon.com> That's right ! is NOT part of the language. You need to use the word "not". Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Gillies > Sent: Tuesday, October 21, 2008 1:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Expression based filters not working? > > Hi Phil, > > I found that ! didn't work for me in my expression style configs. I had > to use 'not' instead: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat > if $msg contains 'VOIPAAA' then -?DailyCDR > if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then > -?DailyLogs > > David Gillies > Systems Engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller > Sent: Tuesday, 21 October 2008 4:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Expression based filters not working? > > Hello, > > Running 3.19.9 with the following old style config bits: > > :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, > "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs > > This works just fine, though the messages from the ASA are output to > the > DailyLogs template twice. I have not seen an example how one might use > a logical AND with the old style config. > > So, I've tried the new expression style config: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains > 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg > !contains '%ASA-' then -?DailyLogs > > However, this style seems to not work at all. That is, no messages are > recorded to DailyLogs or DailyCDR. > > Any suggestions? > > Regards, > > Phil > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended recipient, > any use, dissemination, reliance, forwarding, printing or copying of > this e-mail or any attached files is unauthorised. This e-mail is > subject to copyright. No part of it should be reproduced, adapted or > communicated without the written consent of the copyright owner. If you > have received this e-mail in error please advise the sender immediately > by return e-mail or telephone and delete all copies. Fairfax does not > guarantee the accuracy or completeness of any information contained in > this e-mail or attached files. Internet communications are not secure, > therefore Fairfax does not accept legal responsibility for the contents > of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 22 10:13:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 10:13:23 +0200 Subject: [rsyslog] rsyslog 3.21.6 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F414@grfint2.intern.adiscon.com> Hi all, rsyslog 3.21.6, a member of the development branch, has just been released. Most importantly, it fixes a segfault condition which could occur during processing. It also contains some other bugfixes, many imported from the stable and beta branches. Feature-wise, performance has been improved, permitting rsyslogd to handle an even higher number of messages per second. This is a recommended update for all devel branch users. This release is scheduled to become the new beta soon. Changelog: http://www.rsyslog.com/Article292.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-135.phtml As always, feedback is appreciated. Rainer From epiphani at gmail.com Wed Oct 22 18:05:06 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 12:05:06 -0400 Subject: [rsyslog] output compression... Message-ID: I've been looking for some method to automatically compress output files from a syslog service - rsyslogd seems to be the most flexible in terms of base architecture. Has anyone written a zlib-based (or some gzip implementation) output module for writing files to disk already compressed? Would it be something difficult to add to the roadmap? :) Thanks, -Aaron From aoz.syn at gmail.com Wed Oct 22 18:30:38 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 10:30:38 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <4255c2570810220930h3b63c09awbe6d993da0a5270b@mail.gmail.com> On Wed, Oct 22, 2008 at 10:05, Aaron Wiebe wrote: > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) It's on my personal roadmap, but fell behind circular logs (FBSD clog-compatible) and my screwing around with Windows .evt formats. Don't know if anyone else has looked at it. Even though it should be a well-received addition, the problem is that most users do well enough with logrotate or its ilk to manage log compression and don't count it too high on their priorities. From rgerhards at hq.adiscon.com Wed Oct 22 18:37:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 18:37:03 +0200 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Hi, I have not much to add to RB's remark. I agree to his analysis. I'd like to do it, but always higher priority things come in between... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 22, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] output compression... > > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) > > Thanks, > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Wed Oct 22 20:55:03 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 14:55:03 -0400 Subject: [rsyslog] output compression... In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: Understood - thanks for the response. Can you point me at some docs for your modules API? I may see what I can do on my own here... Thanks, -Aaron On Wed, Oct 22, 2008 at 12:37 PM, Rainer Gerhards wrote: > Hi, > > I have not much to add to RB's remark. I agree to his analysis. I'd like > to do it, but always higher priority things come in between... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Wednesday, October 22, 2008 6:05 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] output compression... >> >> I've been looking for some method to automatically compress output >> files from a syslog service - rsyslogd seems to be the most flexible >> in terms of base architecture. Has anyone written a zlib-based (or >> some gzip implementation) output module for writing files to disk >> already compressed? >> >> Would it be something difficult to add to the roadmap? :) >> >> Thanks, >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 22 21:40:05 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 13:40:05 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> > Can you point me at some docs for your modules API? I may see what I > can do on my own here... I would suggest starting with plugins/omtesting, plugins/imtemplate and tools/omfile.* - those helped me jump in PDQ. The following URLs were also helpful: http://www.rsyslog.com/doc-generic_design.html http://www.rsyslog.com/doc-modules.html From rgerhards at hq.adiscon.com Thu Oct 23 13:23:10 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 23 Oct 2008 13:23:10 +0200 Subject: [rsyslog] output compression... In-Reply-To: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> Message-ID: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-22 at 13:40 -0600, RB wrote: > > Can you point me at some docs for your modules API? I may see what I > > can do on my own here... > > I would suggest starting with plugins/omtesting, plugins/imtemplate > and tools/omfile.* - those helped me jump in PDQ. The following URLs > were also helpful: > > http://www.rsyslog.com/doc-generic_design.html > http://www.rsyslog.com/doc-modules.html That's a good suggestion. In this case, I would also tend to modifying omfile.c directly. This, too me, sounds much like just adding the ability to zip before doing the write() call. If you intend to contribute back the patch, I can lend you a helping hand (as much as time permits, unfortunately) during the process. Rainer From epiphani at gmail.com Thu Oct 23 17:12:49 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Thu, 23 Oct 2008 11:12:49 -0400 Subject: [rsyslog] output compression... In-Reply-To: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> Message-ID: On Thu, Oct 23, 2008 at 7:23 AM, Rainer Gerhards wrote: > > That's a good suggestion. In this case, I would also tend to modifying > omfile.c directly. This, too me, sounds much like just adding the > ability to zip before doing the write() call. Yes, I've read omfile.c - I will likely be following this route. > If you intend to contribute back the patch, I can lend you a helping > hand (as much as time permits, unfortunately) during the process. Unfortunately that is at the whim of my employer. I have a process started to try to clear such things, but for now I will have to work alone. Thanks, -Aaron From david at lang.hm Fri Oct 24 23:47:54 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 24 Oct 2008 14:47:54 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? Message-ID: I'm constructing custom output formats and it looks like things are not gettng parsed as I would expect. the incoming logs look like 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 143 E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 plug-gw[13051]: disconnect host= /192.168.242.12 destination=179.50.100.52/14872 in=1069 out=71 duration=1 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 135 E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p plug-gw[10883]: disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 out=720 duration=1 what is unexpected is that tag is the hostname/IP and the plug-gw is part of the message the hostname field is getting populated with what I would expect to be in fromhost (the relay box that sent the message to me) the syslog daemons sending me the logs have been modified, so there is a possibility that I messed up on them and the format that's being sent isn't right, but if so I'm not seeing anything wrong with it. I am using the nextmaster branch. David Lang From rgerhards at hq.adiscon.com Sun Oct 26 20:41:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 26 Oct 2008 20:41:12 +0100 Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Could you get me the debug output while such a message is being processed? That would probably be useful... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 24, 2008 11:48 PM > To: rsyslog-users > Subject: [rsyslog] parsing error with imudp? > > I'm constructing custom output formats and it looks like > things are not > gettng parsed as I would expect. > > the incoming logs look like > > 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 143 > E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 > plug-gw[13051]: disconnect host= /192.168.242.12 > destination=179.50.100.52/14872 in=1069 out=71 duration=1 > > 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 135 > E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p > plug-gw[10883]: > disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 > out=720 duration=1 > > > what is unexpected is that tag is the hostname/IP and the > plug-gw is part > of the message > > the hostname field is getting populated with what I would > expect to be in > fromhost (the relay box that sent the message to me) > > the syslog daemons sending me the logs have been modified, so > there is a > possibility that I messed up on them and the format that's being sent > isn't right, but if so I'm not seeing anything wrong with it. > > I am using the nextmaster branch. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 27 01:38:44 2008 From: david at lang.hm (david at lang.hm) Date: Sun, 26 Oct 2008 17:38:44 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, Rainer Gerhards wrote: > Could you get me the debug output while such a message is being > processed? That would probably be useful... I'll have to recompile to get a build with all the debug fetures enabled. I will do this tomorrow. I did check rsyslog->rsyslog relay and saw the same thing (actually it was sysklogd -> rsyslog -> rsyslog. the first rsyslog shows the correct thing, the second rsyslog shows the hostname of the first, with the hostname of the sysklog box in the message) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 24, 2008 11:48 PM >> To: rsyslog-users >> Subject: [rsyslog] parsing error with imudp? >> >> I'm constructing custom output formats and it looks like >> things are not >> gettng parsed as I would expect. >> >> the incoming logs look like >> >> 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 143 >> E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 >> plug-gw[13051]: disconnect host= /192.168.242.12 >> destination=179.50.100.52/14872 in=1069 out=71 duration=1 >> >> 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 135 >> E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p >> plug-gw[10883]: >> disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 >> out=720 duration=1 >> >> >> what is unexpected is that tag is the hostname/IP and the >> plug-gw is part >> of the message >> >> the hostname field is getting populated with what I would >> expect to be in >> fromhost (the relay box that sent the message to me) >> >> the syslog daemons sending me the logs have been modified, so >> there is a >> possibility that I messed up on them and the format that's being sent >> isn't right, but if so I'm not seeing anything wrong with it. >> >> I am using the nextmaster branch. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From luigi.perroti at googlemail.com Tue Oct 28 10:12:36 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 10:12:36 +0100 Subject: [rsyslog] New HUP directive effects on ommysql Message-ID: Hi, maybe the developers would like to add this finding to the ommysql module documentation. I found out that using '$HUPisRestart off' with 3.21.6 has an effect on the ommysql module too. More precisely I noted that changes made to $ActionOmmysqlServerPort become effective. This might be useful if it's needed to change the database port without doing a full configuration reload. Please correct me if you think I got this wrong, thanks. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 14:00:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 14:00:03 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Hi Luigi, could you please double-check your observation? This should not happen, as ommysql does not have any HUP handler. If that happens, we may have a new track to what is the bug in your environment. I will concentrate on a lab with MySQL enabled after I hear back from you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Tuesday, October 28, 2008 10:13 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] New HUP directive effects on ommysql > > Hi, > > maybe the developers would like to add this finding to the ommysql > module documentation. > > I found out that using '$HUPisRestart off' with 3.21.6 has an effect > on the ommysql module too. > More precisely I noted that changes made to $ActionOmmysqlServerPort > become effective. This might be useful if it's needed to change the > database port without doing a full configuration reload. > > Please correct me if you think I got this wrong, thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Tue Oct 28 14:59:01 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 14:59:01 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: Hi, I looked into it more carefully and I get this by checking the configuration file: /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 rsyslogd: version 3.21.6, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 rsyslogd: End of config validation run. Bye. Indeed, line 31 is where I have the $HUPisRestart directive. Even if 3.21.6 is the latest released development version it occurred to me that this feature might only be in the repository at this time. If that's the case then I'm sorry about the noise. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 15:12:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:12:25 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: <1225203145.23821.131.camel@rgf9dev.intern.adiscon.com> Oh, sorry, I think we got the trouble spot ;) It is currently available as part of the "nextmaster" branch. I am right now in the process of switching stable/beta/master versions and nextmaster is what will be the new master when this is done. Rainer On Tue, 2008-10-28 at 14:59 +0100, Luigi Perroti wrote: > Hi, > > I looked into it more carefully and I get this by checking the > configuration file: > > /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 > > rsyslogd: version 3.21.6, config validation run (level 1), master > config /etc/rsyslog.conf > > rsyslogd: invalid or yet-unknown config file command - have you > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] > > rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 > rsyslogd: End of config validation run. Bye. > > Indeed, line 31 is where I have the $HUPisRestart directive. > > Even if 3.21.6 is the latest released development version it occurred > to me that this feature might only be in the repository at this time. > If that's the case then I'm sorry about the noise. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 28 15:55:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:55:15 +0100 Subject: [rsyslog] new v3-stable release candidate Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Hi all, I am about to switch the v3-stable release to 3.20, which means it will be based on the 3.19.x beta branch. I have created a test tarball, available at: http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz I would appreciate if some could try it out and report any potential issues they see. I intend to release this tomorrow or Thursday if nothing bad happens. Rainer From david at lang.hm Tue Oct 28 18:26:14 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 10:26:14 -0700 (PDT) Subject: [rsyslog] new v3-stable release candidate In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: On Tue, 28 Oct 2008, Rainer Gerhards wrote: > Hi all, > > I am about to switch the v3-stable release to 3.20, which means it will > be based on the 3.19.x beta branch. I have created a test tarball, > available at: > > http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz > > I would appreciate if some could try it out and report any potential > issues they see. I intend to release this tomorrow or Thursday if > nothing bad happens. one problem I have been seeing in the nextmaster branch is that with imudp on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the box rsyslog goes into a loop (eating 100% cpu, but not doing anything) when it recieves a IPv4 syslog packet. Rainer has not been able to duplicate the problem, could someone else test this? I've been seeing it on Debian amd64 boxes. David Lang From rgerhards at hq.adiscon.com Tue Oct 28 18:29:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 18:29:38 +0100 Subject: [rsyslog] new v3-stable release candidate In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BF@grfint2.intern.adiscon.com> > one problem I have been seeing in the nextmaster branch is that with > imudp > on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the > box > rsyslog goes into a loop (eating 100% cpu, but not doing anything) when > it > recieves a IPv4 syslog packet. > > Rainer has not been able to duplicate the problem, could someone else > test > this? That would be really good. But please note that nextmaster will become 3.23.x, not 3.20. The upcoming 3.20 stable release is based on the current beta. I just wanted to make sure we are all talking about the right versions :) Rainer From david at lang.hm Wed Oct 29 02:44:04 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 18:44:04 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, david at lang.hm wrote: > On Sun, 26 Oct 2008, Rainer Gerhards wrote: > >> Could you get me the debug output while such a message is being >> processed? That would probably be useful... > > I'll have to recompile to get a build with all the debug fetures enabled. > I will do this tomorrow. 1264.971559438:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1264.971582626:{6000}imudp.c: main queue: entry added, size now 1 entries 1264.971595478:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1264.971611682:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1264.971627886:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1264.971649678:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1264.971664206:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1264.971684042:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1264.971701084:{6000}imudp.c: 1264.971717009:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000049 1264.971747461:{5997}main queue:Reg/w0: Message has legacy syslog format. 1264.971767577:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1264.971784898:{5997}main queue:Reg/w0: (/var/log/messages) 1264.971798867:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1265.971497411:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1265.971509424:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1265.971527863:{6000}imudp.c: main queue: entry added, size now 1 entries 1265.971538480:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1265.971554125:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1265.971569491:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1265.971587372:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1265.971603017:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000050 1265.971617824:{5997}main queue:Reg/w0: Message has legacy syslog format. 1265.971633749:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1265.971647159:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1265.971664760:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1265.971681523:{6000}imudp.c: 1265.971696610:{5997}main queue:Reg/w0: (/var/log/messages) 1265.971712814:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1266.971494055:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1266.971506068:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 From rgerhards at hq.adiscon.com Wed Oct 29 15:23:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:23:46 +0100 Subject: [rsyslog] regex help wanted Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Hi folks, I am tracking down a problem with a user here in this forum thread: http://kb.monitorware.com/regex-match-for-port-t8764.html#p14411 He claims that rsyslog does not correctly handle POSIX ERE regular expressions, but I now have written a simple program (tester) that works in the same way. I don't think I have set any options wrong. However, I am not at all an regex expert. Maybe some of you could shed some light on this? I would really appreciate some help, as the situation sounds sub-optimal, at least until we have an explanation for it... Thanks, Rainer From aoz.syn at gmail.com Wed Oct 29 15:47:23 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 08:47:23 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> > He claims that rsyslog does not correctly handle POSIX ERE regular > expressions, but I now have written a simple program (tester) that works > in the same way. I don't think I have set any options wrong. However, I > am not at all an regex expert. I've not carefully read the portion of the source where you do your matching; can you point me in a general direction? What library are you using? RB From rgerhards at hq.adiscon.com Wed Oct 29 15:50:20 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:50:20 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 3:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > He claims that rsyslog does not correctly handle POSIX ERE regular > > expressions, but I now have written a simple program (tester) that > works > > in the same way. I don't think I have set any options wrong. However, > I > > am not at all an regex expert. > > I've not carefully read the portion of the source where you do your > matching; can you point me in a general direction? What library are > you using? Sure. There is a very simple sample program inside the forum thread. This is a direct link: http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 Well... I'll also include it after my sig. I use the plain old clib regex library. The problem can fully be reproduced with the minimalistic sample, so I think it is better to look at it than at the actual rsyslog source (but most of the regex functionality is in msg.c). Thanks, Rainer #include #include #include #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 (198.39.187.236/54751)" int main() { regex_t preg; size_t nmatch = 10; regmatch_t pmatch[10]; char str[] = STR; int i; i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); printf("regcomp returns %d\n", i); i = regexec(&preg, str, nmatch, pmatch, 0); printf("regexec returns %d\n", i); if(i == REG_NOMATCH) { printf("found no match!\n"); return 1; } printf("returned substrings:\n"); for(i = 0 ; i < 10 ; i++) { printf("%d: so %d, eo %d", i, pmatch[i].rm_so, pmatch[i].rm_eo); if(pmatch[i].rm_so != -1) { int j; printf(", text: '"); for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) putchar(str[j]); putchar('\''); } putchar('\n'); } return 0; } From hks.private at gmail.com Wed Oct 29 16:07:18 2008 From: hks.private at gmail.com ((private) HKS) Date: Wed, 29 Oct 2008 11:07:18 -0400 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: The lazy quantifier (?) is not being respected, so rather than returning the first match of "stuff, a forward slash, more stuff, space", it returns the last one. I know zero C, so I'm afraid I can't be much help in figuring out why. You could try the universal regexp trick: if in doubt, escape it! -HKS On Wed, Oct 29, 2008 at 10:50 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, October 29, 2008 3:47 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] regex help wanted >> >> > He claims that rsyslog does not correctly handle POSIX ERE regular >> > expressions, but I now have written a simple program (tester) that >> works >> > in the same way. I don't think I have set any options wrong. > However, >> I >> > am not at all an regex expert. >> >> I've not carefully read the portion of the source where you do your >> matching; can you point me in a general direction? What library are >> you using? > > Sure. There is a very simple sample program inside the forum thread. > This is a direct link: > > http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 > > Well... I'll also include it after my sig. I use the plain old clib > regex library. The problem can fully be reproduced with the minimalistic > sample, so I think it is better to look at it than at the actual rsyslog > source (but most of the regex functionality is in msg.c). > > Thanks, > Rainer > > #include > #include > #include > > #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 > for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 > (198.39.187.236/54751)" > int main() > { > regex_t preg; > size_t nmatch = 10; > regmatch_t pmatch[10]; > char str[] = STR; > int i; > > i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); > printf("regcomp returns %d\n", i); > i = regexec(&preg, str, nmatch, pmatch, 0); > printf("regexec returns %d\n", i); > if(i == REG_NOMATCH) { > printf("found no match!\n"); > return 1; > } > > printf("returned substrings:\n"); > for(i = 0 ; i < 10 ; i++) { > printf("%d: so %d, eo %d", i, pmatch[i].rm_so, > pmatch[i].rm_eo); > if(pmatch[i].rm_so != -1) { > int j; > printf(", text: '"); > for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) > putchar(str[j]); > putchar('\''); > } > putchar('\n'); > } > return 0; > } > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 29 16:32:39 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 09:32:39 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> > The lazy quantifier (?) is not being respected, so rather than > returning the first match of "stuff, a forward slash, more stuff, > space", it returns the last one. Correct, because the lazy quantifier is not POSIX and therefore does not work when using POSIX extended REs. I was wondering why the extra '?', but had forgotten about lazy notation, preferring to make my regexes a bit more strict. RB From rgerhards at hq.adiscon.com Wed Oct 29 16:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 16:34:52 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com><4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4DA@grfint2.intern.adiscon.com> Ah, thanks a lot to both of you. This greatly helps and makes me feel much better. Of course, we can add additional regex libs, but it has been proven that the current implementation works OK. Thanks again, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 4:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > The lazy quantifier (?) is not being respected, so rather than > > returning the first match of "stuff, a forward slash, more stuff, > > space", it returns the last one. > > Correct, because the lazy quantifier is not POSIX and therefore does > not work when using POSIX extended REs. I was wondering why the extra > '?', but had forgotten about lazy notation, preferring to make my > regexes a bit more strict. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 02:13:54 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:13:54 -0700 (PDT) Subject: [rsyslog] git url? Message-ID: what is the url for the rsyslog git repository? David Lang From mbiebl at gmail.com Wed Oct 1 02:19:05 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 1 Oct 2008 02:19:05 +0200 Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: 2008/10/1 : > what is the url for the rsyslog git repository? gitweb: http://git.adiscon.com/?p=rsyslog.git;a=summary clonable urls: git://git.adiscon.com/git/rsyslog.git http://git.adiscon.com/git/rsyslog.git -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Oct 1 02:38:46 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:38:46 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Wed, 1 Oct 2008, Michael Biebl wrote: > 2008/10/1 : >> what is the url for the rsyslog git repository? > > gitweb: > http://git.adiscon.com/?p=rsyslog.git;a=summary > > clonable urls: > git://git.adiscon.com/git/rsyslog.git > http://git.adiscon.com/git/rsyslog.git thanks, much easier then guessing. now that I have his main branch cloned, what do I need to do to have it fetch all the other branches? David Lang From david at lang.hm Wed Oct 1 02:54:01 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 30 Sep 2008 17:54:01 -0700 (PDT) Subject: [rsyslog] git url? In-Reply-To: References: Message-ID: On Tue, 30 Sep 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Michael Biebl wrote: > >> 2008/10/1 : >>> what is the url for the rsyslog git repository? >> >> gitweb: >> http://git.adiscon.com/?p=rsyslog.git;a=summary >> >> clonable urls: >> git://git.adiscon.com/git/rsyslog.git >> http://git.adiscon.com/git/rsyslog.git > > thanks, much easier then guessing. > > now that I have his main branch cloned, what do I need to do to have it > fetch all the other branches? found it, git branch -r David Lang From luigi.perroti at googlemail.com Wed Oct 1 10:57:41 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 10:57:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated Message-ID: Hi, I'd like to ask if it's normal that %timegenerated:::date-subseconds% returns a value lower than %timereported:::date-subseconds%. Intuitively I'd say it is since this makes sense by taking in consideration the property names. Still, since the documentation says that timegenerated is the timestamp when the message was received I would have expected a value higher than the one returned by timereported. I'm afraid this is a dumb question but this behaviour got me confused, if anyone could shed some light on this I'd be grateful. Thanks. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 11:24:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 11:24:08 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> It is just returning the subsecond *part* of the timestamp. So while 9:13:14.123 is "higher" than 8:13:14.567, the subsecond part of the later (567) is higher than the subsecond part of the earlier (123). It is important that you are extracting a partial timestamp, which also implies that this is no longer monotonically increasing. Actually, it no longer has any sense of "timing" at all. This is NOT the timestamp converted to nanoseconds (or whatever). I hope this clarifies, if not, keep posting ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 10:58 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Subseconds in timereported and timegenerated > > Hi, > > I'd like to ask if it's normal that %timegenerated:::date-subseconds% > returns a value lower than %timereported:::date-subseconds%. > > Intuitively I'd say it is since this makes sense by taking in > consideration the property names. > Still, since the documentation says that timegenerated is the > timestamp when the message was received I would have expected a value > higher than the one returned by timereported. > > I'm afraid this is a dumb question but this behaviour got me confused, > if anyone could shed some light on this I'd be grateful. Thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 12:00:37 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:00:37 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: Thanks for the thorough explanation but I'm afraid that I still don't understand why I'm seeing this: http://imagebin.ca/view/2EknmP0.html This is how the columns are populated: sent <- %timereported:::date-mysql% sent_subseconds <- %timereported:::date-subseconds% received_by_rsyslog <- %timegenerated:::date-mysql% received_subseconds <- %timegenerated:::date-subseconds% Most probably I didn't get the difference between timereported and timegenerated. Meaning that I should just switch the sent* and received* columns and that nothing is wrong with Rsyslog. I'd be great if you could could confirm this scenario, thanks for your time! Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 12:03:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:03:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Just a quick note while working on a bigger thing: Is the sent timestamp form an instance that received data via the network (data that contained a high-precision timestamp). Also, which version are you using? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks for the thorough explanation but I'm afraid that I still don't > understand why I'm seeing this: > > http://imagebin.ca/view/2EknmP0.html > > This is how the columns are populated: > sent <- %timereported:::date-mysql% > sent_subseconds <- %timereported:::date-subseconds% > received_by_rsyslog <- %timegenerated:::date-mysql% > received_subseconds <- %timegenerated:::date-subseconds% > > Most probably I didn't get the difference between timereported and > timegenerated. > Meaning that I should just switch the sent* and received* columns and > that nothing is wrong with Rsyslog. > > I'd be great if you could could confirm this scenario, thanks for your > time! > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 12:00:47 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 12:00:47 +0200 Subject: [rsyslog] Development of failsafe disk based queue Message-ID: <48E34A4F.308@ecker-software.de> Hi, I am looking for a failsafe solution to store syslog messages localy until they could be send later. I already looked at the disk based memory queue and the disk based queue. Both queue's don't work if you just power down the system immediatly actually loosing the whole queue. I already looked at queue.c and it seemed to me that both queues were not designed for that kind of failure, but I could be wrong there. Since an immediate power down of the system is the major failure which will occure pretty often I need to create a soltution there. Did you already start to develop something addressing that problem? Could you help me extend rsyslog (3.18.4) so that I can develop a new queue myself? I would contribute the code to the rsyslog project if you would like afterwards. bye David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:22:19 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:22:19 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: Here's a better screenshot: http://imagebin.ca/view/8p8hSnT.html Everything is happening on the same machine. I have these modules loaded: imfile, imuxsock, imklog, ommysql. The php-fpm entries are read from a log file by imfile. Right now I'm trying out 3.21.5 but I think I had already seen this behaviour on 3.21.3 but I'm not too sure about the version number. Luigi On 10/1/08, Rainer Gerhards wrote: > Just a quick note while working on a bigger thing: Is the sent timestamp > form an instance that received data via the network (data that contained > a high-precision timestamp). Also, which version are you using? > > Rainer > > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti >> Sent: Wednesday, October 01, 2008 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated >> >> Thanks for the thorough explanation but I'm afraid that I still don't >> understand why I'm seeing this: >> >> http://imagebin.ca/view/2EknmP0.html >> >> This is how the columns are populated: >> sent <- %timereported:::date-mysql% >> sent_subseconds <- %timereported:::date-subseconds% >> received_by_rsyslog <- %timegenerated:::date-mysql% >> received_subseconds <- %timegenerated:::date-subseconds% >> >> Most probably I didn't get the difference between timereported and >> timegenerated. >> Meaning that I should just switch the sent* and received* columns and >> that nothing is wrong with Rsyslog. >> >> I'd be great if you could could confirm this scenario, thanks for your >> time! >> >> Regards, >> Luigi >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 12:28:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:28:09 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Please post your config, that makes it easier for me ;). I think I know where this stems back from - we had two time calls inside rsyslog done, which has been removed in the experimental "perf" git branch. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 12:22 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Here's a better screenshot: > http://imagebin.ca/view/8p8hSnT.html > > Everything is happening on the same machine. > I have these modules loaded: imfile, imuxsock, imklog, ommysql. > The php-fpm entries are read from a log file by imfile. > > Right now I'm trying out 3.21.5 but I think I had already seen this > behaviour on 3.21.3 but I'm not too sure about the version number. > > Luigi > > On 10/1/08, Rainer Gerhards wrote: > > Just a quick note while working on a bigger thing: Is the sent > timestamp > > form an instance that received data via the network (data that > contained > > a high-precision timestamp). Also, which version are you using? > > > > Rainer > > > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > >> Sent: Wednesday, October 01, 2008 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > >> > >> Thanks for the thorough explanation but I'm afraid that I still > don't > >> understand why I'm seeing this: > >> > >> http://imagebin.ca/view/2EknmP0.html > >> > >> This is how the columns are populated: > >> sent <- %timereported:::date-mysql% > >> sent_subseconds <- %timereported:::date-subseconds% > >> received_by_rsyslog <- %timegenerated:::date-mysql% > >> received_subseconds <- %timegenerated:::date-subseconds% > >> > >> Most probably I didn't get the difference between timereported and > >> timegenerated. > >> Meaning that I should just switch the sent* and received* columns > and > >> that nothing is wrong with Rsyslog. > >> > >> I'd be great if you could could confirm this scenario, thanks for > your > >> time! > >> > >> Regards, > >> Luigi > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:32:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 12:32:26 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. > Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. I doubt there is a software soution against this (one that does not depend on a transactional file system, of course). What prevents you from using a UPS? I'd say that a sudden power-loss is by far the least probable error cause for a system that is configured to do any serious work. Please elaborate why you (or others ;)) consider this case important. > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker From luigi.perroti at googlemail.com Wed Oct 1 12:37:04 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 12:37:04 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: Here it is: $ModLoad imfile $ModLoad imuxsock $ModLoad imklog $ModLoad ommysql $ActionOmmysqlServerPort 1234 $MaxMessageSize 8k $EscapeControlCharactersOnReceive off $FileOwner root $FileGroup root $FileCreateMode 0600 $InputFileName /path/to/log/php-fpm.log $InputFileTag php-fpm $InputFileStateFile php-fpm.state $InputFileSeverity notice $InputFileFacility local5 $InputRunFileMonitor $MainMsgQueueType LinkedList $MainMsgQueueSize 20000 $MainMsgQueueFileName main-spool $MainMsgQueueMaxDiskSpace 200M $MainMsgQueueWorkerThreadMinimumMessages 2500 $MainMsgQueueWorkerThreads 1 $MainMsgQueueWorkerTimeoutThreadShutdown 10000 $MainMsgQueueSaveOnShutdown on $template my-rsyslog-schema,"insert into messages(hostname, sent, sent_subseconds, received_by_rsyslog, received_subseconds, process, facility, severity, message) values ('%hostname%', '%timereported:::date-mysql%', %timereported:::date-subseconds%, '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', '%msg%')",sql *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema From rgerhards at hq.adiscon.com Wed Oct 1 12:46:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:46:42 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> Message-ID: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Thanks, this is useful and helps understanding what is going on. I think what you see is an artifact of the two time() calls I mentioned. To verify, I'd appreciate if you could try out the latest from the perf branch, which can be found here: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=1531c541b294b6c029e5de9a01d4d8499f7e4718;sf=tgz Note that you need to setup the build environment (quite easy, but you need to know). Details at: http://www.rsyslog.com/doc-build_from_repo.html These two timestamps, for internally generated sources, should be the same in that version. Note, however, that, for externally generated messages, timereceived is whatever is recorded inside the message. There are various reasons for this not being correct, clock skew being a highly probably cause (which is also often found). HTH Rainer On Wed, 2008-10-01 at 12:37 +0200, Luigi Perroti wrote: > Here it is: > > $ModLoad imfile > $ModLoad imuxsock > $ModLoad imklog > $ModLoad ommysql > $ActionOmmysqlServerPort 1234 > > $MaxMessageSize 8k > > $EscapeControlCharactersOnReceive off > > $FileOwner root > $FileGroup root > $FileCreateMode 0600 > > $InputFileName /path/to/log/php-fpm.log > $InputFileTag php-fpm > $InputFileStateFile php-fpm.state > $InputFileSeverity notice > $InputFileFacility local5 > $InputRunFileMonitor > > $MainMsgQueueType LinkedList > $MainMsgQueueSize 20000 > $MainMsgQueueFileName main-spool > $MainMsgQueueMaxDiskSpace 200M > $MainMsgQueueWorkerThreadMinimumMessages 2500 > $MainMsgQueueWorkerThreads 1 > $MainMsgQueueWorkerTimeoutThreadShutdown 10000 > $MainMsgQueueSaveOnShutdown on > > $template my-rsyslog-schema,"insert into messages(hostname, sent, > sent_subseconds, received_by_rsyslog, received_subseconds, process, > facility, severity, message) values ('%hostname%', > '%timereported:::date-mysql%', %timereported:::date-subseconds%, > '%timegenerated:::date-mysql%', %timegenerated:::date-subseconds%, > '%programname%', '%syslogfacility-text%', '%syslogseverity-text%', > '%msg%')",sql > > *.* :ommysql:127.0.0.1,rsyslog,rsyslog_user,password;my-rsyslog-schema > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 12:50:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 12:50:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> One thing I forgot to mention: a pure disk queue (not a disk-assisted one) gets you as close to your goal as possible (well, mostly - we could, at a considerable performance expense, require synced writing). With that case, all data is immediately stored on disk. You can configure it to also write the meta data out immediately (and again with sync, not yet supported). However, you still have a window of exposure, for example if the power loss happens right in the middle of when the disk actually writes data to the disk sector. I still wonder why this scenario would be useful to address... Rainer On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. > > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 13:02:50 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:02:50 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> Message-ID: <48E358DA.5080104@ecker-software.de> Rainer Gerhards schrieb: >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole >> > queue. > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. >> Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> > > I doubt there is a software soution against this (one that does not > depend on a transactional file system, of course). What prevents you > from using a UPS? I'd say that a sudden power-loss is by far the least > probable error cause for a system that is configured to do any serious > work. > > Please elaborate why you (or others ;)) consider this case important. > > The client systems (about 200 of them planned) are stationed in public places around the world connected to centralized servers through vpn connections over an unreliable network connection. Since space and look requirement is important a UPS won't fit there. There is actually no space for an UPS. The main problem is that customers are actually pulling the plug to restart the system, to charge their laptops or mobile phones or just for the fun of it. The client base image is a read-only system (Knoppix Like) with an extra hard disk for swap and other informations like syslog messages. Since there are no administrators close to the client system the client itself needs to have the capability to send all the missing log information between a network failure and an immediate power down to the central server for error analysis since those are usualy the most important once to pinpoint the cause of the inital error. My approach would be to use a block device directly since a file system if fault-prone if you shut down the system immediatly. Each entry including the header information guarded by a checksum value. It would be actually something like a fixed array based queue just that it would store the information in a block device. But this is just an inital thought. >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if >> > you > >> would like afterwards. >> >> bye >> David Ecker >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 13:15:22 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 13:15:22 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E35BCA.7030808@ecker-software.de> Rainer Gerhards schrieb: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > If that would work it would be perfect. For testing I could actually pass the correct fctl flag inside of queue.c hardcoded if that is all that is needed. > I still wonder why this scenario would be useful to address... > > > David > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 13:48:32 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:48:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E35BCA.7030808@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <48E35BCA.7030808@ecker-software.de> Message-ID: <1222861712.2682.450.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 13:15 +0200, David Ecker wrote: > Rainer Gerhards schrieb: > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > > If that would work it would be perfect. For testing I could actually > pass the correct fctl flag inside of queue.c hardcoded if that is all > that is needed. that works ;) You just need to fine-tune the queue params. If you find problems with that, I am more than happy to help. > > I still wonder why this scenario would be useful to address... very interesting scenario. Never thought about such one :) If you need to enhance the queue (for block devices), I can provide a some hints. But, given the other priorities which are required by a much broader user base, involving me more than a hint here or there would probably require a consulting contract. Sorry for being so bluntly, but I think it is always good to set the right expectation level. Rainer From david at lang.hm Wed Oct 1 13:55:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 04:55:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E34A4F.308@ecker-software.de> References: <48E34A4F.308@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Hi, > > I am looking for a failsafe solution to store syslog messages localy > until they could be send later. I already looked at the disk based > memory queue and the disk based queue. Both queue's don't work if you > just power down the system immediatly actually loosing the whole queue. are you sure about the disk based queue? per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue can be set to do a commit of the metadata after each message. Disk Queues Disk queues use disk drives for buffering. The important fact is that the always use the disk and do not buffer anything in memory. Thus, the queue is ultra-reliable, but by far the slowest mode. For regular use cases, this queue mode is not recommended. It is useful if log data is so important that it must not be lost, even in extreme cases. When a disk queue is written, it is done in chunks. Each chunk receives its individual file. Files are named with a prefix (set via the "$QueueFilename" config directive) and followed by a 7-digit number (starting at one and incremented for each file). Chunks are 10mb by default, a different size can be set via the"$QueueMaxFileSize" config directive. Note that the size limit is not a sharp one: rsyslog always writes one complete queue entry, even if it violates the size limit. So chunks are actually a little but (usually less than 1k) larger then the configured size. Each chunk also has a different size for the same reason. If you observe different chunk sizes, you can relax: this is not a problem. Writing in chunks is used so that processed data can quickly be deleted and is free for other uses - while at the same time keeping no artificial upper limit on disk space used. If a disk quota is set (instructions further below), be sure that the quota/chunk size allows at least two chunks to be written. Rsyslog currently does not check that and will fail miserably if a single chunk is over the quota. Creating new chunks costs performance but provides quicker ability to free disk space. The 10mb default is considered a good compromise between these two. However, it may make sense to adapt these settings to local policies. For example, if a disk queue is written on a dedicated 200gb disk, it may make sense to use a 2gb (or even larger) chunk size. Please note, however, that the disk queue by default does not update its housekeeping structures every time it writes to disk. This is for performance reasons. In the event of failure, data will still be lost (except when manually is mangled with the file structures). However, disk queues can be set to write bookkeeping information on checkpoints (every n records), so that this can be made ultra-reliable, too. If the checkpoint interval is set to one, no data can be lost, but the queue is exceptionally slow. Each queue can be placed on a different disk for best performance and/or isolation. This is currently selected by specifying different $WorkDirectory config directives before the queue creation statement. To create a disk queue, use the "$QueueType Disk" config directive. Checkpoint intervals can be specified via "$QueueCheckpointInterval", with 0 meaning no checkpoints. you also need to specificly enable syncing (from http://www.rsyslog.com/doc-v3compatibility.html ) Output File Syncing Rsyslogd tries to keep as compatible to stock syslogd as possible. As such, it retained stock syslogd's default of syncing every file write if not specified otherwise (by placing a dash in front of the output file name). While this was a useful feature in past days where hardware was much less reliable and UPS seldom, this no longer is useful in today's worl. Instead, the syncing is a high performace hit. With it, rsyslogd writes files around 50 *times* slower than without it. It also affects overall system performance due to the high IO activity. In rsyslog v3, syncing has been turned off by default. This is done via a specific configuration directive "$ActionFileEnableSync on/off" which is off by default. So even if rsyslogd finds sync selector lines, it ignores them by default. In order to enable file syncing, the administrator must specify "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that syncing only happens in some installations where the administrator actually wanted that (performance-intense) feature. In the fast majority of cases (if not all), this dramatically increases rsyslogd performance without any negative effects. > I already looked at queue.c and it seemed to me that both queues were > not designed for that kind of failure, but I could be wrong there. Since > an immediate power down of the system is the major failure which will > occure pretty often I need to create a soltution there. with checkpoint interval set to 1 and syncing enabled the data should be in on the disk safely (assuming you have hardware that supports this) and a power-off won't affect it. David Lang > Did you already start to develop something addressing that problem? > Could you help me extend rsyslog (3.18.4) so that I can develop a new > queue myself? I would contribute the code to the rsyslog project if you > would like afterwards. > > bye > David Ecker > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 13:57:55 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 13:57:55 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> David, the file syncing mentioned in the compatibility doc applies to the output action, only. The queue does never do synchronous writes - I always assumed that a critical system would have a UPS and could never think (so far) about a valid reason for not having it. So the queue would need to have an extra option to do sync writes. Obviously, that's not a big deal. Performance, of course, will be extremely terrible with such a setup... Rainer On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Hi, > > > > I am looking for a failsafe solution to store syslog messages localy > > until they could be send later. I already looked at the disk based > > memory queue and the disk based queue. Both queue's don't work if you > > just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that the > always use the disk and do not buffer anything in memory. Thus, the queue > is ultra-reliable, but by far the slowest mode. For regular use cases, > this queue mode is not recommended. It is useful if log data is so > important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk receives > its individual file. Files are named with a prefix (set via the > "$QueueFilename" config directive) and followed by a 7-digit > number (starting at one and incremented for each file). Chunks are 10mb by > default, a different size can be set via the"$QueueMaxFileSize" > config directive. Note that the size limit is not a sharp one: rsyslog > always writes one complete queue entry, even if it violates the size > limit. So chunks are actually a little but (usually less than 1k) larger > then the configured size. Each chunk also has a different size for the > same reason. If you observe different chunk sizes, you can relax: this is > not a problem. > > Writing in chunks is used so that processed data can quickly be deleted > and is free for other uses - while at the same time keeping no artificial > upper limit on disk space used. If a disk quota is set (instructions > further below), be sure that the quota/chunk size allows at least two > chunks to be written. Rsyslog currently does not check that and will fail > miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to free > disk space. The 10mb default is considered a good compromise between these > two. However, it may make sense to adapt these settings to local policies. > For example, if a disk queue is written on a dedicated 200gb disk, it may > make sense to use a 2gb (or even larger) chunk size. > > Please note, however, that the disk queue by default does not update its > housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, disk > queues can be set to write bookkeeping information on checkpoints (every n > records), so that this can be made ultra-reliable, too. If the checkpoint > interval is set to one, no data can be lost, but the queue is > exceptionally slow. > > Each queue can be placed on a different disk for best performance and/or > isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write if > not specified otherwise (by placing a dash in front of the output file > name). While this was a useful feature in past days where hardware was > much less reliable and UPS seldom, this no longer is useful in today's > worl. Instead, the syncing is a high performace hit. With it, rsyslogd > writes files around 50 *times* slower than without it. It also affects > overall system performance due to the high IO activity. In rsyslog v3, > syncing has been turned off by default. This is done via a specific > configuration directive "$ActionFileEnableSync on/off" which is off by > default. So even if rsyslogd finds sync selector lines, it ignores them by > default. In order to enable file syncing, the administrator must specify > "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > syncing only happens in some installations where the administrator > actually wanted that (performance-intense) feature. In the fast majority > of cases (if not all), this dramatically increases rsyslogd performance > without any negative effects. > > > > > I already looked at queue.c and it seemed to me that both queues were > > not designed for that kind of failure, but I could be wrong there. Since > > an immediate power down of the system is the major failure which will > > occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should be > in on the disk safely (assuming you have hardware that supports this) and > a power-off won't affect it. > > David Lang > > > > > Did you already start to develop something addressing that problem? > > Could you help me extend rsyslog (3.18.4) so that I can develop a new > > queue myself? I would contribute the code to the rsyslog project if you > > would like afterwards. > > > > bye > > David Ecker > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:02:37 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:02:37 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E358DA.5080104@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F261@grfint2.intern.adiscon.com> <48E358DA.5080104@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Rainer Gerhards schrieb: >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole >>> >> queue. >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. >>> Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> >> I doubt there is a software soution against this (one that does not >> depend on a transactional file system, of course). What prevents you >> from using a UPS? I'd say that a sudden power-loss is by far the least >> probable error cause for a system that is configured to do any serious >> work. >> >> Please elaborate why you (or others ;)) consider this case important. >> >> > The client systems (about 200 of them planned) are stationed in public > places around the world connected to centralized servers through vpn > connections over an unreliable network connection. Since space and look > requirement is important a UPS won't fit there. There is actually no > space for an UPS. The main problem is that customers are actually > pulling the plug to restart the system, to charge their laptops or > mobile phones or just for the fun of it. you can get UPS systems that are PCI cards, completly internal. they still may not fit, but you at least have a chance. > The client base image is a read-only system (Knoppix Like) with an extra > hard disk for swap and other informations like syslog messages. Since > there are no administrators close to the client system the client itself > needs to have the capability to send all the missing log information > between a network failure and an immediate power down to the central > server for error analysis since those are usualy the most important once > to pinpoint the cause of the inital error. > > My approach would be to use a block device directly since a file system > if fault-prone if you shut down the system immediatly. Each entry > including the header information guarded by a checksum value. It would > be actually something like a fixed array based queue just that it would > store the information in a block device. But this is just an inital thought. you are inventing a new filesystem here. it's not that easy to be reliable becouse the disk can lie to you. unless you are doing interesting things at the ATA/SCSI command level the disk may re-order your writes and may cache them in memory on the drive for an unknown time before actually writing them if you need reliable writes at anything close to a reasonable speed you need to have a battery backed cache or solid state drive in your machine (and the solid state drives are not all fast to write to) David Lang >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if >>> >> you >> >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 14:05:42 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:05:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E36796.9060509@ecker-software.de> Hi, should I use 3.18.4 (latest stable, I would preferr that one) or do I need the latest development version? I would actually alter queue.c directly changing the fctl flags in the disk based queue (O_DIRECT,O_SYNC,.O_NOATIME). Performance is not really an issue. There will be only 1000 to 2000 Messages per hour in peak times bye David Ecker Rainer Gerhards schrieb: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. > > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. > > Performance, of course, will be extremely terrible with such a setup... > > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >>> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >>> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:07:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:07:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > One thing I forgot to mention: a pure disk queue (not a disk-assisted > one) gets you as close to your goal as possible (well, mostly - we > could, at a considerable performance expense, require synced writing). > With that case, all data is immediately stored on disk. You can > configure it to also write the meta data out immediately (and again with > sync, not yet supported). However, you still have a window of exposure, > for example if the power loss happens right in the middle of when the > disk actually writes data to the disk sector. > > I still wonder why this scenario would be useful to address... not all uses of rsyslog are for simple system logs. it's a good general purpose log tool, and there are some cases where you want to be as sure as you possibly can be that once a message has been acknowledged it has no chance of being lost. useing some form of solid-state reliable storage (battery backed ram on a raid controller, a battery backed ram disk, a flash disk) it is possible (but not nessasarily cheap) to get the ability to do tens to hundreds of thousands of writes + syncs per second David Lang > Rainer > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. >> >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:07:32 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:07:32 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> Message-ID: <48E36804.3030107@ecker-software.de> Hi David, the actuall problem was that the qi file was not correct after an immediate restart. All messages were actually correctly written except the qi file. bye David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, David Ecker wrote: > >> Hi, >> >> I am looking for a failsafe solution to store syslog messages localy >> until they could be send later. I already looked at the disk based >> memory queue and the disk based queue. Both queue's don't work if you >> just power down the system immediatly actually loosing the whole queue. > > are you sure about the disk based queue? > > per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based > queue can be set to do a commit of the metadata after each message. > > Disk Queues > > Disk queues use disk drives for buffering. The important fact is that > the always use the disk and do not buffer anything in memory. Thus, > the queue is ultra-reliable, but by far the slowest mode. For regular > use cases, this queue mode is not recommended. It is useful if log > data is so important that it must not be lost, even in extreme cases. > > When a disk queue is written, it is done in chunks. Each chunk > receives its individual file. Files are named with a prefix (set via > the "$QueueFilename" config directive) and followed by a > 7-digit number (starting at one and incremented for each file). Chunks > are 10mb by default, a different size can be set via > the"$QueueMaxFileSize" config directive. Note that the size > limit is not a sharp one: rsyslog always writes one complete queue > entry, even if it violates the size limit. So chunks are actually a > little but (usually less than 1k) larger then the configured size. > Each chunk also has a different size for the same reason. If you > observe different chunk sizes, you can relax: this is not a problem. > > Writing in chunks is used so that processed data can quickly be > deleted and is free for other uses - while at the same time keeping no > artificial upper limit on disk space used. If a disk quota is set > (instructions further below), be sure that the quota/chunk size allows > at least two chunks to be written. Rsyslog currently does not check > that and will fail miserably if a single chunk is over the quota. > > Creating new chunks costs performance but provides quicker ability to > free disk space. The 10mb default is considered a good compromise > between these two. However, it may make sense to adapt these settings > to local policies. For example, if a disk queue is written on a > dedicated 200gb disk, it may make sense to use a 2gb (or even larger) > chunk size. > > Please note, however, that the disk queue by default does not update > its housekeeping structures every time it writes to disk. This is for > performance reasons. In the event of failure, data will still be lost > (except when manually is mangled with the file structures). However, > disk queues can be set to write bookkeeping information on checkpoints > (every n records), so that this can be made ultra-reliable, too. If > the checkpoint interval is set to one, no data can be lost, but the > queue is exceptionally slow. > > Each queue can be placed on a different disk for best performance > and/or isolation. This is currently selected by specifying different > $WorkDirectory config directives before the queue creation statement. > > To create a disk queue, use the "$QueueType Disk" config > directive. Checkpoint intervals can be specified via > "$QueueCheckpointInterval", with 0 meaning no checkpoints. > > > > > > you also need to specificly enable syncing (from > http://www.rsyslog.com/doc-v3compatibility.html ) > > Output File Syncing > Rsyslogd tries to keep as compatible to stock syslogd as possible. As > such, it retained stock syslogd's default of syncing every file write > if not specified otherwise (by placing a dash in front of the output > file name). While this was a useful feature in past days where > hardware was much less reliable and UPS seldom, this no longer is > useful in today's worl. Instead, the syncing is a high performace hit. > With it, rsyslogd writes files around 50 *times* slower than without > it. It also affects overall system performance due to the high IO > activity. In rsyslog v3, syncing has been turned off by default. This > is done via a specific configuration directive "$ActionFileEnableSync > on/off" which is off by default. So even if rsyslogd finds sync > selector lines, it ignores them by default. In order to enable file > syncing, the administrator must specify "$ActionFileEnableSync on" at > the top of rsyslog.conf. This ensures that syncing only happens in > some installations where the administrator actually wanted that > (performance-intense) feature. In the fast majority of cases (if not > all), this dramatically increases rsyslogd performance without any > negative effects. > > > >> I already looked at queue.c and it seemed to me that both queues were >> not designed for that kind of failure, but I could be wrong there. Since >> an immediate power down of the system is the major failure which will >> occure pretty often I need to create a soltution there. > > with checkpoint interval set to 1 and syncing enabled the data should > be in on the disk safely (assuming you have hardware that supports > this) and a power-off won't affect it. > > David Lang > > > >> Did you already start to develop something addressing that problem? >> Could you help me extend rsyslog (3.18.4) so that I can develop a new >> queue myself? I would contribute the code to the rsyslog project if you >> would like afterwards. >> >> bye >> David Ecker >> > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > ------------------------------------------------------------------------ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:09:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:09:33 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E36796.9060509@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <48E36796.9060509@ecker-software.de> Message-ID: <1222862973.2682.456.camel@rgf9dev.intern.adiscon.com> I am currently working on the queue engine in the devel branch. In that light, it could make sense to start with that branch. Also, I won't make any non-bug related changes in the stable version, so if you run into anything I can quickly change, that will happen in devel. Rainer On Wed, 2008-10-01 at 14:05 +0200, David Ecker wrote: > Hi, > > should I use 3.18.4 (latest stable, I would preferr that one) or do I > need the latest development version? I would actually alter queue.c > directly changing the fctl flags in the disk based queue > (O_DIRECT,O_SYNC,.O_NOATIME). > > Performance is not really an issue. There will be only 1000 to 2000 > Messages per hour in peak times > > bye > David Ecker > > Rainer Gerhards schrieb: > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > > > Performance, of course, will be extremely terrible with such a setup... > > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >>> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >>> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Wed Oct 1 14:09:53 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Wed, 1 Oct 2008 14:09:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: I installed the snapshot but apparently the issue persists. If there's something else that I can do, now or when you have more time, let me know. Regards, Luigi From rgerhards at hq.adiscon.com Wed Oct 1 14:12:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:12:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > One thing I forgot to mention: a pure disk queue (not a disk-assisted > > one) gets you as close to your goal as possible (well, mostly - we > > could, at a considerable performance expense, require synced writing). > > With that case, all data is immediately stored on disk. You can > > configure it to also write the meta data out immediately (and again with > > sync, not yet supported). However, you still have a window of exposure, > > for example if the power loss happens right in the middle of when the > > disk actually writes data to the disk sector. > > > > I still wonder why this scenario would be useful to address... > > not all uses of rsyslog are for simple system logs. it's a good general > purpose log tool, and there are some cases where you want to be as sure as > you possibly can be that once a message has been acknowledged it has no > chance of being lost. I designed the engine for audit-class reliability. However, I assumed that the rest of the system is also playing in that class. Doing everything with a potential power failure in mind creates a lot of extra demands. And I have never heard of anybody doing serious datacenter work without a proper UPS. Is this *really* an issue? Rainer > > useing some form of solid-state reliable storage (battery backed ram on a > raid controller, a battery backed ram disk, a flash disk) it is possible > (but not nessasarily cheap) to get the ability to do tens to hundreds of > thousands of writes + syncs per second > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: > >> Hi, > >> > >> I am looking for a failsafe solution to store syslog messages localy > >> until they could be send later. I already looked at the disk based > >> memory queue and the disk based queue. Both queue's don't work if you > >> just power down the system immediatly actually loosing the whole queue. > >> I already looked at queue.c and it seemed to me that both queues were > >> not designed for that kind of failure, but I could be wrong there. Since > >> an immediate power down of the system is the major failure which will > >> occure pretty often I need to create a soltution there. > >> > >> Did you already start to develop something addressing that problem? > >> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >> queue myself? I would contribute the code to the rsyslog project if you > >> would like afterwards. > >> > >> bye > >> David Ecker > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 14:15:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:15:14 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222863314.2682.463.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:09 +0200, Luigi Perroti wrote: > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. That's interesting. Could you please get me a debug log (you can mail it privately). Here is info about how to do that: http://www.rsyslog.com/doc-troubleshoot.html Rainer From david at lang.hm Wed Oct 1 14:25:18 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:25:18 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > the file syncing mentioned in the compatibility doc applies to the > output action, only. ouch. > The queue does never do synchronous writes - I always assumed that a > critical system would have a UPS and could never think (so far) about a > valid reason for not having it. So the queue would need to have an extra > option to do sync writes. Obviously, that's not a big deal. good > Performance, of course, will be extremely terrible with such a setup... only if you have to wait for a spinning disk to do the write. this is the same problem that databases have. they need to guarentee that once the database tells the writing program that the data is written it will be there even if the system looses power immediatly. if you run a database on standard desktop hardware (and it doesn't have this safety disabled) you cannot do more then about 80 writes/second. If you upgrade to the super speedy 15K rpm drives you can do ~160 writes/second. given that you need to write the data + metadata it gets even uglier, so what the databases do (and some journaling filesystems) is to write a log that says what they are going to do, sync that, and then later write the data to the actual files (updating the journal when they complete the write) it sounds like you order your write correctly for a disk-based queue, but you would need the option of issuing the syncs (probably when you do the checkpoints) if you do this on the wrong hardware (say a laptop 5200 rpm drive or the wrong flash drive), the fact that you need to do four writes per log entry (data to queue, metadata to queue, data to output, update metadata for queue) could drop you to below 15 logs/sec (60/4 but then you loose time to seeking as well) however, with the correct drive to write to (say a $2,400 80G fusion-io flash card that can do ~100k IO ops/sec) you should be able to sustain 20,000 logs/sec. realisticly very few people need the sustained write capacity that you would get from such a setup. but if you go with a $500-$700 raid card with a battery-backed cache you get very similar performance, but with some possibility that you can't sustain it forever. David Lang > Rainer > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, David Ecker wrote: >> >>> Hi, >>> >>> I am looking for a failsafe solution to store syslog messages localy >>> until they could be send later. I already looked at the disk based >>> memory queue and the disk based queue. Both queue's don't work if you >>> just power down the system immediatly actually loosing the whole queue. >> >> are you sure about the disk based queue? >> >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >> can be set to do a commit of the metadata after each message. >> >> Disk Queues >> >> Disk queues use disk drives for buffering. The important fact is that the >> always use the disk and do not buffer anything in memory. Thus, the queue >> is ultra-reliable, but by far the slowest mode. For regular use cases, >> this queue mode is not recommended. It is useful if log data is so >> important that it must not be lost, even in extreme cases. >> >> When a disk queue is written, it is done in chunks. Each chunk receives >> its individual file. Files are named with a prefix (set via the >> "$QueueFilename" config directive) and followed by a 7-digit >> number (starting at one and incremented for each file). Chunks are 10mb by >> default, a different size can be set via the"$QueueMaxFileSize" >> config directive. Note that the size limit is not a sharp one: rsyslog >> always writes one complete queue entry, even if it violates the size >> limit. So chunks are actually a little but (usually less than 1k) larger >> then the configured size. Each chunk also has a different size for the >> same reason. If you observe different chunk sizes, you can relax: this is >> not a problem. >> >> Writing in chunks is used so that processed data can quickly be deleted >> and is free for other uses - while at the same time keeping no artificial >> upper limit on disk space used. If a disk quota is set (instructions >> further below), be sure that the quota/chunk size allows at least two >> chunks to be written. Rsyslog currently does not check that and will fail >> miserably if a single chunk is over the quota. >> >> Creating new chunks costs performance but provides quicker ability to free >> disk space. The 10mb default is considered a good compromise between these >> two. However, it may make sense to adapt these settings to local policies. >> For example, if a disk queue is written on a dedicated 200gb disk, it may >> make sense to use a 2gb (or even larger) chunk size. >> >> Please note, however, that the disk queue by default does not update its >> housekeeping structures every time it writes to disk. This is for >> performance reasons. In the event of failure, data will still be lost >> (except when manually is mangled with the file structures). However, disk >> queues can be set to write bookkeeping information on checkpoints (every n >> records), so that this can be made ultra-reliable, too. If the checkpoint >> interval is set to one, no data can be lost, but the queue is >> exceptionally slow. >> >> Each queue can be placed on a different disk for best performance and/or >> isolation. This is currently selected by specifying different >> $WorkDirectory config directives before the queue creation statement. >> >> To create a disk queue, use the "$QueueType Disk" config >> directive. Checkpoint intervals can be specified via >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >> >> >> >> >> >> you also need to specificly enable syncing (from >> http://www.rsyslog.com/doc-v3compatibility.html ) >> >> Output File Syncing >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >> such, it retained stock syslogd's default of syncing every file write if >> not specified otherwise (by placing a dash in front of the output file >> name). While this was a useful feature in past days where hardware was >> much less reliable and UPS seldom, this no longer is useful in today's >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >> writes files around 50 *times* slower than without it. It also affects >> overall system performance due to the high IO activity. In rsyslog v3, >> syncing has been turned off by default. This is done via a specific >> configuration directive "$ActionFileEnableSync on/off" which is off by >> default. So even if rsyslogd finds sync selector lines, it ignores them by >> default. In order to enable file syncing, the administrator must specify >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >> syncing only happens in some installations where the administrator >> actually wanted that (performance-intense) feature. In the fast majority >> of cases (if not all), this dramatically increases rsyslogd performance >> without any negative effects. >> >> >> >>> I already looked at queue.c and it seemed to me that both queues were >>> not designed for that kind of failure, but I could be wrong there. Since >>> an immediate power down of the system is the major failure which will >>> occure pretty often I need to create a soltution there. >> >> with checkpoint interval set to 1 and syncing enabled the data should be >> in on the disk safely (assuming you have hardware that supports this) and >> a power-off won't affect it. >> >> David Lang >> >> >> >>> Did you already start to develop something addressing that problem? >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>> queue myself? I would contribute the code to the rsyslog project if you >>> would like afterwards. >>> >>> bye >>> David Ecker >>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:29:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:29:03 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David, > > > > the file syncing mentioned in the compatibility doc applies to the > > output action, only. > > ouch. > > > The queue does never do synchronous writes - I always assumed that a > > critical system would have a UPS and could never think (so far) about a > > valid reason for not having it. So the queue would need to have an extra > > option to do sync writes. Obviously, that's not a big deal. > > good > > > Performance, of course, will be extremely terrible with such a setup... > > only if you have to wait for a spinning disk to do the write. I agree to the rest of your argument below. But the question raised here was in regard to a system without any battery backup. So I would need to wait. Even then, in the worst case, I think it would be possible that the disk does only a partial write. I am not sure if that's really the case with today's disk drives (which I think have capacitors to prevent this scenario), but with past drives this could happen (I know all too well - a few years ago that cost me a weekend ;)). Rainer > > this is the same problem that databases have. they need to guarentee that > once the database tells the writing program that the data is written it > will be there even if the system looses power immediatly. > > if you run a database on standard desktop hardware (and it doesn't have > this safety disabled) you cannot do more then about 80 writes/second. If > you upgrade to the super speedy 15K rpm drives you can do ~160 > writes/second. > > given that you need to write the data + metadata it gets even uglier, so > what the databases do (and some journaling filesystems) is to write a log > that says what they are going to do, sync that, and then later write the > data to the actual files (updating the journal when they complete the > write) > > it sounds like you order your write correctly for a disk-based queue, but > you would need the option of issuing the syncs (probably when you do the > checkpoints) > > if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > wrong flash drive), the fact that you need to do four writes per log entry > (data to queue, metadata to queue, data to output, update metadata for > queue) could drop you to below 15 logs/sec (60/4 but then you loose time > to seeking as well) > > however, with the correct drive to write to (say a $2,400 80G fusion-io > flash card that can do ~100k IO ops/sec) you should be able to sustain > 20,000 logs/sec. > > realisticly very few people need the sustained write capacity that you > would get from such a setup. but if you go with a $500-$700 raid card with > a battery-backed cache you get very similar performance, but with some > possibility that you can't sustain it forever. > > David Lang > > > Rainer > > > > On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >>> Hi, > >>> > >>> I am looking for a failsafe solution to store syslog messages localy > >>> until they could be send later. I already looked at the disk based > >>> memory queue and the disk based queue. Both queue's don't work if you > >>> just power down the system immediatly actually loosing the whole queue. > >> > >> are you sure about the disk based queue? > >> > >> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >> can be set to do a commit of the metadata after each message. > >> > >> Disk Queues > >> > >> Disk queues use disk drives for buffering. The important fact is that the > >> always use the disk and do not buffer anything in memory. Thus, the queue > >> is ultra-reliable, but by far the slowest mode. For regular use cases, > >> this queue mode is not recommended. It is useful if log data is so > >> important that it must not be lost, even in extreme cases. > >> > >> When a disk queue is written, it is done in chunks. Each chunk receives > >> its individual file. Files are named with a prefix (set via the > >> "$QueueFilename" config directive) and followed by a 7-digit > >> number (starting at one and incremented for each file). Chunks are 10mb by > >> default, a different size can be set via the"$QueueMaxFileSize" > >> config directive. Note that the size limit is not a sharp one: rsyslog > >> always writes one complete queue entry, even if it violates the size > >> limit. So chunks are actually a little but (usually less than 1k) larger > >> then the configured size. Each chunk also has a different size for the > >> same reason. If you observe different chunk sizes, you can relax: this is > >> not a problem. > >> > >> Writing in chunks is used so that processed data can quickly be deleted > >> and is free for other uses - while at the same time keeping no artificial > >> upper limit on disk space used. If a disk quota is set (instructions > >> further below), be sure that the quota/chunk size allows at least two > >> chunks to be written. Rsyslog currently does not check that and will fail > >> miserably if a single chunk is over the quota. > >> > >> Creating new chunks costs performance but provides quicker ability to free > >> disk space. The 10mb default is considered a good compromise between these > >> two. However, it may make sense to adapt these settings to local policies. > >> For example, if a disk queue is written on a dedicated 200gb disk, it may > >> make sense to use a 2gb (or even larger) chunk size. > >> > >> Please note, however, that the disk queue by default does not update its > >> housekeeping structures every time it writes to disk. This is for > >> performance reasons. In the event of failure, data will still be lost > >> (except when manually is mangled with the file structures). However, disk > >> queues can be set to write bookkeeping information on checkpoints (every n > >> records), so that this can be made ultra-reliable, too. If the checkpoint > >> interval is set to one, no data can be lost, but the queue is > >> exceptionally slow. > >> > >> Each queue can be placed on a different disk for best performance and/or > >> isolation. This is currently selected by specifying different > >> $WorkDirectory config directives before the queue creation statement. > >> > >> To create a disk queue, use the "$QueueType Disk" config > >> directive. Checkpoint intervals can be specified via > >> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >> > >> > >> > >> > >> > >> you also need to specificly enable syncing (from > >> http://www.rsyslog.com/doc-v3compatibility.html ) > >> > >> Output File Syncing > >> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >> such, it retained stock syslogd's default of syncing every file write if > >> not specified otherwise (by placing a dash in front of the output file > >> name). While this was a useful feature in past days where hardware was > >> much less reliable and UPS seldom, this no longer is useful in today's > >> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >> writes files around 50 *times* slower than without it. It also affects > >> overall system performance due to the high IO activity. In rsyslog v3, > >> syncing has been turned off by default. This is done via a specific > >> configuration directive "$ActionFileEnableSync on/off" which is off by > >> default. So even if rsyslogd finds sync selector lines, it ignores them by > >> default. In order to enable file syncing, the administrator must specify > >> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >> syncing only happens in some installations where the administrator > >> actually wanted that (performance-intense) feature. In the fast majority > >> of cases (if not all), this dramatically increases rsyslogd performance > >> without any negative effects. > >> > >> > >> > >>> I already looked at queue.c and it seemed to me that both queues were > >>> not designed for that kind of failure, but I could be wrong there. Since > >>> an immediate power down of the system is the major failure which will > >>> occure pretty often I need to create a soltution there. > >> > >> with checkpoint interval set to 1 and syncing enabled the data should be > >> in on the disk safely (assuming you have hardware that supports this) and > >> a power-off won't affect it. > >> > >> David Lang > >> > >> > >> > >>> Did you already start to develop something addressing that problem? > >>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>> queue myself? I would contribute the code to the rsyslog project if you > >>> would like afterwards. > >>> > >>> bye > >>> David Ecker > >>> > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 14:35:17 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:35:17 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:07 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> One thing I forgot to mention: a pure disk queue (not a disk-assisted >>> one) gets you as close to your goal as possible (well, mostly - we >>> could, at a considerable performance expense, require synced writing). >>> With that case, all data is immediately stored on disk. You can >>> configure it to also write the meta data out immediately (and again with >>> sync, not yet supported). However, you still have a window of exposure, >>> for example if the power loss happens right in the middle of when the >>> disk actually writes data to the disk sector. >>> >>> I still wonder why this scenario would be useful to address... >> >> not all uses of rsyslog are for simple system logs. it's a good general >> purpose log tool, and there are some cases where you want to be as sure as >> you possibly can be that once a message has been acknowledged it has no >> chance of being lost. > > I designed the engine for audit-class reliability. However, I assumed > that the rest of the system is also playing in that class. Doing > everything with a potential power failure in mind creates a lot of extra > demands. And I have never heard of anybody doing serious datacenter work > without a proper UPS. Is this *really* an issue? Yes. UPSs fail. generators fail power cords come loose. power cords get unplugged by someone who thinks they are unplugging a different system people bump power switches on power strips. power supplies are defective I had one production outage where a visiting tech pulled a power cord from an overhead plug and dropped it on the ground, where it happened to hit the power switch on a power strip. I've had high-end systems with redundant power supplies go down becouse of faulty hardware that decided to disble both power supplies at once (it turned out that there was a defect in the whole batch of servers, but it took IBM several weeks to figure out what was going on) I've had UPS systems blow up (literally) I've had a datacenter go down becouse the it was running on generator power (due to other issues), and the refueling guy filled the tank incorrectly and got air bubbles into the fuel system, a few min later the 500Kw diesel generator couldn't maintain constant speed and the safety triggers kicked in and disabled it. it's amazing the things that happen in real-life David Lang > Rainer >> >> useing some form of solid-state reliable storage (battery backed ram on a >> raid controller, a battery backed ram disk, a flash disk) it is possible >> (but not nessasarily cheap) to get the ability to do tens to hundreds of >> thousands of writes + syncs per second >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 12:00 +0200, David Ecker wrote: >>>> Hi, >>>> >>>> I am looking for a failsafe solution to store syslog messages localy >>>> until they could be send later. I already looked at the disk based >>>> memory queue and the disk based queue. Both queue's don't work if you >>>> just power down the system immediatly actually loosing the whole queue. >>>> I already looked at queue.c and it seemed to me that both queues were >>>> not designed for that kind of failure, but I could be wrong there. Since >>>> an immediate power down of the system is the major failure which will >>>> occure pretty often I need to create a soltution there. >>>> >>>> Did you already start to develop something addressing that problem? >>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>> queue myself? I would contribute the code to the rsyslog project if you >>>> would like afterwards. >>>> >>>> bye >>>> David Ecker >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 14:39:54 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 05:39:54 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >> >> ouch. >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >> >> good >> >>> Performance, of course, will be extremely terrible with such a setup... >> >> only if you have to wait for a spinning disk to do the write. > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. no UPS is not nessasarily the same as no battey backup. you could use a compact flash drive and probably get better performance/reliability than spinning disks with no battery at all. > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). current disks do not have capacitors to prevent partial writes or to flush their caches. but options like the linux ext3 data-journaled make it so that you have your data in the journal safely, and the various solid-state options solve that problem. David Lang > Rainer > >> >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 14:45:06 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 14:45:06 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E370D2.7080901@ecker-software.de> Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David, >>> >>> the file syncing mentioned in the compatibility doc applies to the >>> output action, only. >>> >> ouch. >> >> >>> The queue does never do synchronous writes - I always assumed that a >>> critical system would have a UPS and could never think (so far) about a >>> valid reason for not having it. So the queue would need to have an extra >>> option to do sync writes. Obviously, that's not a big deal. >>> >> good >> >> >>> Performance, of course, will be extremely terrible with such a setup... >>> >> only if you have to wait for a spinning disk to do the write. >> > > I agree to the rest of your argument below. But the question raised here > was in regard to a system without any battery backup. So I would need to > wait. > > Even then, in the worst case, I think it would be possible that the disk > does only a partial write. I am not sure if that's really the case with > today's disk drives (which I think have capacitors to prevent this > scenario), but with past drives this could happen (I know all too well - > a few years ago that cost me a weekend ;)). > > Rainer > Hi, as long as you do sector based writes (512 byte per sector, usual) you can be sure that the write wasn"t partial.. Writing more than one sector or not starting at a correct offset (n*512,n=0,1,2,...x) might result in a partial write. I'll already tested that with my devel client here. So fencing each sector with a crc32 value would help detecting errors during a write operation. This is actually only a problem if you are writing directly to a block device like any filesystem does and yes, reordering is definitly a problem. So validating the content written to the disk afterwards is important. If writing through a filesystem reserving space in the destination file beforehand actually minimizes errors since the file system table doesn't have to be updated (you should also use the Flag O_NOATIME for that case). See for example VMWare ESX VMDK file handling. David > >> this is the same problem that databases have. they need to guarentee that >> once the database tells the writing program that the data is written it >> will be there even if the system looses power immediatly. >> >> if you run a database on standard desktop hardware (and it doesn't have >> this safety disabled) you cannot do more then about 80 writes/second. If >> you upgrade to the super speedy 15K rpm drives you can do ~160 >> writes/second. >> >> given that you need to write the data + metadata it gets even uglier, so >> what the databases do (and some journaling filesystems) is to write a log >> that says what they are going to do, sync that, and then later write the >> data to the actual files (updating the journal when they complete the >> write) >> >> it sounds like you order your write correctly for a disk-based queue, but >> you would need the option of issuing the syncs (probably when you do the >> checkpoints) >> >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >> wrong flash drive), the fact that you need to do four writes per log entry >> (data to queue, metadata to queue, data to output, update metadata for >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >> to seeking as well) >> >> however, with the correct drive to write to (say a $2,400 80G fusion-io >> flash card that can do ~100k IO ops/sec) you should be able to sustain >> 20,000 logs/sec. >> >> realisticly very few people need the sustained write capacity that you >> would get from such a setup. but if you go with a $500-$700 raid card with >> a battery-backed cache you get very similar performance, but with some >> possibility that you can't sustain it forever. >> >> David Lang >> >> >>> Rainer >>> >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>> >>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>> >>>> >>>>> Hi, >>>>> >>>>> I am looking for a failsafe solution to store syslog messages localy >>>>> until they could be send later. I already looked at the disk based >>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>> just power down the system immediatly actually loosing the whole queue. >>>>> >>>> are you sure about the disk based queue? >>>> >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>> can be set to do a commit of the metadata after each message. >>>> >>>> Disk Queues >>>> >>>> Disk queues use disk drives for buffering. The important fact is that the >>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>> this queue mode is not recommended. It is useful if log data is so >>>> important that it must not be lost, even in extreme cases. >>>> >>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>> its individual file. Files are named with a prefix (set via the >>>> "$QueueFilename" config directive) and followed by a 7-digit >>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>> default, a different size can be set via the"$QueueMaxFileSize" >>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>> always writes one complete queue entry, even if it violates the size >>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>> then the configured size. Each chunk also has a different size for the >>>> same reason. If you observe different chunk sizes, you can relax: this is >>>> not a problem. >>>> >>>> Writing in chunks is used so that processed data can quickly be deleted >>>> and is free for other uses - while at the same time keeping no artificial >>>> upper limit on disk space used. If a disk quota is set (instructions >>>> further below), be sure that the quota/chunk size allows at least two >>>> chunks to be written. Rsyslog currently does not check that and will fail >>>> miserably if a single chunk is over the quota. >>>> >>>> Creating new chunks costs performance but provides quicker ability to free >>>> disk space. The 10mb default is considered a good compromise between these >>>> two. However, it may make sense to adapt these settings to local policies. >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>> make sense to use a 2gb (or even larger) chunk size. >>>> >>>> Please note, however, that the disk queue by default does not update its >>>> housekeeping structures every time it writes to disk. This is for >>>> performance reasons. In the event of failure, data will still be lost >>>> (except when manually is mangled with the file structures). However, disk >>>> queues can be set to write bookkeeping information on checkpoints (every n >>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>> interval is set to one, no data can be lost, but the queue is >>>> exceptionally slow. >>>> >>>> Each queue can be placed on a different disk for best performance and/or >>>> isolation. This is currently selected by specifying different >>>> $WorkDirectory config directives before the queue creation statement. >>>> >>>> To create a disk queue, use the "$QueueType Disk" config >>>> directive. Checkpoint intervals can be specified via >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>> >>>> >>>> >>>> >>>> >>>> you also need to specificly enable syncing (from >>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>> >>>> Output File Syncing >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>> such, it retained stock syslogd's default of syncing every file write if >>>> not specified otherwise (by placing a dash in front of the output file >>>> name). While this was a useful feature in past days where hardware was >>>> much less reliable and UPS seldom, this no longer is useful in today's >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>> writes files around 50 *times* slower than without it. It also affects >>>> overall system performance due to the high IO activity. In rsyslog v3, >>>> syncing has been turned off by default. This is done via a specific >>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>> default. In order to enable file syncing, the administrator must specify >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>> syncing only happens in some installations where the administrator >>>> actually wanted that (performance-intense) feature. In the fast majority >>>> of cases (if not all), this dramatically increases rsyslogd performance >>>> without any negative effects. >>>> >>>> >>>> >>>> >>>>> I already looked at queue.c and it seemed to me that both queues were >>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>> an immediate power down of the system is the major failure which will >>>>> occure pretty often I need to create a soltution there. >>>>> >>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>> in on the disk safely (assuming you have hardware that supports this) and >>>> a power-off won't affect it. >>>> >>>> David Lang >>>> >>>> >>>> >>>> >>>>> Did you already start to develop something addressing that problem? >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>> would like afterwards. >>>>> >>>>> bye >>>>> David Ecker >>>>> >>>>> >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 14:55:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:55:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E370D2.7080901@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> Message-ID: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: [snip] > as long as you do sector based writes (512 byte per sector, usual) you > can be sure that the write wasn"t partial.. Writing more than one sector > or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > a partial write. I'll already tested that with my devel client here. So > fencing each sector with a crc32 value would help detecting errors > during a write operation. This is actually only a problem if you are > writing directly to a block device like any filesystem does and yes, > reordering is definitly a problem. So validating the content written to > the disk afterwards is important. > > If writing through a filesystem reserving space in the destination file > beforehand actually minimizes errors since the file system table doesn't > have to be updated (you should also use the Flag O_NOATIME for that > case). See for example VMWare ESX VMDK file handling. Well, first of all let me re-iterate that I do not intend to do a block device driver for rsyslog (but I definitely do not object getting one contributed ;)). Still thinking about the case and thinking about non-solid-state, non-internal-battery-backed-up disk, I can't see how you can be sure the data will be written. David just told me there are no capacitors. So if power fails, it fails rather quickly. So how can you be sure the disk will be able to finish writing that sector? Let's say the drive has begun to write the sector and been able to write the first 5 bytes. Now power fails. No capacitors, no battery-backup, so why should there be enough power to drive the disk write head for another 507 bytes? It the drives assures it can do that, it needs capacitors - doesn't it? Am I overlooking something obvious? Rainer From rgerhards at hq.adiscon.com Wed Oct 1 14:57:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 14:57:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> David, going back to the higher layer: do you say that immediate power failure is a case that you consider needed to be addressed in an enterprise logging system? Anybody else with an opinion? Rainer On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> > >>> David, > >>> > >>> the file syncing mentioned in the compatibility doc applies to the > >>> output action, only. > >> > >> ouch. > >> > >>> The queue does never do synchronous writes - I always assumed that a > >>> critical system would have a UPS and could never think (so far) about a > >>> valid reason for not having it. So the queue would need to have an extra > >>> option to do sync writes. Obviously, that's not a big deal. > >> > >> good > >> > >>> Performance, of course, will be extremely terrible with such a setup... > >> > >> only if you have to wait for a spinning disk to do the write. > > > > I agree to the rest of your argument below. But the question raised here > > was in regard to a system without any battery backup. So I would need to > > wait. > > no UPS is not nessasarily the same as no battey backup. > > you could use a compact flash drive and probably get better > performance/reliability than spinning disks with no battery at all. > > > Even then, in the worst case, I think it would be possible that the disk > > does only a partial write. I am not sure if that's really the case with > > today's disk drives (which I think have capacitors to prevent this > > scenario), but with past drives this could happen (I know all too well - > > a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. > > David Lang > > > Rainer > > > >> > >> this is the same problem that databases have. they need to guarentee that > >> once the database tells the writing program that the data is written it > >> will be there even if the system looses power immediatly. > >> > >> if you run a database on standard desktop hardware (and it doesn't have > >> this safety disabled) you cannot do more then about 80 writes/second. If > >> you upgrade to the super speedy 15K rpm drives you can do ~160 > >> writes/second. > >> > >> given that you need to write the data + metadata it gets even uglier, so > >> what the databases do (and some journaling filesystems) is to write a log > >> that says what they are going to do, sync that, and then later write the > >> data to the actual files (updating the journal when they complete the > >> write) > >> > >> it sounds like you order your write correctly for a disk-based queue, but > >> you would need the option of issuing the syncs (probably when you do the > >> checkpoints) > >> > >> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the > >> wrong flash drive), the fact that you need to do four writes per log entry > >> (data to queue, metadata to queue, data to output, update metadata for > >> queue) could drop you to below 15 logs/sec (60/4 but then you loose time > >> to seeking as well) > >> > >> however, with the correct drive to write to (say a $2,400 80G fusion-io > >> flash card that can do ~100k IO ops/sec) you should be able to sustain > >> 20,000 logs/sec. > >> > >> realisticly very few people need the sustained write capacity that you > >> would get from such a setup. but if you go with a $500-$700 raid card with > >> a battery-backed cache you get very similar performance, but with some > >> possibility that you can't sustain it forever. > >> > >> David Lang > >> > >>> Rainer > >>> > >>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: > >>>> On Wed, 1 Oct 2008, David Ecker wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I am looking for a failsafe solution to store syslog messages localy > >>>>> until they could be send later. I already looked at the disk based > >>>>> memory queue and the disk based queue. Both queue's don't work if you > >>>>> just power down the system immediatly actually loosing the whole queue. > >>>> > >>>> are you sure about the disk based queue? > >>>> > >>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue > >>>> can be set to do a commit of the metadata after each message. > >>>> > >>>> Disk Queues > >>>> > >>>> Disk queues use disk drives for buffering. The important fact is that the > >>>> always use the disk and do not buffer anything in memory. Thus, the queue > >>>> is ultra-reliable, but by far the slowest mode. For regular use cases, > >>>> this queue mode is not recommended. It is useful if log data is so > >>>> important that it must not be lost, even in extreme cases. > >>>> > >>>> When a disk queue is written, it is done in chunks. Each chunk receives > >>>> its individual file. Files are named with a prefix (set via the > >>>> "$QueueFilename" config directive) and followed by a 7-digit > >>>> number (starting at one and incremented for each file). Chunks are 10mb by > >>>> default, a different size can be set via the"$QueueMaxFileSize" > >>>> config directive. Note that the size limit is not a sharp one: rsyslog > >>>> always writes one complete queue entry, even if it violates the size > >>>> limit. So chunks are actually a little but (usually less than 1k) larger > >>>> then the configured size. Each chunk also has a different size for the > >>>> same reason. If you observe different chunk sizes, you can relax: this is > >>>> not a problem. > >>>> > >>>> Writing in chunks is used so that processed data can quickly be deleted > >>>> and is free for other uses - while at the same time keeping no artificial > >>>> upper limit on disk space used. If a disk quota is set (instructions > >>>> further below), be sure that the quota/chunk size allows at least two > >>>> chunks to be written. Rsyslog currently does not check that and will fail > >>>> miserably if a single chunk is over the quota. > >>>> > >>>> Creating new chunks costs performance but provides quicker ability to free > >>>> disk space. The 10mb default is considered a good compromise between these > >>>> two. However, it may make sense to adapt these settings to local policies. > >>>> For example, if a disk queue is written on a dedicated 200gb disk, it may > >>>> make sense to use a 2gb (or even larger) chunk size. > >>>> > >>>> Please note, however, that the disk queue by default does not update its > >>>> housekeeping structures every time it writes to disk. This is for > >>>> performance reasons. In the event of failure, data will still be lost > >>>> (except when manually is mangled with the file structures). However, disk > >>>> queues can be set to write bookkeeping information on checkpoints (every n > >>>> records), so that this can be made ultra-reliable, too. If the checkpoint > >>>> interval is set to one, no data can be lost, but the queue is > >>>> exceptionally slow. > >>>> > >>>> Each queue can be placed on a different disk for best performance and/or > >>>> isolation. This is currently selected by specifying different > >>>> $WorkDirectory config directives before the queue creation statement. > >>>> > >>>> To create a disk queue, use the "$QueueType Disk" config > >>>> directive. Checkpoint intervals can be specified via > >>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> you also need to specificly enable syncing (from > >>>> http://www.rsyslog.com/doc-v3compatibility.html ) > >>>> > >>>> Output File Syncing > >>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As > >>>> such, it retained stock syslogd's default of syncing every file write if > >>>> not specified otherwise (by placing a dash in front of the output file > >>>> name). While this was a useful feature in past days where hardware was > >>>> much less reliable and UPS seldom, this no longer is useful in today's > >>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd > >>>> writes files around 50 *times* slower than without it. It also affects > >>>> overall system performance due to the high IO activity. In rsyslog v3, > >>>> syncing has been turned off by default. This is done via a specific > >>>> configuration directive "$ActionFileEnableSync on/off" which is off by > >>>> default. So even if rsyslogd finds sync selector lines, it ignores them by > >>>> default. In order to enable file syncing, the administrator must specify > >>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that > >>>> syncing only happens in some installations where the administrator > >>>> actually wanted that (performance-intense) feature. In the fast majority > >>>> of cases (if not all), this dramatically increases rsyslogd performance > >>>> without any negative effects. > >>>> > >>>> > >>>> > >>>>> I already looked at queue.c and it seemed to me that both queues were > >>>>> not designed for that kind of failure, but I could be wrong there. Since > >>>>> an immediate power down of the system is the major failure which will > >>>>> occure pretty often I need to create a soltution there. > >>>> > >>>> with checkpoint interval set to 1 and syncing enabled the data should be > >>>> in on the disk safely (assuming you have hardware that supports this) and > >>>> a power-off won't affect it. > >>>> > >>>> David Lang > >>>> > >>>> > >>>> > >>>>> Did you already start to develop something addressing that problem? > >>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new > >>>>> queue myself? I would contribute the code to the rsyslog project if you > >>>>> would like afterwards. > >>>>> > >>>>> bye > >>>>> David Ecker > >>>>> > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Wed Oct 1 15:09:08 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:09:08 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37674.9090408@ecker-software.de> Hi, [quote] Depending on the media and the block device driver design, individual sector writes may not be atomic. A physical sector in typical devices is 512 bytes. In most cases, physical sector writes are atomic (either completely written, or not modified at all). A truly reliable file system, however, cannot count on this. [/quote] In most cases it works but some way of validating the data is needed if you want ultra reliability, which I don't need. If the last few messages a few seconds before an immediate shutdown are lost but all other messages are send correctly afterwards then that would be OK in my case. I'll just test version 2.21.5 with the altered open behauvior. The disk based queue-array developed by myself is just a fallback solution if the disk-based queue doesn't work with an immediate shutdown. David Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. >> > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:11:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:11:09 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Sorry, I overlooked this mail in the big bunch of messages. That's good reasoning. To cover these scenarios, we need to do everything with syncing. This also means that you can not use any of the disk-assisted modes, because in these modes we always try to keep things in memory in order to save writes. So while you have convinced me things can go wrong, I'd still say that is is very unusual (at least very costly) to care for all these things. But, of course, there are situations where it is needed. I'll probably see that I provide a facility to open files in "always sync" mode, but that for sure will not be the default setting ;) But even with the fast solid state disks (and similar methods) you mention, I think there will be a severe impact on performance because everything now needs to go through two write (data+metadata) and two read (again, data+metadata) OS call where we currently simply update an in-memory structure. Just out of curiosity: do you expect the majority of you rollouts to be using such methods? Rainer On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > > ... And I have never heard of anybody doing serious datacenter work > > without a proper UPS. Is this *really* an issue? > > Yes. > > UPSs fail. > generators fail > power cords come loose. > power cords get unplugged by someone who thinks they are unplugging a > different system > people bump power switches on power strips. > power supplies are defective > > I had one production outage where a visiting tech pulled a power cord from > an overhead plug and dropped it on the ground, where it happened to hit > the power switch on a power strip. > > I've had high-end systems with redundant power supplies go down becouse of > faulty hardware that decided to disble both power supplies at once (it > turned out that there was a defect in the whole batch of servers, but it > took IBM several weeks to figure out what was going on) > > I've had UPS systems blow up (literally) > > I've had a datacenter go down becouse the it was running on generator > power (due to other issues), and the refueling guy filled the tank > incorrectly and got air bubbles into the fuel system, a few min later the > 500Kw diesel generator couldn't maintain constant speed and the safety > triggers kicked in and disabled it. > > it's amazing the things that happen in real-life From david at lang.hm Wed Oct 1 15:17:42 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:17:42 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. I'm replying out of order as I see things anyway > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. I think you are saying that we must use the disk-only mode which is correct. > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. absolutly!! > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) thanks. > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. given the performance gains that we have seen by eliminating syscalls, it will hurt to add these back in, even with solid-state disks. that being said, it looks like the output module is nowhere close to being the limit (when I could get a good, stable reading on it, it looked like it was eating ~15% cpu compared to the input module at 100%) so it may not make much of a difference. > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? absolutly not. I have one case I am considering (the one I am talking to you about more efficiant database writes) that would be this paranoid, but the rest of it will be optimized for speed (battery-backed disk caches on the final server, but everything else can just use ram) David Lang > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 1 15:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:17:42 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37674.9090408@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <48E37674.9090408@ecker-software.de> Message-ID: <1222867062.2682.492.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:09 +0200, David Ecker wrote: > Hi, > > [quote] > Depending on the media and the block device driver design, individual > sector writes may > not be atomic. A physical sector in typical devices is 512 bytes. In > most cases, physical > sector writes are atomic (either completely written, or not modified at > all). A truly > reliable file system, however, cannot count on this. > [/quote] > > In most cases it works exactly - in most cases. That means it does not work always. > but some way of validating the data how can you validate if there is no power and the machine is off? > is needed if > you want ultra reliability, which I don't need. If the last few messages > a few seconds before an immediate shutdown are lost but all other > messages are send correctly afterwards then that would be OK in my case. but we can not guarantee that, at least not in all cases. Let's assume the disk died in the middle of the write access. Chances are good you'll never be able to read that sector again. Using a journaling file system will help, but without it, you may just have destroyed the sector that contained the .qi file. So on next startup the .qi is either not readable at all or not pointing at the correct information. The end result can be total loss of information. This scenario is probably acceptable in your case, because it is really, really highly unlikely. But it still exists. > I'll just test version 2.21.5 with the altered open behauvior. The disk > based queue-array developed by myself is just a fallback solution if the > disk-based queue doesn't work with an immediate shutdown. If it does not work under the constraints described here, this would point to a problem in the queue implementation (I have to admit the reason to provide a capability to write periodic qi file updates was related to a scenario like this, though not thought in this extreme ;)). Rainer > > David > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > > > [snip] > > > > > >> as long as you do sector based writes (512 byte per sector, usual) you > >> can be sure that the write wasn"t partial.. Writing more than one sector > >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in > >> a partial write. I'll already tested that with my devel client here. So > >> fencing each sector with a crc32 value would help detecting errors > >> during a write operation. This is actually only a problem if you are > >> writing directly to a block device like any filesystem does and yes, > >> reordering is definitly a problem. So validating the content written to > >> the disk afterwards is important. > >> > >> If writing through a filesystem reserving space in the destination file > >> beforehand actually minimizes errors since the file system table doesn't > >> have to be updated (you should also use the Flag O_NOATIME for that > >> case). See for example VMWare ESX VMDK file handling. > >> > > > > Well, first of all let me re-iterate that I do not intend to do a block > > device driver for rsyslog (but I definitely do not object getting one > > contributed ;)). > > > > Still thinking about the case and thinking about non-solid-state, > > non-internal-battery-backed-up disk, I can't see how you can be sure the > > data will be written. David just told me there are no capacitors. So if > > power fails, it fails rather quickly. So how can you be sure the disk > > will be able to finish writing that sector? Let's say the drive has > > begun to write the sector and been able to write the first 5 bytes. Now > > power fails. No capacitors, no battery-backup, so why should there be > > enough power to drive the disk write head for another 507 bytes? It the > > drives assures it can do that, it needs capacitors - doesn't it? > > > > Am I overlooking something obvious? > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Oct 1 15:18:55 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:18:55 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? in an audit-grade logging system yes, in an enterprise-grade system probably not. David Lang > Anybody else with an opinion? > Rainer > > On Wed, 2008-10-01 at 05:39 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> On Wed, 2008-10-01 at 05:25 -0700, david at lang.hm wrote: >>>> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> David, >>>>> >>>>> the file syncing mentioned in the compatibility doc applies to the >>>>> output action, only. >>>> >>>> ouch. >>>> >>>>> The queue does never do synchronous writes - I always assumed that a >>>>> critical system would have a UPS and could never think (so far) about a >>>>> valid reason for not having it. So the queue would need to have an extra >>>>> option to do sync writes. Obviously, that's not a big deal. >>>> >>>> good >>>> >>>>> Performance, of course, will be extremely terrible with such a setup... >>>> >>>> only if you have to wait for a spinning disk to do the write. >>> >>> I agree to the rest of your argument below. But the question raised here >>> was in regard to a system without any battery backup. So I would need to >>> wait. >> >> no UPS is not nessasarily the same as no battey backup. >> >> you could use a compact flash drive and probably get better >> performance/reliability than spinning disks with no battery at all. >> >>> Even then, in the worst case, I think it would be possible that the disk >>> does only a partial write. I am not sure if that's really the case with >>> today's disk drives (which I think have capacitors to prevent this >>> scenario), but with past drives this could happen (I know all too well - >>> a few years ago that cost me a weekend ;)). >> >> current disks do not have capacitors to prevent partial writes or to flush >> their caches. but options like the linux ext3 data-journaled make it so >> that you have your data in the journal safely, and the various solid-state >> options solve that problem. >> >> David Lang >> >>> Rainer >>> >>>> >>>> this is the same problem that databases have. they need to guarentee that >>>> once the database tells the writing program that the data is written it >>>> will be there even if the system looses power immediatly. >>>> >>>> if you run a database on standard desktop hardware (and it doesn't have >>>> this safety disabled) you cannot do more then about 80 writes/second. If >>>> you upgrade to the super speedy 15K rpm drives you can do ~160 >>>> writes/second. >>>> >>>> given that you need to write the data + metadata it gets even uglier, so >>>> what the databases do (and some journaling filesystems) is to write a log >>>> that says what they are going to do, sync that, and then later write the >>>> data to the actual files (updating the journal when they complete the >>>> write) >>>> >>>> it sounds like you order your write correctly for a disk-based queue, but >>>> you would need the option of issuing the syncs (probably when you do the >>>> checkpoints) >>>> >>>> if you do this on the wrong hardware (say a laptop 5200 rpm drive or the >>>> wrong flash drive), the fact that you need to do four writes per log entry >>>> (data to queue, metadata to queue, data to output, update metadata for >>>> queue) could drop you to below 15 logs/sec (60/4 but then you loose time >>>> to seeking as well) >>>> >>>> however, with the correct drive to write to (say a $2,400 80G fusion-io >>>> flash card that can do ~100k IO ops/sec) you should be able to sustain >>>> 20,000 logs/sec. >>>> >>>> realisticly very few people need the sustained write capacity that you >>>> would get from such a setup. but if you go with a $500-$700 raid card with >>>> a battery-backed cache you get very similar performance, but with some >>>> possibility that you can't sustain it forever. >>>> >>>> David Lang >>>> >>>>> Rainer >>>>> >>>>> On Wed, 2008-10-01 at 04:55 -0700, david at lang.hm wrote: >>>>>> On Wed, 1 Oct 2008, David Ecker wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am looking for a failsafe solution to store syslog messages localy >>>>>>> until they could be send later. I already looked at the disk based >>>>>>> memory queue and the disk based queue. Both queue's don't work if you >>>>>>> just power down the system immediatly actually loosing the whole queue. >>>>>> >>>>>> are you sure about the disk based queue? >>>>>> >>>>>> per file:///usr/src/rsyslog-3.21.4/doc/queues.html the disk based queue >>>>>> can be set to do a commit of the metadata after each message. >>>>>> >>>>>> Disk Queues >>>>>> >>>>>> Disk queues use disk drives for buffering. The important fact is that the >>>>>> always use the disk and do not buffer anything in memory. Thus, the queue >>>>>> is ultra-reliable, but by far the slowest mode. For regular use cases, >>>>>> this queue mode is not recommended. It is useful if log data is so >>>>>> important that it must not be lost, even in extreme cases. >>>>>> >>>>>> When a disk queue is written, it is done in chunks. Each chunk receives >>>>>> its individual file. Files are named with a prefix (set via the >>>>>> "$QueueFilename" config directive) and followed by a 7-digit >>>>>> number (starting at one and incremented for each file). Chunks are 10mb by >>>>>> default, a different size can be set via the"$QueueMaxFileSize" >>>>>> config directive. Note that the size limit is not a sharp one: rsyslog >>>>>> always writes one complete queue entry, even if it violates the size >>>>>> limit. So chunks are actually a little but (usually less than 1k) larger >>>>>> then the configured size. Each chunk also has a different size for the >>>>>> same reason. If you observe different chunk sizes, you can relax: this is >>>>>> not a problem. >>>>>> >>>>>> Writing in chunks is used so that processed data can quickly be deleted >>>>>> and is free for other uses - while at the same time keeping no artificial >>>>>> upper limit on disk space used. If a disk quota is set (instructions >>>>>> further below), be sure that the quota/chunk size allows at least two >>>>>> chunks to be written. Rsyslog currently does not check that and will fail >>>>>> miserably if a single chunk is over the quota. >>>>>> >>>>>> Creating new chunks costs performance but provides quicker ability to free >>>>>> disk space. The 10mb default is considered a good compromise between these >>>>>> two. However, it may make sense to adapt these settings to local policies. >>>>>> For example, if a disk queue is written on a dedicated 200gb disk, it may >>>>>> make sense to use a 2gb (or even larger) chunk size. >>>>>> >>>>>> Please note, however, that the disk queue by default does not update its >>>>>> housekeeping structures every time it writes to disk. This is for >>>>>> performance reasons. In the event of failure, data will still be lost >>>>>> (except when manually is mangled with the file structures). However, disk >>>>>> queues can be set to write bookkeeping information on checkpoints (every n >>>>>> records), so that this can be made ultra-reliable, too. If the checkpoint >>>>>> interval is set to one, no data can be lost, but the queue is >>>>>> exceptionally slow. >>>>>> >>>>>> Each queue can be placed on a different disk for best performance and/or >>>>>> isolation. This is currently selected by specifying different >>>>>> $WorkDirectory config directives before the queue creation statement. >>>>>> >>>>>> To create a disk queue, use the "$QueueType Disk" config >>>>>> directive. Checkpoint intervals can be specified via >>>>>> "$QueueCheckpointInterval", with 0 meaning no checkpoints. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> you also need to specificly enable syncing (from >>>>>> http://www.rsyslog.com/doc-v3compatibility.html ) >>>>>> >>>>>> Output File Syncing >>>>>> Rsyslogd tries to keep as compatible to stock syslogd as possible. As >>>>>> such, it retained stock syslogd's default of syncing every file write if >>>>>> not specified otherwise (by placing a dash in front of the output file >>>>>> name). While this was a useful feature in past days where hardware was >>>>>> much less reliable and UPS seldom, this no longer is useful in today's >>>>>> worl. Instead, the syncing is a high performace hit. With it, rsyslogd >>>>>> writes files around 50 *times* slower than without it. It also affects >>>>>> overall system performance due to the high IO activity. In rsyslog v3, >>>>>> syncing has been turned off by default. This is done via a specific >>>>>> configuration directive "$ActionFileEnableSync on/off" which is off by >>>>>> default. So even if rsyslogd finds sync selector lines, it ignores them by >>>>>> default. In order to enable file syncing, the administrator must specify >>>>>> "$ActionFileEnableSync on" at the top of rsyslog.conf. This ensures that >>>>>> syncing only happens in some installations where the administrator >>>>>> actually wanted that (performance-intense) feature. In the fast majority >>>>>> of cases (if not all), this dramatically increases rsyslogd performance >>>>>> without any negative effects. >>>>>> >>>>>> >>>>>> >>>>>>> I already looked at queue.c and it seemed to me that both queues were >>>>>>> not designed for that kind of failure, but I could be wrong there. Since >>>>>>> an immediate power down of the system is the major failure which will >>>>>>> occure pretty often I need to create a soltution there. >>>>>> >>>>>> with checkpoint interval set to 1 and syncing enabled the data should be >>>>>> in on the disk safely (assuming you have hardware that supports this) and >>>>>> a power-off won't affect it. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> >>>>>>> Did you already start to develop something addressing that problem? >>>>>>> Could you help me extend rsyslog (3.18.4) so that I can develop a new >>>>>>> queue myself? I would contribute the code to the rsyslog project if you >>>>>>> would like afterwards. >>>>>>> >>>>>>> bye >>>>>>> David Ecker >>>>>>> >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at ecker-software.de Wed Oct 1 15:20:29 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:20:29 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E3791D.3050304@ecker-software.de> Hi, this is already the second version of this systems we develop. Not being able to do error analysis beacause of missing log data was one of the big problems including filesystem crashes. Having all logs and being able to proof the cause also helps a lot afterwards not only for creating a workaround for the incident but also to proof that the Service Level Agreement wasn't violated. bye David Ecker Rainer Gerhards schrieb: > Sorry, I overlooked this mail in the big bunch of messages. That's good > reasoning. > > To cover these scenarios, we need to do everything with syncing. This > also means that you can not use any of the disk-assisted modes, because > in these modes we always try to keep things in memory in order to save > writes. > > So while you have convinced me things can go wrong, I'd still say that > is is very unusual (at least very costly) to care for all these things. > But, of course, there are situations where it is needed. I'll probably > see that I provide a facility to open files in "always sync" mode, but > that for sure will not be the default setting ;) > > But even with the fast solid state disks (and similar methods) you > mention, I think there will be a severe impact on performance because > everything now needs to go through two write (data+metadata) and two > read (again, data+metadata) OS call where we currently simply update an > in-memory structure. > > Just out of curiosity: do you expect the majority of you rollouts to be > using such methods? > > Rainer > > On Wed, 2008-10-01 at 05:35 -0700, david at lang.hm wrote: > >>> ... And I have never heard of anybody doing serious datacenter work >>> without a proper UPS. Is this *really* an issue? >>> >> Yes. >> >> UPSs fail. >> generators fail >> power cords come loose. >> power cords get unplugged by someone who thinks they are unplugging a >> different system >> people bump power switches on power strips. >> power supplies are defective >> >> I had one production outage where a visiting tech pulled a power cord from >> an overhead plug and dropped it on the ground, where it happened to hit >> the power switch on a power strip. >> >> I've had high-end systems with redundant power supplies go down becouse of >> faulty hardware that decided to disble both power supplies at once (it >> turned out that there was a defect in the whole batch of servers, but it >> took IBM several weeks to figure out what was going on) >> >> I've had UPS systems blow up (literally) >> >> I've had a datacenter go down becouse the it was running on generator >> power (due to other issues), and the refueling guy filled the tank >> incorrectly and got air bubbles into the fuel system, a few min later the >> 500Kw diesel generator couldn't maintain constant speed and the safety >> triggers kicked in and disabled it. >> >> it's amazing the things that happen in real-life >> > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:24:33 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:24:33 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 14:45 +0200, David Ecker wrote: > > [snip] > >> as long as you do sector based writes (512 byte per sector, usual) you >> can be sure that the write wasn"t partial.. Writing more than one sector >> or not starting at a correct offset (n*512,n=0,1,2,...x) might result in >> a partial write. I'll already tested that with my devel client here. So >> fencing each sector with a crc32 value would help detecting errors >> during a write operation. This is actually only a problem if you are >> writing directly to a block device like any filesystem does and yes, >> reordering is definitly a problem. So validating the content written to >> the disk afterwards is important. >> >> If writing through a filesystem reserving space in the destination file >> beforehand actually minimizes errors since the file system table doesn't >> have to be updated (you should also use the Flag O_NOATIME for that >> case). See for example VMWare ESX VMDK file handling. > > Well, first of all let me re-iterate that I do not intend to do a block > device driver for rsyslog (but I definitely do not object getting one > contributed ;)). > > Still thinking about the case and thinking about non-solid-state, > non-internal-battery-backed-up disk, I can't see how you can be sure the > data will be written. David just told me there are no capacitors. So if > power fails, it fails rather quickly. So how can you be sure the disk > will be able to finish writing that sector? Let's say the drive has > begun to write the sector and been able to write the first 5 bytes. Now > power fails. No capacitors, no battery-backup, so why should there be > enough power to drive the disk write head for another 507 bytes? It the > drives assures it can do that, it needs capacitors - doesn't it? > > Am I overlooking something obvious? one possible thing is tht if the write has not completed then the system sending you the logs has not received confirmation that you have the log yet, so they are the ones responsible for it. it's only after you acknowledge the message (via relp or equivalent) that you are required to not loose the log message. with ext3 and data=journaled you can do this. the writes to the journal are done by the filesystem in such a way that they can be considered atomic (either they happen and are reliable, or they can be treated as if they never happened), once that write (or rather the sync on that write) has completed the data is safe. the filesystem will later modify the actual sector on disk, but it jumps through hoops to make that safe from power outages. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:24:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:24:24 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. mhhh... This sounds like you still may be in trouble if the file system crashes (as the queue files are stored inside that system). Rainer From david at lang.hm Wed Oct 1 15:31:15 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:31:15 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). pick the right filesystem and you should still be safe ext3+data=journaled is safe databases do the journaling themselves and like to store their journal files on ext2. you can use a log-structured filesystem that never overwrites data. it writes the new sector to a new place on disk then modifies pointers to tell the filesystem that the data is in the new place instead of the old place (the results in horrible fragmentation in many cases, but it's _very_ safe) some of this can be done without doing a sync for every write with barriers, which can guarentee that a write before the barrier takes place before any writes after the barrier. this lets you do things like modify the queue and then (after you know the data is safe) modify the metadata to indicate that the new data is there. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:31:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:31:19 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > one possible thing is tht if the write has not completed then the system > sending you the logs has not received confirmation that you have the log > yet, so they are the ones responsible for it. I concur. But as I understood the scenario here, the log messages are emited from a process *inside* the failing machine. So that process fails, too, and we do not have any interim that has a copy of the data. So if it is not persisted to disk, it is lost. Anyhow, this requirement has been relaxed in later posting. Also note that I was just thinking about the physical layer, considering a single physical write - far away from rsyslog. > it's only after you acknowledge the message (via relp or equivalent) that > you are required to not loose the log message. I concur and this is how RELP handles this. Well, actually I think there currently is a very slight (but still existing) window of exposure. I think RELP acks when the message is submited to the queue engine. That does not necessarily mean the message is already present on disk. Also, I think some mild duplication of messages may happen with RELP in a power fail scenario. It is not doing a two-phase commit, thought it tries very hard to get a perfect understanding of what is written and what not. I could check this with spec/code, but I think this is not justified at this point in time ;) > > with ext3 and data=journaled you can do this. the writes to the journal > are done by the filesystem in such a way that they can be considered > atomic (either they happen and are reliable, or they can be treated as if > they never happened), once that write (or rather the sync on that write) > has completed the data is safe. the filesystem will later modify the > actual sector on disk, but it jumps through hoops to make that safe from > power outages. > sure, definitely. And I assume a sync'ed write will return only after all this has happened. So journaled ext3 should be able to solve the problem described here. Rainer From david at lang.hm Wed Oct 1 15:35:01 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:35:01 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > >> Even then, in the worst case, I think it would be possible that the disk >> does only a partial write. I am not sure if that's really the case with >> today's disk drives (which I think have capacitors to prevent this >> scenario), but with past drives this could happen (I know all too well - >> a few years ago that cost me a weekend ;)). > > current disks do not have capacitors to prevent partial writes or to flush > their caches. but options like the linux ext3 data-journaled make it so > that you have your data in the journal safely, and the various solid-state > options solve that problem. actually, I need to correct my answer. I know that disks do not have capacitors large enough to write their buffer. I'm pretty sure that they don't have capacitors large enough to write an entire track. but they may have capacitors large enough to finish the sector they are on before stopping (considering the number of sectors in a track nowdays this is a _very_ sort time) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:34:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:34:40 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: > some of this can be done without doing a sync for every write with > barriers, which can guarentee that a write before the barrier takes place > before any writes after the barrier. this lets you do things like modify > the queue and then (after you know the data is safe) modify the metadata > to indicate that the new data is there. That sounds interesting - can you point me to some API documentation? A good place to do that would be inside the queue after the qi file has been persisted. That could reduce performance toll of synchronous writes. Especially if loss of a few messages is acceptable (as David Ecker just said). Rainer From david at ecker-software.de Wed Oct 1 15:35:18 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 15:35:18 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E37C96.3080904@ecker-software.de> I know. I used reiserfs before. The client systems were turned off like 4 times a day (average). At that point we had only 60 clients running. At least once a day one client needed to be reimaged since the filesystem was broke. The system itself wrote too much to the local disk during operation. So the failure rate was actually below 0,5% but it keept a constand buzz for the support center. I just was lucky that I implemented a failback image so that the systems could reimage themselfs without anybody going there. In this since the I would do it like this: Since the filesystem would only store the queue files of the syslog and nothing else I would check the filesystem at startup. If it's broken I would reformat the fs automtically. So that would leave me with a loss of 0,5% of all logs. Thats a lot better than nothing. I just have to make sure, that a rollback of the fs leaves the syslog data queue intact. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 15:20 +0200, David Ecker wrote: > >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. >> > > mhhh... This sounds like you still may be in trouble if the file system > crashes (as the queue files are stored inside that system). > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 15:40:30 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:40:30 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > >> one possible thing is tht if the write has not completed then the system >> sending you the logs has not received confirmation that you have the log >> yet, so they are the ones responsible for it. > > I concur. But as I understood the scenario here, the log messages are > emited from a process *inside* the failing machine. So that process > fails, too, and we do not have any interim that has a copy of the data. > So if it is not persisted to disk, it is lost. Anyhow, this requirement > has been relaxed in later posting. > > Also note that I was just thinking about the physical layer, considering > a single physical write - far away from rsyslog. > >> it's only after you acknowledge the message (via relp or equivalent) that >> you are required to not loose the log message. > > I concur and this is how RELP handles this. Well, actually I think there > currently is a very slight (but still existing) window of exposure. I > think RELP acks when the message is submited to the queue engine. That > does not necessarily mean the message is already present on disk. right now it doesn't make a difference (since the queue doesn't sync the data), but if/when this is added checking that the call to queue the data doesn't return until after that point should be done. > Also, > I think some mild duplication of messages may happen with RELP in a > power fail scenario. It is not doing a two-phase commit, thought it > tries very hard to get a perfect understanding of what is written and > what not. I could check this with spec/code, but I think this is not > justified at this point in time ;) no matter what checks you do, it's always possible for things to fail after succeeding on doing the work and before the acknowlegement gets back to the sender (with relp, if the receiving machine sends the acknowledgement, but the sending machine crashes before fully processing it, the receiving machine has no way of detecting this) on the other hand, this is fairly easy to deal with by making messages include a sequence number to guarentee that they are all unique, and then have something filter out duplicates later. David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:39:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:39:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868396.2682.511.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:35 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, david at lang.hm wrote: > > > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > >> Even then, in the worst case, I think it would be possible that the disk > >> does only a partial write. I am not sure if that's really the case with > >> today's disk drives (which I think have capacitors to prevent this > >> scenario), but with past drives this could happen (I know all too well - > >> a few years ago that cost me a weekend ;)). > > > > current disks do not have capacitors to prevent partial writes or to flush > > their caches. but options like the linux ext3 data-journaled make it so > > that you have your data in the journal safely, and the various solid-state > > options solve that problem. > > actually, I need to correct my answer. > > I know that disks do not have capacitors large enough to write their > buffer. absolutely > > I'm pretty sure that they don't have capacitors large enough to write an > entire track. > but they may have capacitors large enough to finish the sector they are on > before stopping (considering the number of sectors in a track nowdays this > is a _very_ sort time) *That* I had expected. Especially if you think about the low-level sync marks which otherwise could be affected. I think loss of disk could otherwise be the extreme result of a power failure (or at least loss of disk track). But you never know what a failing write arm does... So I would think (but have no evidence) that current disk drives have capacitors large enough to finish the write AND shut down the write mechanism in an orderly manner. As you say, there are "not many electrons" required to ensure this. Rainer From david at lang.hm Wed Oct 1 15:42:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 06:42:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <1222868080.2682.505.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > On Wed, 2008-10-01 at 06:31 -0700, david at lang.hm wrote: >> some of this can be done without doing a sync for every write with >> barriers, which can guarentee that a write before the barrier takes place >> before any writes after the barrier. this lets you do things like modify >> the queue and then (after you know the data is safe) modify the metadata >> to indicate that the new data is there. > > That sounds interesting - can you point me to some API documentation? A > good place to do that would be inside the queue after the qi file has > been persisted. That could reduce performance toll of synchronous > writes. Especially if loss of a few messages is acceptable (as David > Ecker just said). I'll try to find it (I've seen it discussed on the linux-kernel mailing list by the filesystem developers, but I don't know how it's exposed to userspace) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 15:46:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:46:52 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <48E370D2.7080901@ecker-software.de> <1222865740.2682.477.camel@rgf9dev.intern.adiscon.com> <1222867879.2682.502.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222868812.2682.515.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 06:40 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > On Wed, 2008-10-01 at 06:24 -0700, david at lang.hm wrote: > > > >> one possible thing is tht if the write has not completed then the system > >> sending you the logs has not received confirmation that you have the log > >> yet, so they are the ones responsible for it. > > > > I concur. But as I understood the scenario here, the log messages are > > emited from a process *inside* the failing machine. So that process > > fails, too, and we do not have any interim that has a copy of the data. > > So if it is not persisted to disk, it is lost. Anyhow, this requirement > > has been relaxed in later posting. > > > > Also note that I was just thinking about the physical layer, considering > > a single physical write - far away from rsyslog. > > > >> it's only after you acknowledge the message (via relp or equivalent) that > >> you are required to not loose the log message. > > > > I concur and this is how RELP handles this. Well, actually I think there > > currently is a very slight (but still existing) window of exposure. I > > think RELP acks when the message is submited to the queue engine. That > > does not necessarily mean the message is already present on disk. > > right now it doesn't make a difference (since the queue doesn't sync the > data), but if/when this is added checking that the call to queue the data > doesn't return until after that point should be done. I just checked the code. Actually, librelp already ensures this. It acks only after the callback has successfully completed. > > > Also, > > I think some mild duplication of messages may happen with RELP in a > > power fail scenario. It is not doing a two-phase commit, thought it > > tries very hard to get a perfect understanding of what is written and > > what not. I could check this with spec/code, but I think this is not > > justified at this point in time ;) > > no matter what checks you do, it's always possible for things to fail > after succeeding on doing the work and before the acknowlegement gets back > to the sender (with relp, if the receiving machine sends the > acknowledgement, but the sending machine crashes before fully processing > it, the receiving machine has no way of detecting this) RELP has a "Window of uncertainty", in which peers negotiate where to restart. But power failure was not on my spec list, so this only works if the shutdown was relatively clean. > > on the other hand, this is fairly easy to deal with by making messages > include a sequence number to guarentee that they are all unique, and then > have something filter out duplicates later. well said - among others one reason why RELP currently is not receiving more attention that just letting it mature... Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 15:59:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 15:59:34 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E37C96.3080904@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> Message-ID: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> David (Ecker ;)), I'd appreciate if you let me know your results after you have tested with $..QueueCheckpointInterval 1 Rainer From david at lang.hm Wed Oct 1 16:10:38 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 07:10:38 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David (Ecker ;)), > > I'd appreciate if you let me know your results after you have tested > with $..QueueCheckpointInterval 1 also, try mounting the filesystem -o sync this should make everything on it a sync write without Rainer having to change his code (won't work for other situations, but will definantly work for a test) David Lang From david at ecker-software.de Wed Oct 1 16:19:45 2008 From: david at ecker-software.de (David Ecker) Date: Wed, 01 Oct 2008 16:19:45 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <48E38701.80707@ecker-software.de> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My guess is, that the O_DIRECT in combination with the O_SYNC flag (turning of cache) will have an impact. I won't be able to test it tomorrow and since Friday is a vaction in germany I'll be able to tell you on monday the results of my tests. Thanks for your help. I was actually positivly suprised by the constructiveness of our discussion. David Ecker david at lang.hm schrieb: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > >> David (Ecker ;)), >> >> I'd appreciate if you let me know your results after you have tested >> with $..QueueCheckpointInterval 1 >> > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 1 16:32:28 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:32:28 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > going back to the higher layer: do you say that immediate power failure > is a case that you consider needed to be addressed in an enterprise > logging system? > > Anybody else with an opinion? Yes, I have an opinion: no. Enterprise-grade applications need to make the best effort they can to be fault-tolerant and reliable, but we're following an exponential curve - at some point, you _have_ to give over to the hardware and OS engineers to do their job properly. Don't reinvent the hammer & chisel to reinvent the wheel - software can _never_ account for all hardware failures and should never be expected to do so. RB From rgerhards at hq.adiscon.com Wed Oct 1 16:39:36 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 16:39:36 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> Message-ID: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Hi RB, I guess you've seen in the discussion (well, if you read all these mails, I just noticed how many they were...) I am coming from the same camp. But after all the dust settles, it looks like a very minor change to rsyslog to allow the OS to do what it is expected to do. The queue logic already cares about proper sequence (at least it should), the only thing that is missing is to tell the OS to sync when we need it. The spot to do so is also well defined, that is when the qi file is persisted (because that completes the "queue transaction"). I will verify with the code, but my current guess is that no more than 10 lines of code will be needed to support this functionality. If so, I think it is worth it. More complex questions may arise (I have some on my mind) and that will probably go beyond a general-purpose solution. Rainer On Wed, 2008-10-01 at 08:32 -0600, RB wrote: > On Wed, Oct 1, 2008 at 06:57, Rainer Gerhards wrote: > > going back to the higher layer: do you say that immediate power failure > > is a case that you consider needed to be addressed in an enterprise > > logging system? > > > > Anybody else with an opinion? > > Yes, I have an opinion: no. Enterprise-grade applications need to > make the best effort they can to be fault-tolerant and reliable, but > we're following an exponential curve - at some point, you _have_ to > give over to the hardware and OS engineers to do their job properly. > Don't reinvent the hammer & chisel to reinvent the wheel - software > can _never_ account for all hardware failures and should never be > expected to do so. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Wed Oct 1 16:40:21 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:40:21 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E3791D.3050304@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> Message-ID: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> > this is already the second version of this systems we develop. Not being > able to do error analysis beacause of missing log data was one of the > big problems including filesystem crashes. Having all logs and being > able to proof the cause also helps a lot afterwards not only for > creating a workaround for the incident but also to proof that the > Service Level Agreement wasn't violated. Somewhat randomly selecting one to respond to - you guys generated a lot of chatter. My view as both a software and hardware engineer on this is simple: never try to solve a hardware problem with a software solution. You will never win. It is possible to account for many edge cases, but no matter how paranoid you may be, hardware always trumps software - it has the last word, regardless of what you may try to do. RB From aoz.syn at gmail.com Wed Oct 1 16:50:24 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 08:50:24 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> Message-ID: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> > I guess you've seen in the discussion (well, if you read all these > mails, I just noticed how many they were...) ;-) I was wondering why my hip was rattling during the entire commute today. > persisted (because that completes the "queue transaction"). I will > verify with the code, but my current guess is that no more than 10 lines > of code will be needed to support this functionality. If so, I think it > is worth it. If it's something you can solve without worrying about block-level writes and whether the underlying drive (if indeed there is even a "drive") has a battery-backed cache or sufficient capacitor charge to write your data, I'm all for it. Even better if it's POSIX. Although a block driver and other filesystem-bypassing solutions may be interesting in limited cases, I'd rather not see anyone stab their wife over it. From rgerhards at hq.adiscon.com Wed Oct 1 17:18:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:18:11 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> Message-ID: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> David, On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, Rainer Gerhards wrote: > > > David (Ecker ;)), > > > > I'd appreciate if you let me know your results after you have tested > > with $..QueueCheckpointInterval 1 > > also, try mounting the filesystem -o sync > > this should make everything on it a sync write without Rainer having to > change his code (won't work for other situations, but will definantly work > for a test) I after all this discussion I will take a few minutes to see if I can capture results in code. There are two things and I guess you have an opinion on that. First of all, the doc I have on fsync() is a bit cryptic. It says: ---- fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)). ---- so it looks like fsync() is sufficient to flush both the data as well as the size information (so that data is not only present but can actually be read ;). But in the next paragraph, the man page says: ---- Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed. ---- which sound much like the opposite. I know there are many flavours of *nix and even though this is a POSIX call, things may be different in different OS... Do you have an opinion of we can assume a call to fsync() on the descriptor in questions solves the issue? This would probably avoid the need to open with always synced writes and thus could offer a somewhat better performance. Also, the thought about O_NOATIME looks interesting. For internal queue files, I think the last access time is of very limited interest, so it would probably be useful to always specify this option (at least by default). Feedback appreciated. Rainer From david at lang.hm Wed Oct 1 17:41:11 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:41:11 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <1222874291.2682.532.camel@rgf9dev.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: > David, > > On Wed, 2008-10-01 at 07:10 -0700, david at lang.hm wrote: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) > > I after all this discussion I will take a few minutes to see if I can > capture results in code. There are two things and I guess you have an > opinion on that. > > First of all, the doc I have on fsync() is a bit cryptic. It says: > > ---- > fsync() transfers ("flushes") all modified in-core data of (i.e., > modified buffer cache pages for) the file referred to by the file > descriptor fd to the disk device (or other permanent storage device) > where that file resides. The call blocks until the device reports that > the transfer has completed. It also flushes metadata information > associated with the file (see stat(2)). > ---- > > so it looks like fsync() is sufficient to flush both the data as well as > the size information (so that data is not only present but can actually > be read ;). But in the next paragraph, the man page says: > > ---- > Calling fsync() does not necessarily ensure that the entry in the > directory containing the file has also reached disk. For that an > explicit fsync() on a file descriptor for the directory is also needed. > ---- > > which sound much like the opposite. I know there are many flavours of > *nix and even though this is a POSIX call, things may be different in > different OS... > > Do you have an opinion of we can assume a call to fsync() on the > descriptor in questions solves the issue? This would probably avoid the > need to open with always synced writes and thus could offer a somewhat > better performance. my understanding is that you may need to call both. if you have pre-allocated the file you would not need to call fsync on the directory, if you are dynamicly extending it you will need to. > Also, the thought about O_NOATIME looks interesting. For internal queue > files, I think the last access time is of very limited interest, so it > would probably be useful to always specify this option (at least by > default). in this case noatime won't make much difference. normally you want it so that when you read a file it doesn't have to update the 'last accessed' time in the directory, but since we are writing to these just about every time we access them we are having to update the 'last modified' time there anyway. I am assuming that even though the data is being written to the disk queue file, it's still available in memory so you aren't having to do a disk read to read the message from the queue. If this is not the case then noatime could make a difference if the file is opened O_SYNC becouse the read would then require the directory update to be synced before you could do anything else. If you are dong the fsync calls yourself it shouldn't matter much becouse the atime update will (probably) not get pushed out to disk before you are going back and modifying the queue to indicate that you wrote the message to the output (when the 'last modified' times will get updated) David Lang > Feedback appreciated. > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Oct 1 17:43:52 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:43:52 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48E38701.80707@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: On Wed, 1 Oct 2008, David Ecker wrote: > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > of cache) will have an impact. O_DIRECT is doing very different things. I don't think you need to worry about those things, having the data not go into the OS cache is a drawback not an advantage becouse it means that when you go to pull the data back out of the file it will need to actually touch disk. it also imposes significant alignment issues on the application that I don't think you want to have to desl with. David Lang > I won't be able to test it tomorrow and since Friday is a vaction in > germany I'll be able to tell you on monday the results of my tests. > > Thanks for your help. I was actually positivly suprised by the > constructiveness of our discussion. > David Ecker > > david at lang.hm schrieb: >> On Wed, 1 Oct 2008, Rainer Gerhards wrote: >> >> >>> David (Ecker ;)), >>> >>> I'd appreciate if you let me know your results after you have tested >>> with $..QueueCheckpointInterval 1 >>> >> >> also, try mounting the filesystem -o sync >> >> this should make everything on it a sync write without Rainer having to >> change his code (won't work for other situations, but will definantly work >> for a test) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > -------------- next part -------------- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 1 17:44:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:44:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> Message-ID: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, David Ecker wrote: > > > Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My > > guess is, that the O_DIRECT in combination with the O_SYNC flag (turning > > of cache) will have an impact. > > O_DIRECT is doing very different things. I don't think you need to worry > about those things, having the data not go into the OS cache is a drawback > not an advantage becouse it means that when you go to pull the data back > out of the file it will need to actually touch disk. it also imposes > significant alignment issues on the application that I don't think you > want to have to desl with. plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... Rainer From david at lang.hm Wed Oct 1 17:45:16 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:45:16 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> this is already the second version of this systems we develop. Not being >> able to do error analysis beacause of missing log data was one of the >> big problems including filesystem crashes. Having all logs and being >> able to proof the cause also helps a lot afterwards not only for >> creating a workaround for the incident but also to proof that the >> Service Level Agreement wasn't violated. > > Somewhat randomly selecting one to respond to - you guys generated a > lot of chatter. > > My view as both a software and hardware engineer on this is simple: > never try to solve a hardware problem with a software solution. You > will never win. It is possible to account for many edge cases, but no > matter how paranoid you may be, hardware always trumps software - it > has the last word, regardless of what you may try to do. I'm not understanding the point you are trying to make. are you saying that it's a bad idea to try and have an option to do the syncing we are talking about for the queue? David Lang From david at lang.hm Wed Oct 1 17:48:57 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 08:48:57 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I guess you've seen in the discussion (well, if you read all these >> mails, I just noticed how many they were...) > > ;-) I was wondering why my hip was rattling during the entire commute today. > >> persisted (because that completes the "queue transaction"). I will >> verify with the code, but my current guess is that no more than 10 lines >> of code will be needed to support this functionality. If so, I think it >> is worth it. > > If it's something you can solve without worrying about block-level > writes and whether the underlying drive (if indeed there is even a > "drive") has a battery-backed cache or sufficient capacitor charge to > write your data, I'm all for it. Even better if it's POSIX. > > Although a block driver and other filesystem-bypassing solutions may > be interesting in limited cases, I'd rather not see anyone stab their > wife over it. I agree that trying to bypass the filesystem is highly questionable, and not something for a core change (as always a contributed vrsion can be tested to see if it makes a difference) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 17:58:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 01 Oct 2008 17:58:21 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222862275.2682.453.camel@rgf9dev.intern.adiscon.com> <1222864143.2682.467.camel@rgf9dev.intern.adiscon.com> <1222865840.2682.479.camel@rgf9dev.intern.adiscon.com> <4255c2570810010732y44aa560cgfd91d5feecf92b3d@mail.gmail.com> <1222871976.2682.523.camel@rgf9dev.intern.adiscon.com> <4255c2570810010750p9c2dc60j3dd3c81b9fd2c6ce@mail.gmail.com> Message-ID: <1222876701.2682.541.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-01 at 08:48 -0700, david at lang.hm wrote: > On Wed, 1 Oct 2008, RB wrote: > > >> I guess you've seen in the discussion (well, if you read all these > >> mails, I just noticed how many they were...) > > > > ;-) I was wondering why my hip was rattling during the entire commute today. > > > >> persisted (because that completes the "queue transaction"). I will > >> verify with the code, but my current guess is that no more than 10 lines > >> of code will be needed to support this functionality. If so, I think it > >> is worth it. > > > > If it's something you can solve without worrying about block-level > > writes and whether the underlying drive (if indeed there is even a > > "drive") has a battery-backed cache or sufficient capacitor charge to > > write your data, I'm all for it. Even better if it's POSIX. > > > > Although a block driver and other filesystem-bypassing solutions may > > be interesting in limited cases, I'd rather not see anyone stab their > > wife over it. > > I agree that trying to bypass the filesystem is highly questionable, and > not something for a core change (as always a contributed vrsion can be > tested to see if it makes a difference) Just FYI: preliminary analysis indicates that is probably around the 10 lines of code that need to be added in stream.c, maybe a few more. The idea is that I can set a flag similar to O_SYNC on stream creation but then sync when the "atomic" writes are done. This may save some few cycles over an O_SYNC open(). However, there need to be a few config settings, which in turn need to be passed down to the queue and stream class. That adds more code, maybe around 100 lines (the config interface needs to be redone, thus the many LOC required, it's on the todo list...). An alternative is to use a simple global $AllWritesSync on/off option, which would probably be sufficient and cut down changes required. For a robust implementation, some more analysis is required (including thinking about the implications of fsync()). So it is not totally trivial, but well doable. I am just not sure if I'll do it immediately, there are many things in the work queue. Testing effort is probably much bigger than implementation effort, there are so many cases to check out... Rainer From aoz.syn at gmail.com Wed Oct 1 18:00:10 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 1 Oct 2008 10:00:10 -0600 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> Message-ID: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> > I'm not understanding the point you are trying to make. > > are you saying that it's a bad idea to try and have an option to do the > syncing we are talking about for the queue? Sync: not as long as it's optional. Worrying about sector writes and capacitors/battery-backed cache on the underlying drives: yes. Unless rsyslog starts writing to raw devices, it is my opinion that once it reasonably hands the data off to the filesystem, it becomes a kernel and/or hardware problem and unnecessarily complex for a userspace application to govern. I'm all for anything rsyslog can do to encourage proper behavior (like calling sync()) without getting into the kernel/hardware space. RB From david at lang.hm Wed Oct 1 18:07:47 2008 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Oct 2008 09:07:47 -0700 (PDT) Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <4255c2570810010740i79e79df4i42c27455179c494@mail.gmail.com> <4255c2570810010900i7a209803gdbebb8f987183e62@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, RB wrote: >> I'm not understanding the point you are trying to make. >> >> are you saying that it's a bad idea to try and have an option to do the >> syncing we are talking about for the queue? > > Sync: not as long as it's optional. Worrying about sector writes and > capacitors/battery-backed cache on the underlying drives: yes. Unless > rsyslog starts writing to raw devices, it is my opinion that once it > reasonably hands the data off to the filesystem, it becomes a kernel > and/or hardware problem and unnecessarily complex for a userspace > application to govern. I'm all for anything rsyslog can do to > encourage proper behavior (like calling sync()) without getting into > the kernel/hardware space. Ok, we are on the same page then. in case I confused anyone, the reason I went into detail on the hardware side of things was to explain how proper hardware selection could result in good performance while poor hardware selection would result is dismal performance (and to give people who aren't familiar with the options some hints as to what they could do) David Lang From rgerhards at hq.adiscon.com Wed Oct 1 18:51:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 18:51:52 +0200 Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> [going back to the list with Lorenzo's permission] Hi Lorenzo, finally... I did not realize that the queue is running in direct mode, but now I saw it and can reproduce a failure. That should make things much better. Anyhow, it is very unusual that the main message queue runs in direct mode, so I guess it is a side-effect of a config file. In those files you sent me, I don't see it, but I see a .d directory is included. May there be something? In any case, I hope I will be able to fix the bug now that I can reproduce it. Looks like a productive day today :) Thanks for your persistence, Rainer > -----Original Message----- > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > Sent: Wednesday, October 01, 2008 1:46 PM > To: Rainer Gerhards > Subject: Re: [rsyslog] abort case > > Sorry for the delay, I've just run helgrind's head, bau as you can see, > it > crashed almost as soon as I started. > > The log is enclosed; I'm now checking-out head and will retry. > > Yours, > > lorenzo > > RG> I have created a new version with one slight change, to be found in > RG> the helgrind branch. > RG> > RG> There is also a new valgrind tool called drd inside the valgrind > RG> development tree. I think you already downloaded that tree. If so, > RG> could you please replace > RG> > RG> Valgrind --tool=helgrind .. rsyslogd ... > RG> > RG> with > RG> > RG> Valgrind --tool=drd .. rsyslogd ... > RG> > RG> Drd does an even better job than helgrind. I also changed the > source > RG> to remove some debug-system related warning, which otherwise would > RG> clutter up the error message. But unfortunately, in my lab I did > not > RG> find any more problems, except for the small change I mentioned. > But > RG> that one affects program termination (and in a very subtle way), so > it > RG> should not change anything for you. I'd still be interested in a > new > RG> run, including debug info, from you. > RG> > > > +-------------------------+-------------------------------------------- > --+ > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > | > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > Vergata" | > | | Via O. Raimondo 18 ** I-00173 ROMA ** > ITALY | > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > | > +-------------------------+-------------------------------------------- > --+ From rgerhards at hq.adiscon.com Wed Oct 1 19:02:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 1 Oct 2008 19:02:15 +0200 Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 01, 2008 6:52 PM > To: Lorenzo M. Catucci > Cc: rsyslog-users > Subject: Re: [rsyslog] abort case > > [going back to the list with Lorenzo's permission] > > Hi Lorenzo, > > finally... I did not realize that the queue is running in direct mode, > but now I saw it and can reproduce a failure. That should make things > much better. Anyhow, it is very unusual that the main message queue > runs in direct mode, so I guess it is a side-effect of a config file. > In those files you sent me, I don't see it, but I see a .d directory is > included. May there be something? In any case, I hope I will be able to > fix the bug now that I can reproduce it. > > Looks like a productive day today :) > > Thanks for your persistence, > Rainer > > > -----Original Message----- > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] > > Sent: Wednesday, October 01, 2008 1:46 PM > > To: Rainer Gerhards > > Subject: Re: [rsyslog] abort case > > > > Sorry for the delay, I've just run helgrind's head, bau as you can > see, > > it > > crashed almost as soon as I started. > > > > The log is enclosed; I'm now checking-out head and will retry. > > > > Yours, > > > > lorenzo > > > > RG> I have created a new version with one slight change, to be found > in > > RG> the helgrind branch. > > RG> > > RG> There is also a new valgrind tool called drd inside the valgrind > > RG> development tree. I think you already downloaded that tree. If > so, > > RG> could you please replace > > RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... > > RG> > > RG> with > > RG> > > RG> Valgrind --tool=drd .. rsyslogd ... > > RG> > > RG> Drd does an even better job than helgrind. I also changed the > > source > > RG> to remove some debug-system related warning, which otherwise > would > > RG> clutter up the error message. But unfortunately, in my lab I did > > not > > RG> find any more problems, except for the small change I mentioned. > > But > > RG> that one affects program termination (and in a very subtle way), > so > > it > > RG> should not change anything for you. I'd still be interested in a > > new > > RG> run, including debug info, from you. > > RG> > > > > > > +-------------------------+------------------------------------------ > -- > > --+ > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione > > | > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor > > Vergata" | > > | | Via O. Raimondo 18 ** I-00173 ROMA ** > > ITALY | > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 > > | > > +-------------------------+------------------------------------------ > -- > > --+ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:05:31 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:05:31 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> Message-ID: On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> [going back to the list with Lorenzo's permission] RG> [...] RG> RG> finally... I did not realize that the queue is running in direct mode, RG> but now I saw it and can reproduce a failure. That should make things RG> much better. Anyhow, it is very unusual that the main message queue RG> runs in direct mode, so I guess it is a side-effect of a config file. RG> No, they are really the only config files present on the server (rsyslog.conf is in /etc, the other one is the only thing inside /etc/rsyslog.d/) Really, I wouldn't go back to a saner configuration until we zeroed on the bug... still, nowhere I touched the MainQueue options! Hope to hear from you soon! Yours, lorenzo RG> RG> In those files you sent me, I don't see it, but I see a .d directory RG> is included. May there be something? In any case, I hope I will be RG> able to fix the bug now that I can reproduce it. RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:16:23 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:16:23 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: It just crashed under drd (sending both direct and through the list, it should get to you!). I've restarted under helgrind, just for the sake of not wasting the upcoming night... Hear you tomorrow! Yours, lorenzo On Wed, 1 Oct 2008, Rainer Gerhards wrote: RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) RG> RG> Rainer RG> RG> > -----Original Message----- RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards RG> > Sent: Wednesday, October 01, 2008 6:52 PM RG> > To: Lorenzo M. Catucci RG> > Cc: rsyslog-users RG> > Subject: Re: [rsyslog] abort case RG> > RG> > [going back to the list with Lorenzo's permission] RG> > RG> > Hi Lorenzo, RG> > RG> > finally... I did not realize that the queue is running in direct mode, RG> > but now I saw it and can reproduce a failure. That should make things RG> > much better. Anyhow, it is very unusual that the main message queue RG> > runs in direct mode, so I guess it is a side-effect of a config file. RG> > In those files you sent me, I don't see it, but I see a .d directory is RG> > included. May there be something? In any case, I hope I will be able to RG> > fix the bug now that I can reproduce it. RG> > RG> > Looks like a productive day today :) RG> > RG> > Thanks for your persistence, RG> > Rainer RG> > RG> > > -----Original Message----- RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] RG> > > Sent: Wednesday, October 01, 2008 1:46 PM RG> > > To: Rainer Gerhards RG> > > Subject: Re: [rsyslog] abort case RG> > > RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can RG> > see, RG> > > it RG> > > crashed almost as soon as I started. RG> > > RG> > > The log is enclosed; I'm now checking-out head and will retry. RG> > > RG> > > Yours, RG> > > RG> > > lorenzo RG> > > RG> > > RG> I have created a new version with one slight change, to be found RG> > in RG> > > RG> the helgrind branch. RG> > > RG> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind RG> > > RG> development tree. I think you already downloaded that tree. If RG> > so, RG> > > RG> could you please replace RG> > > RG> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... RG> > > RG> RG> > > RG> with RG> > > RG> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... RG> > > RG> RG> > > RG> Drd does an even better job than helgrind. I also changed the RG> > > source RG> > > RG> to remove some debug-system related warning, which otherwise RG> > would RG> > > RG> clutter up the error message. But unfortunately, in my lab I did RG> > > not RG> > > RG> find any more problems, except for the small change I mentioned. RG> > > But RG> > > RG> that one affects program termination (and in a very subtle way), RG> > so RG> > > it RG> > > RG> should not change anything for you. I'd still be interested in a RG> > > new RG> > > RG> run, including debug info, from you. RG> > > RG> RG> > > RG> > > RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione RG> > > | RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor RG> > > Vergata" | RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** RG> > > ITALY | RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 RG> > > | RG> > > +-------------------------+------------------------------------------ RG> > -- RG> > > --+ RG> > _______________________________________________ RG> > rsyslog mailing list RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog RG> > http://www.rsyslog.com RG> _______________________________________________ RG> rsyslog mailing list RG> http://lists.adiscon.net/mailman/listinfo/rsyslog RG> http://www.rsyslog.com RG> +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From lorenzo at sancho.ccd.uniroma2.it Wed Oct 1 20:20:48 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Wed, 1 Oct 2008 20:20:48 +0200 (CEST) Subject: [rsyslog] abort case In-Reply-To: References: <59e975c10809221126o37d8919w674f13aee5acc16f@mail.gmail.com><1222173877.2682.188.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F21A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F268@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F26A@grfint2.intern.adiscon.com> Message-ID: This time I attach the file! Sorry! On Wed, 1 Oct 2008, Lorenzo M. Catucci wrote: LMC> It just crashed under drd (sending both direct and through the list, it LMC> should get to you!). I've restarted under helgrind, just for the sake of LMC> not wasting the upcoming night... LMC> LMC> Hear you tomorrow! Yours, LMC> LMC> lorenzo LMC> LMC> On Wed, 1 Oct 2008, Rainer Gerhards wrote: LMC> LMC> RG> As I thought... stupid error. Please pull helgrind branch again and give it another try ;) LMC> RG> LMC> RG> Rainer LMC> RG> LMC> RG> > -----Original Message----- LMC> RG> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- LMC> RG> > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards LMC> RG> > Sent: Wednesday, October 01, 2008 6:52 PM LMC> RG> > To: Lorenzo M. Catucci LMC> RG> > Cc: rsyslog-users LMC> RG> > Subject: Re: [rsyslog] abort case LMC> RG> > LMC> RG> > [going back to the list with Lorenzo's permission] LMC> RG> > LMC> RG> > Hi Lorenzo, LMC> RG> > LMC> RG> > finally... I did not realize that the queue is running in direct mode, LMC> RG> > but now I saw it and can reproduce a failure. That should make things LMC> RG> > much better. Anyhow, it is very unusual that the main message queue LMC> RG> > runs in direct mode, so I guess it is a side-effect of a config file. LMC> RG> > In those files you sent me, I don't see it, but I see a .d directory is LMC> RG> > included. May there be something? In any case, I hope I will be able to LMC> RG> > fix the bug now that I can reproduce it. LMC> RG> > LMC> RG> > Looks like a productive day today :) LMC> RG> > LMC> RG> > Thanks for your persistence, LMC> RG> > Rainer LMC> RG> > LMC> RG> > > -----Original Message----- LMC> RG> > > From: Lorenzo M. Catucci [mailto:lorenzo at sancho.ccd.uniroma2.it] LMC> RG> > > Sent: Wednesday, October 01, 2008 1:46 PM LMC> RG> > > To: Rainer Gerhards LMC> RG> > > Subject: Re: [rsyslog] abort case LMC> RG> > > LMC> RG> > > Sorry for the delay, I've just run helgrind's head, bau as you can LMC> RG> > see, LMC> RG> > > it LMC> RG> > > crashed almost as soon as I started. LMC> RG> > > LMC> RG> > > The log is enclosed; I'm now checking-out head and will retry. LMC> RG> > > LMC> RG> > > Yours, LMC> RG> > > LMC> RG> > > lorenzo LMC> RG> > > LMC> RG> > > RG> I have created a new version with one slight change, to be found LMC> RG> > in LMC> RG> > > RG> the helgrind branch. LMC> RG> > > RG> LMC> RG> > > RG> There is also a new valgrind tool called drd inside the valgrind LMC> RG> > > RG> development tree. I think you already downloaded that tree. If LMC> RG> > so, LMC> RG> > > RG> could you please replace LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=helgrind .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> with LMC> RG> > > RG> LMC> RG> > > RG> Valgrind --tool=drd .. rsyslogd ... LMC> RG> > > RG> LMC> RG> > > RG> Drd does an even better job than helgrind. I also changed the LMC> RG> > > source LMC> RG> > > RG> to remove some debug-system related warning, which otherwise LMC> RG> > would LMC> RG> > > RG> clutter up the error message. But unfortunately, in my lab I did LMC> RG> > > not LMC> RG> > > RG> find any more problems, except for the small change I mentioned. LMC> RG> > > But LMC> RG> > > RG> that one affects program termination (and in a very subtle way), LMC> RG> > so LMC> RG> > > it LMC> RG> > > RG> should not change anything for you. I'd still be interested in a LMC> RG> > > new LMC> RG> > > RG> run, including debug info, from you. LMC> RG> > > RG> LMC> RG> > > LMC> RG> > > LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > > | Lorenzo M. Catucci | Centro di Calcolo e Documentazione LMC> RG> > > | LMC> RG> > > | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor LMC> RG> > > Vergata" | LMC> RG> > > | | Via O. Raimondo 18 ** I-00173 ROMA ** LMC> RG> > > ITALY | LMC> RG> > > | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 LMC> RG> > > | LMC> RG> > > +-------------------------+------------------------------------------ LMC> RG> > -- LMC> RG> > > --+ LMC> RG> > _______________________________________________ LMC> RG> > rsyslog mailing list LMC> RG> > http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> > http://www.rsyslog.com LMC> RG> _______________________________________________ LMC> RG> rsyslog mailing list LMC> RG> http://lists.adiscon.net/mailman/listinfo/rsyslog LMC> RG> http://www.rsyslog.com LMC> RG> LMC> LMC> +-------------------------+----------------------------------------------+ LMC> | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | LMC> | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | LMC> | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | LMC> | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | LMC> +-------------------------+----------------------------------------------+ +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Thu Oct 2 15:14:34 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:14:34 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Thanks to the debug data provided, I could find out that the timestamps stem back to imfile and the way (sequence) in which it generates the timestamps. I will take the opportunity and fix it in a way that also improves performance. I just thought I let you know, more info follows when I have written the fix. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Wednesday, October 01, 2008 2:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > I installed the snapshot but apparently the issue persists. > > If there's something else that I can do, now or when you have more > time, let me know. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Oct 2 15:54:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Oct 2008 15:54:53 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25D@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> I (think) I have now fixed it, but not verified all cases. I would appreciate if you could give the new snapshot a try: http://git.adiscon.com/?p=rsyslog.git;a=snapshot;h=39000a62024510cd62607 200e6100e3cd7c05005;sf=tgz Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, October 02, 2008 3:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Thanks to the debug data provided, I could find out that the timestamps > stem back to imfile and the way (sequence) in which it generates the > timestamps. I will take the opportunity and fix it in a way that also > improves performance. I just thought I let you know, more info follows > when I have written the fix. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > > Sent: Wednesday, October 01, 2008 2:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > > > I installed the snapshot but apparently the issue persists. > > > > If there's something else that I can do, now or when you have more > > time, let me know. > > > > Regards, > > Luigi > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From martinmie at PartyGaming.com Thu Oct 2 17:29:30 2008 From: martinmie at PartyGaming.com (Martin Mielke) Date: Thu, 2 Oct 2008 17:29:30 +0200 Subject: [rsyslog] could not load TLS module even when it's there! Message-ID: <0B1B9163138571439EAF804646F3F6AA05917715@GIBSVWIN004X.partygaming.local> Hi list, I'm setting up rsyslog to use TLS. The server-side comes up fine and I hope it's expecting encrypted traffic... Anyway, when I restart rsyslog on a client which is supposed to forward its syslogs to the logserver I see the following error message: -- ... rsyslogd: # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so'/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078 : No such file or directory -- ...but funny enough: -- # ls -l /usr/lib/rsyslog/lmnsd_gtls.so -rwxr-xr-x 1 root root 82311 Sep 8 08:53 /usr/lib/rsyslog/lmnsd_gtls.so -- What am I doing wrong?? TIA, Martin From luigi.perroti at googlemail.com Fri Oct 3 17:00:57 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Fri, 3 Oct 2008 17:00:57 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com> <1222858002.2682.441.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: Everything's fine here. Now I get the same subsecond part for all the events. This is how it should be since they are being generated on the same host. If it's of relevance I'm using these modules: imfile, imuxsock, imklog, ommysql. I'm also listening on a couple of additional unix sockets. Everything seems to be working fine. Thanks for the quick turnaround. Regards, Luigi From rgerhards at hq.adiscon.com Fri Oct 3 17:47:41 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 3 Oct 2008 17:47:41 +0200 Subject: [rsyslog] Subseconds in timereported and timegenerated In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F25E@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F260@grfint2.intern.adiscon.com><1222858002.2682.441.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F281@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F282@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F28A@grfint2.intern.adiscon.com> Excellent, thanks for the confirmation. The modules are no longer relevant, they all get the time from a consistent call now (not doing this was part of the problem, if not the whole ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Friday, October 03, 2008 5:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Subseconds in timereported and timegenerated > > Everything's fine here. Now I get the same subsecond part for all the > events. > This is how it should be since they are being generated on the same > host. > > If it's of relevance I'm using these modules: > imfile, imuxsock, imklog, ommysql. > I'm also listening on a couple of additional unix sockets. > Everything seems to be working fine. > > Thanks for the quick turnaround. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Oct 6 11:24:05 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:24:05 +0200 Subject: [rsyslog] HUP action Message-ID: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Hi List, I again have a backwards compatibility question. HUP, in sysklogd, means a full restart of the syslogd. This is done because the configuration may have been changed and to apply the new one, the previous one must be shut down. This is even more so the case in rsyslog, which has multiple dynamically loadable plugins and all that... rsyslog has implemented HUP processing to be compatible to sysklogd. However, in practice (as far as I know) HUP is almost always being used to close the output files after log rotation. An automatted config reaload indication usually does not happen (but operators use HUP after doing a config change). HUP, as it currently is, is an extremely expensive operation. Keep in mind that it is a full daemon restart, so, among other things, it unloads modules and tears down (TCP, RELP, TLS, ...) connections. In most cases, though, this would not be needed. I now consider changing the way HUP works. I would like to model it in a way that closes files and clears caches (where useful), but not do a full restart. Initially, I thought to use a special signal (e.g. USR2) for this purpose. But still logrotate, as a well-known example, would cause a daemon restart where none is needed. So I now think it would be best to change HUP in the way described above. Those that need to apply a config change would then need to do a regular daemon restart - the same thing that is done with other daemons. In order to keep compatible, I would add an $HUPisRestart option, by default off, which could be used to re-enable previous behavior. I would appreciate feedback on this plan. Are there any concerns? Is this a good idea or a bad one? Does it break something I have not yet mentioned? Thanks, Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 11:42:36 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 11:42:36 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> RG> [...] RG> RG> Initially, I thought to use a special signal (e.g. USR2) for this RG> purpose. But still logrotate, as a well-known example, would cause a RG> daemon restart where none is needed. So I now think it would be best to RG> change HUP in the way described above. Those that need to apply a config RG> change would then need to do a regular daemon restart - the same thing RG> that is done with other daemons. In order to keep compatible, I would RG> add an $HUPisRestart option, by default off, which could be used to RG> re-enable previous behavior. RG> Still, I'd like to have a way to force full restart without really tearing down and reloading the server. What about (optionally?) binding current HUP behaviour to USR2 adding a default on option like $USR2isRestart? Cheers, and a nice forthweek to all, yours lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 11:44:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 11:44:25 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Thanks for the feedback, but please let me ask one question: On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: > Still, I'd like to have a way to force full restart without really tearing > down and reloading the server. What about (optionally?) binding current > HUP behaviour to USR2 adding a default on option like $USR2isRestart? Why is this useful? I am trying to understand the use cases and why a "normal" restart is not desirable. Rainer From lorenzo at sancho.ccd.uniroma2.it Mon Oct 6 13:33:05 2008 From: lorenzo at sancho.ccd.uniroma2.it (Lorenzo M. Catucci) Date: Mon, 6 Oct 2008 13:33:05 +0200 (CEST) Subject: [rsyslog] HUP action In-Reply-To: <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: RG> Thanks for the feedback, but please let me ask one question: RG> RG> On Mon, 2008-10-06 at 11:42 +0200, Lorenzo M. Catucci wrote: RG> RG> > Still, I'd like to have a way to force full restart without really tearing RG> > down and reloading the server. What about (optionally?) binding current RG> > HUP behaviour to USR2 adding a default on option like $USR2isRestart? RG> RG> Why is this useful? I am trying to understand the use cases and why a RG> "normal" restart is not desirable. RG> Habits, pid stability, monitoring systems, one less thing to break... etc. Yours, lorenzo +-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci at ccd.uniroma2.it | Universit? degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+ From rgerhards at hq.adiscon.com Mon Oct 6 13:56:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 13:56:50 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Lorenzo, > Habits, pid stability, monitoring systems, one less thing to break... I understand habits and pid stability. What do you mean by "monitoring system" and "one less thing to break"? Anyone else with an opinion? Rainer From david at ecker-software.de Mon Oct 6 14:41:25 2008 From: david at ecker-software.de (David Ecker) Date: Mon, 06 Oct 2008 14:41:25 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EA0775.7050105@ecker-software.de> Hi, 3.21.5 created an assertion error. rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void *)0)' failed. close to the end. Propably short before the abort signal. I mounted /rsyslog (ext3) with the option : sync Here is a copy of my rsyslog.conf: #--------------------------------------- $ModLoad imuxsock.so $ModLoad imklog.so $WorkDirectory /rsyslog $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 1m $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 #--------------------------------------- I attached the output from" rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 Actually only one messagefile was written, no .qi file was created. bye David Ecker Rainer Gerhards schrieb: > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > >> On Wed, 1 Oct 2008, David Ecker wrote: >> >> >>> Already did both with 2.18.3 but'll try again with 3.21.5 and 3.18.4. My >>> guess is, that the O_DIRECT in combination with the O_SYNC flag (turning >>> of cache) will have an impact. >>> >> O_DIRECT is doing very different things. I don't think you need to worry >> about those things, having the data not go into the OS cache is a drawback >> not an advantage becouse it means that when you go to pull the data back >> out of the file it will need to actually touch disk. it also imposes >> significant alignment issues on the application that I don't think you >> want to have to desl with. >> > > plus rsyslog does not care about the alignment (at this time), so I think it is dangerous... > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: error.txt URL: From rgerhards at hq.adiscon.com Mon Oct 6 14:43:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 14:43:12 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EA0775.7050105@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Please use the version from git. I didn't realize that the bug affects normal operations, but obviously it does. This is fixed and I'll see that I release 3.21.6 ASAP, but I am not sure if I manage to do this today. Gitweb available at http://git.adiscon.com Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Monday, October 06, 2008 2:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi, > > 3.21.5 created an assertion error. > > rsyslogd: msg.c:208: MsgUnlockLockingCase: Assertion `pThis != ((void > *)0)' failed. > > close to the end. Propably short before the abort signal. > > I mounted /rsyslog (ext3) with the option : sync > > Here is a copy of my rsyslog.conf: > #--------------------------------------- > $ModLoad imuxsock.so > $ModLoad imklog.so > > $WorkDirectory /rsyslog > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 1m > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > #--------------------------------------- > > I attached the output from" > > rsyslogd -c 3 -f /etc/rsyslog.conf > error.txt 2>&1 > > Actually only one messagefile was written, no .qi file was created. > > bye > David Ecker > > > Rainer Gerhards schrieb: > > On Wed, 2008-10-01 at 08:43 -0700, david at lang.hm wrote: > > > >> On Wed, 1 Oct 2008, David Ecker wrote: > >> > >> > >>> Already did both with 2.18.3 but'll try again with 3.21.5 and > 3.18.4. My > >>> guess is, that the O_DIRECT in combination with the O_SYNC flag > (turning > >>> of cache) will have an impact. > >>> > >> O_DIRECT is doing very different things. I don't think you need to > worry > >> about those things, having the data not go into the OS cache is a > drawback > >> not an advantage becouse it means that when you go to pull the data > back > >> out of the file it will need to actually touch disk. it also imposes > >> significant alignment issues on the application that I don't think > you > >> want to have to desl with. > >> > > > > plus rsyslog does not care about the alignment (at this time), so I > think it is dangerous... > > > > Rainer > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From hks.private at gmail.com Mon Oct 6 16:49:26 2008 From: hks.private at gmail.com ((private) HKS) Date: Mon, 6 Oct 2008 10:49:26 -0400 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: In general, I understand HUP to be exactly what you're talking - a reload rather than a restart. I see little to lose by enforcing that in rsyslogd. The pid stability argument has no relevance for me or the tools I use since I just pull it out of pidfiles anyway. -HKS On Mon, Oct 6, 2008 at 7:56 AM, Rainer Gerhards wrote: > Lorenzo, > >> Habits, pid stability, monitoring systems, one less thing to break... > > I understand habits and pid stability. What do you mean by "monitoring > system" and "one less thing to break"? > > Anyone else with an opinion? > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 6 17:08:07 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:08:07 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > Hi List, > > I again have a backwards compatibility question. > > HUP, in sysklogd, means a full restart of the syslogd. This is done > because the configuration may have been changed and to apply the new > one, the previous one must be shut down. This is even more so the case > in rsyslog, which has multiple dynamically loadable plugins and all > that... > > rsyslog has implemented HUP processing to be compatible to sysklogd. > > However, in practice (as far as I know) HUP is almost always being used > to close the output files after log rotation. An automatted config > reaload indication usually does not happen (but operators use HUP after > doing a config change). > > HUP, as it currently is, is an extremely expensive operation. Keep in > mind that it is a full daemon restart, so, among other things, it > unloads modules and tears down (TCP, RELP, TLS, ...) connections. In > most cases, though, this would not be needed. > > I now consider changing the way HUP works. I would like to model it in a > way that closes files and clears caches (where useful), but not do a > full restart. > > Initially, I thought to use a special signal (e.g. USR2) for this > purpose. But still logrotate, as a well-known example, would cause a > daemon restart where none is needed. So I now think it would be best to > change HUP in the way described above. Those that need to apply a config > change would then need to do a regular daemon restart - the same thing > that is done with other daemons. In order to keep compatible, I would > add an $HUPisRestart option, by default off, which could be used to > re-enable previous behavior. > > I would appreciate feedback on this plan. Are there any concerns? Is > this a good idea or a bad one? Does it break something I have not yet > mentioned? could you do something along the lines of checking to see if the config file changed, and if so do a full restart, otherwise just the flush/close? if you know when you started you may be able to just check the last-modified time of the config file and do a restart if it's been modified after you started. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:08:08 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:08:08 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> > could you do something along the lines of checking to see if the config > file changed, and if so do a full restart, otherwise just the > flush/close? > > if you know when you started you may be able to just check the > last-modified time of the config file and do a restart if it's been > modified after you started. That's a very interesting idea. Do you think this criterion is sufficient? Rainer From david at lang.hm Mon Oct 6 17:15:40 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 6 Oct 2008 08:15:40 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> could you do something along the lines of checking to see if the > config >> file changed, and if so do a full restart, otherwise just the >> flush/close? >> >> if you know when you started you may be able to just check the >> last-modified time of the config file and do a restart if it's been >> modified after you started. > > That's a very interesting idea. Do you think this criterion is > sufficient? I was thinking of other possible items (doing a checksum of the file, doing a syntactic check to see if the new is equivalent to the old, etc) and I think that they don't end up adding noticable value the only way the last-modified check would fail is if your clock goes backwards (and goes backwards far enough that the 'new' file is modified before you last started). two ways to deal with that 1. store the last-modified time of the file when you start up and re-read it if it has changed (still vunerable if the clock moved back to that exact second, but a very small windows) 2. document the issue and tell people that if their clock goes backwards and the HUP doesn't re-read the config file do a 'touch' of the file and try again. David Lang From rgerhards at hq.adiscon.com Mon Oct 6 17:19:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:19:23 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AC@grfint2.intern.adiscon.com> This sounds very reasonable. Thanks for the good suggestion! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, October 06, 2008 5:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > >> could you do something along the lines of checking to see if the > > config > >> file changed, and if so do a full restart, otherwise just the > >> flush/close? > >> > >> if you know when you started you may be able to just check the > >> last-modified time of the config file and do a restart if it's been > >> modified after you started. > > > > That's a very interesting idea. Do you think this criterion is > > sufficient? > > I was thinking of other possible items (doing a checksum of the file, > doing a syntactic check to see if the new is equivalent to the old, > etc) > and I think that they don't end up adding noticable value > > the only way the last-modified check would fail is if your clock goes > backwards (and goes backwards far enough that the 'new' file is > modified > before you last started). two ways to deal with that > > 1. store the last-modified time of the file when you start up and re- > read > it if it has changed (still vunerable if the clock moved back to that > exact second, but a very small windows) > > 2. document the issue and tell people that if their clock goes > backwards > and the HUP doesn't re-read the config file do a 'touch' of the file > and > try again. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Mon Oct 6 17:45:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 17:45:56 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 : > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >>> could you do something along the lines of checking to see if the >> config >>> file changed, and if so do a full restart, otherwise just the >>> flush/close? >>> >>> if you know when you started you may be able to just check the >>> last-modified time of the config file and do a restart if it's been >>> modified after you started. >> >> That's a very interesting idea. Do you think this criterion is >> sufficient? Don't forget $IncludeConfig Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 17:47:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 6 Oct 2008 17:47:46 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Args... thanks Michael, this makes life a little less easy ;) So we do not longer have a clear indication. Would it make sense to require that the main config file is touched when something in the includes is changed? This could be documented. And a restart of course pulls everything. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Monday, October 06, 2008 5:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > 2008/10/6 : > > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > >>> could you do something along the lines of checking to see if the > >> config > >>> file changed, and if so do a full restart, otherwise just the > >>> flush/close? > >>> > >>> if you know when you started you may be able to just check the > >>> last-modified time of the config file and do a restart if it's been > >>> modified after you started. > >> > >> That's a very interesting idea. Do you think this criterion is > >> sufficient? > > Don't forget $IncludeConfig > > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From aoz.syn at gmail.com Mon Oct 6 18:29:03 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 6 Oct 2008 10:29:03 -0600 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> Message-ID: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> On Mon, Oct 6, 2008 at 08:49, (private) HKS wrote: > In general, I understand HUP to be exactly what you're talking - a > reload rather than a restart. I see little to lose by enforcing that > in rsyslogd. Ditto - IMHO SIGHUP has come to signify 'change what you can without restarting' or a "hot restart". Some daemons are capable of fully re-configuring on the fly, i.e. "warm restart". It would be rather complex but one could mark particular configuration pragmas as requiring a full tear-down and issue a warning if a warm restart was issued and those had changed. The third option (which many poorly-written init scripts do anyway) is a "cold restart", where the process is killed off and manually re-started, allowing full-blown reconfiguration. I don't care what signals are used for what, beyond the [de facto?] standard of 'skip a beat' when given SIGHUP. Beyond the scope of this discussion, I've not looked at your configuration parsing but wonder if you're not parsing it into an internal, static structure and handling bits from there. If you were, it would be trivial to parse your new configuration, see how/whether it differs from the running one, and shift over if able - like failsafe firmware in embedded systems. That would also offer interesting implications for alternative configuration syntaxes. RB From rgerhards at hq.adiscon.com Mon Oct 6 18:34:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:34:12 +0200 Subject: [rsyslog] HUP action In-Reply-To: <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <1223286265.2682.564.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F29E@grfint2.intern.adiscon.com> <4255c2570810060929s69ee5b32kf6a44b0eca99479c@mail.gmail.com> Message-ID: <1223310852.2682.566.camel@rgf9dev.intern.adiscon.com> On Mon, 2008-10-06 at 10:29 -0600, RB wrote: > Beyond the scope of this discussion, I've not looked at your > configuration parsing but wonder if you're not parsing it into an > internal, static structure and handling bits from there. If you were, > it would be trivial to parse your new configuration, see how/whether > it differs from the running one, and shift over if able - like > failsafe firmware in embedded systems. That would also offer > interesting implications for alternative configuration syntaxes. That is quite complex, because the configuration defines what can be configured. With the plug-in architecture, we do not have any fixed memory structures. So, we would need to spawn a new process, load plugins there, check what in-memory structures they create and, if they differ, than use this new configuration instead of the old one. Rainer From mbiebl at gmail.com Mon Oct 6 18:37:31 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 6 Oct 2008 18:37:31 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: 2008/10/6 Rainer Gerhards : > Args... thanks Michael, this makes life a little less easy ;) So we do > not longer have a clear indication. > > Would it make sense to require that the main config file is touched when > something in the includes is changed? This could be documented. And a > restart of course pulls everything. I don't think that would be such a good idea. It is not very intuitive and people are not used to such a behaviour. They (at least I do) expect "reload" to reload the complete configuration. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 6 18:46:16 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 06 Oct 2008 18:46:16 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> Message-ID: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> So let me try to sum up where we are now: - it is considered useful to have a full daemon restart be done via HUP - we can not sufficiently simply detect whether the configuration has changed or not - there are situations where it is useful to have the ability to just close files, clear caches etc - people do not like existing things be used in new ways (least surprise principle) So I conclude: - SIGHUP, as ugly as it is, must stay with existing semantics - a new signal can be defined to just do file closure etc Unfortunately, this means that most systems will still use the terribly expensive during e.g. logrotation. However, this is considered acceptable because a) it always was this way, b) a higher demand environment then has options to avoid that. Over time, package maintainers my get maintainers of logrotate involved to change the HUP to the new signal. Am I grasping this right? Rainer On Mon, 2008-10-06 at 18:37 +0200, Michael Biebl wrote: > 2008/10/6 Rainer Gerhards : > > Args... thanks Michael, this makes life a little less easy ;) So we do > > not longer have a clear indication. > > > > Would it make sense to require that the main config file is touched when > > something in the includes is changed? This could be documented. And a > > restart of course pulls everything. > > I don't think that would be such a good idea. > > It is not very intuitive and people are not used to such a behaviour. > They (at least I do) expect "reload" to reload the complete configuration. > > > Michael > From linggao at us.ibm.com Mon Oct 6 20:44:40 2008 From: linggao at us.ibm.com (Ling Gao) Date: Mon, 6 Oct 2008 14:44:40 -0400 Subject: [rsyslog] log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From dt.abernathy at gmail.com Tue Oct 7 18:00:24 2008 From: dt.abernathy at gmail.com (Dan Abernathy) Date: Tue, 7 Oct 2008 10:00:24 -0600 Subject: [rsyslog] FreeBSD and rsyslog Message-ID: Hi, I've been working on integrating rsyslog into a FreeBSD environment, and the following issues and questions have come up along the way: 1. The security facility isn't deprecated in FreeBSD, and I've noticed that this is aliased to the auth facility in rsyslog. I have make a quick patch for this on FreeBSD in order to log to the security facility again, and added the ntp facility (also supported). These changes may be more suitable for the FreeBSD ports tree. 2. The other FreeBSD facility in use is console, which logs all writes to /dev/console to the console log facility. I'm not sure how important this feature is, and it's certainly something I can live without, but it doesn't seem supported in rsyslog. 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. I was curious about support for /var/run/logpriv, however I can't find a great deal of information about how and where this socket is used - other than for privileged applications. 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP address is probably at the top of my list right now. Is there any feature to achieve this that I'm missing? Any thoughts, suggestions and discussion welcome. Dan. From hks.private at gmail.com Tue Oct 7 18:38:16 2008 From: hks.private at gmail.com ((private) HKS) Date: Tue, 7 Oct 2008 12:38:16 -0400 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: On Tue, Oct 7, 2008 at 12:00 PM, Dan Abernathy wrote: > Hi, > > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: > > 1. The security facility isn't deprecated in FreeBSD, and I've noticed that > this is aliased to the auth facility in rsyslog. I have make a quick patch > for this on FreeBSD in order to log to the security facility again, and > added the ntp facility (also supported). These changes may be more suitable > for the FreeBSD ports tree. Excellent, that's good news. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. > > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Can't answer either of these. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? I assume you're running rsyslogd v3. If you're using v2, check the man pages (they're still relevant at that point). When you create the UDP/TCP server in the rsyslog.conf file, you give it the address and port you want it to bind to: $UDPServerAddress 10.123.0.19 $UDPServerRun 514 Hope that helps. -HKS From aoz.syn at gmail.com Tue Oct 7 18:53:05 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 7 Oct 2008 10:53:05 -0600 Subject: [rsyslog] FreeBSD and rsyslog In-Reply-To: References: Message-ID: <4255c2570810070953j7d489dfcre45b6515d9d68b5b@mail.gmail.com> > I've been working on integrating rsyslog into a FreeBSD environment, and the > following issues and questions have come up along the way: You're not alone, I've just gotten distracted along the way and not chased down any issues. > 2. The other FreeBSD facility in use is console, which logs all writes to > /dev/console to the console log facility. I'm not sure how important this > feature is, and it's certainly something I can live without, but it doesn't > seem supported in rsyslog. Curious, I'd not carefully looked at that - it's a character device. No idea. > 3. FreeBSD syslog reads from two sockets, /var/run/log and /var/run/logpriv. > I was curious about support for /var/run/logpriv, however I can't find a > great deal of information about how and where this socket is used - other > than for privileged applications. Should be addressable with imuxsock and permissions; not much to that. > 4. Finding the equivalent to the syslogd -b flag, to bind to a specific IP > address is probably at the top of my list right now. Is there any feature to > achieve this that I'm missing? Configuration pragma $UDPServerAddress From linggao at us.ibm.com Tue Oct 7 19:39:09 2008 From: linggao at us.ibm.com (Ling Gao) Date: Tue, 7 Oct 2008 13:39:09 -0400 Subject: [rsyslog] Fw: log forwarding in version 3 Message-ID: Hi, We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. We'd like to have logs sent from tier3 to tier1 through tier2. (tier3->tier2-> tier1). In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to enable them receiving remote logs. $ModLoad imudp.so $UDPServerRun 514 In tier2, we have set up the following in etc/rsyslog.conf to send the logs to tier1: *.* @tier1 In tier3, we have set up the following in /etc/rsyslog.conf to send the logs to tier2 : *.* @tier2 Question: 1. What other setting we need to enable in tier2 so that the logs from tier3 can be forwarded to tier1 via tier2? (something like -h option in lower versions). 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some are Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using rsyslog v3, do we need to change all to the compatibility mode? Thanks, Ling From rgerhards at hq.adiscon.com Wed Oct 8 11:04:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 11:04:26 +0200 Subject: [rsyslog] my responsiveness Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D4@grfint2.intern.adiscon.com> Hi folks, unfortunately, I am currently a bit unresponsive to mailing list posts. The reason is that I am reviewing the threading model, which is quite complex and does not play well with any distraction. Also, there is some additional (paid) work to be done and this obviously has some priority. Thanks to all who contribute their knowledge. I try my best to catch up with the posts. I think the situation will become much better in a few days at latest. Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:42:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:42:27 +0200 Subject: [rsyslog] Fw: log forwarding in version 3 In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2D9@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ling Gao > Sent: Tuesday, October 07, 2008 7:39 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Fw: log forwarding in version 3 > > > > Hi, > We have a 3 tier cluster that uses Fedora 9 which has rsyslog v3. > We'd > like to have logs sent from tier3 to tier1 through tier2. (tier3- > >tier2-> > tier1). > > In tier2 and tier1, we have setup the following in /etc/rsyslog.conf to > enable them receiving remote logs. > $ModLoad imudp.so > $UDPServerRun 514 > > In tier2, we have set up the following in etc/rsyslog.conf to send the > logs > to tier1: > *.* @tier1 > > In tier3, we have set up the following in /etc/rsyslog.conf to send the > logs to tier2 : > *.* @tier2 > > Question: > 1. What other setting we need to enable in tier2 so that the logs from > tier3 can be forwarded to tier1 via tier2? (something like -h option > in > lower versions). None (but keep the firewalls in mind!) > 2. If some of the nodes in tier3 are Fedora 8 using rsyslog v2 and some > are > Fedora 9 using rsyslog v3. Tier2 and tier1 are both Fedora 9 using > rsyslog > v3, do we need to change all to the compatibility mode? The compatibility mode affects the way the config file and command line options are interpreted. It does not change anything other than those settings. Review the compatibility guide if there are some problems with any of your settings (my guess is "no") and if so, adjust them. It is recommended to run each instance in native mode. Rainer From r.bhatia at ipax.at Wed Oct 8 12:49:05 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Wed, 08 Oct 2008 12:49:05 +0200 Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EC9021.1000701@ipax.at> Rainer Gerhards wrote: > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not no knowing the code, i suggest md5/sha1/... checksum the whole file after integrating everything including rsyslog.d/* into one flat file. > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) i, currently, do not care that much about this issue - at least regarding the reload vs. restart issue. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From rgerhards at hq.adiscon.com Wed Oct 8 12:56:11 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:56:11 +0200 Subject: [rsyslog] HUP action In-Reply-To: <48EC9021.1000701@ipax.at> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <48EC9021.1000701@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Wednesday, October 08, 2008 12:49 PM > To: rsyslog-users > Subject: Re: [rsyslog] HUP action > > Rainer Gerhards wrote: > > - it is considered useful to have a full daemon restart be done via > HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > no knowing the code, i suggest md5/sha1/... checksum the whole file > after integrating everything including rsyslog.d/* into one flat > file. Quite honestly, I do not like the extra effort. I now have to define two signals in any case, so let the user use the right one for what he intends to do ;) Rainer From rgerhards at hq.adiscon.com Wed Oct 8 12:58:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Oct 2008 12:58:26 +0200 Subject: [rsyslog] HUP action In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com><48EC9021.1000701@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA44F2DA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2DB@grfint2.intern.adiscon.com> > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > > Sent: Wednesday, October 08, 2008 12:49 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] HUP action > > > > Rainer Gerhards wrote: > > > - it is considered useful to have a full daemon restart be done via > > HUP > > > - we can not sufficiently simply detect whether the configuration > > > has changed or not > > > > no knowing the code, i suggest md5/sha1/... checksum the whole file > > after integrating everything including rsyslog.d/* into one flat > > file. > > Quite honestly, I do not like the extra effort. I now have to define > two > signals in any case, so let the user use the right one for what he > intends to do ;) Sorry, I was a bit brief: doing the checksum requires interpretation of the config files, at least as far as the $IncludeConfig statements are concerned. This is costly, as is the whole gathering process to build the checksum. So to be on the efficient side, I need to have a dedicated HUP for "non-reload" in any case. Thus I think there is not much added benefit in doing the checksumming. Rainer From david at ecker-software.de Thu Oct 9 10:53:02 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 10:53:02 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> Message-ID: <48EDC66E.1060202@ecker-software.de> Hi Rainer, the assert error seemed to be fixed in HEAD. Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to work a lot better. /etc/rsyslog.conf ---------------- $WorkDirectory /rsyslog/ $ActionQueueFileName buffer $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType Disk $ActionQueueMaxFileSize 100m $ActionQueueSize 10000000 $ActionQueueCheckpointInterval 1 $ActionResumeRetryCount -1 *.* @@10.8.0.1:514 ----------------- I found out that the queue was limited to 1000 elements if you do not define ActionQueueSize inside the config. One difference was that version 3.18.3 actually created one file per msg if I did set ActionQueueCheckpointInterval to 1. Right now only one file in addition to the .qi file is created containing all messages. I haven't waited to see what will happen if I reach 100MB on the data file. If I shutdown rsyslog normally some status information seems to be written to the .qi file (508 bytes -> 1024 bytes). After restarting the qi file actually shrinks to 508 bytes again. If I kill rsyslogd (SIGKILL) the qi file is not updated with this information (as expected). But it looks like the queue is still working correctly after restarting rsyslogd at least it doesn't invalidate the queue or loosing messages. Turning the system off immediatly seems to be working most of the times. After 11 tries the last one failed. It actually looked like that I turned the system off during a write. bye David Ecker Rainer Gerhards schrieb: > Please use the version from git. I didn't realize that the bug affects > normal operations, but obviously it does. This is fixed and I'll see > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > today. > > Gitweb available at http://git.adiscon.com > > Rainer > From rgerhards at hq.adiscon.com Thu Oct 9 13:56:14 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 09 Oct 2008 13:56:14 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDC66E.1060202@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> Message-ID: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Hi David, On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > Hi Rainer, > > the assert error seemed to be fixed in HEAD. It is good to hear this. I begun to have some doubts when I reviewed the code. I would really appreciate if you could download and test this version here: http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz I will probably release that tomorrow and so some indication if the problem is actually gone would be very good. The final 3.21.6 will see one more patch, but nothing that affects the assert in question. > > Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to > work a lot better. > > /etc/rsyslog.conf > ---------------- > $WorkDirectory /rsyslog/ > $ActionQueueFileName buffer > $ActionQueueMaxDiskSpace 1g > $ActionQueueSaveOnShutdown on > $ActionQueueType Disk > $ActionQueueMaxFileSize 100m > $ActionQueueSize 10000000 > $ActionQueueCheckpointInterval 1 > $ActionResumeRetryCount -1 > *.* @@10.8.0.1:514 > ----------------- > > I found out that the queue was limited to 1000 elements if you do not > define ActionQueueSize inside the config. Yes, that's the default for *action* queues (the main message queue default is different, I think 10,000). > One difference was that version 3.18.3 actually created one file per msg > if I did set ActionQueueCheckpointInterval to 1. Right now only one > file in addition to the .qi file is created containing all messages. I > haven't waited to see what will happen if I reach 100MB on the data file. That's probably a result of a bug fixed in the repo but not yet released. But I don't check if it is gone now. 3.18.x will only see one more release, then we move on to 3.20.x. > If I shutdown rsyslog normally some status information seems to be > written to the .qi file (508 bytes -> 1024 bytes). After restarting the > qi file actually shrinks to 508 bytes again. That's right. > > If I kill rsyslogd (SIGKILL) the qi file is not updated with this > information (as expected). But it looks like the queue is still working > correctly after restarting rsyslogd at least it doesn't invalidate the > queue or loosing messages. I'd say it depends on when exactly it is doing when it is being killed. > > Turning the system off immediatly seems to be working most of the times. > After 11 tries the last one failed. It actually looked like that I > turned the system off during a write. This is also within what I expect. If you hit it during a write, things are really bad. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Please use the version from git. I didn't realize that the bug affects > > normal operations, but obviously it does. This is fixed and I'll see > > that I release 3.21.6 ASAP, but I am not sure if I manage to do this > > today. > > > > Gitweb available at http://git.adiscon.com > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at ecker-software.de Thu Oct 9 14:43:30 2008 From: david at ecker-software.de (David Ecker) Date: Thu, 09 Oct 2008 14:43:30 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de> <1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> Message-ID: <48EDFC72.2070404@ecker-software.de> Hi Rainer, with the same config it behaves similar to the HEAD version I tested in the last few days. The assert message did not appear. bye David Ecker Rainer Gerhards schrieb: > Hi David, > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > >> Hi Rainer, >> >> the assert error seemed to be fixed in HEAD. >> > > It is good to hear this. I begun to have some doubts when I reviewed the > code. I would really appreciate if you could download and test this > version here: > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > I will probably release that tomorrow and so some indication if the > problem is actually gone would be very good. The final 3.21.6 will see > one more patch, but nothing that affects the assert in question. > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed to >> work a lot better. >> >> /etc/rsyslog.conf >> ---------------- >> $WorkDirectory /rsyslog/ >> $ActionQueueFileName buffer >> $ActionQueueMaxDiskSpace 1g >> $ActionQueueSaveOnShutdown on >> $ActionQueueType Disk >> $ActionQueueMaxFileSize 100m >> $ActionQueueSize 10000000 >> $ActionQueueCheckpointInterval 1 >> $ActionResumeRetryCount -1 >> *.* @@10.8.0.1:514 >> ----------------- >> >> I found out that the queue was limited to 1000 elements if you do not >> define ActionQueueSize inside the config. >> > > Yes, that's the default for *action* queues (the main message queue > default is different, I think 10,000). > > >> One difference was that version 3.18.3 actually created one file per msg >> if I did set ActionQueueCheckpointInterval to 1. Right now only one >> file in addition to the .qi file is created containing all messages. I >> haven't waited to see what will happen if I reach 100MB on the data file. >> > > That's probably a result of a bug fixed in the repo but not yet > released. But I don't check if it is gone now. 3.18.x will only see one > more release, then we move on to 3.20.x. > > >> If I shutdown rsyslog normally some status information seems to be >> written to the .qi file (508 bytes -> 1024 bytes). After restarting the >> qi file actually shrinks to 508 bytes again. >> > > That's right. > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this >> information (as expected). But it looks like the queue is still working >> correctly after restarting rsyslogd at least it doesn't invalidate the >> queue or loosing messages. >> > > I'd say it depends on when exactly it is doing when it is being killed. > > >> Turning the system off immediatly seems to be working most of the times. >> After 11 tries the last one failed. It actually looked like that I >> turned the system off during a write. >> > > This is also within what I expect. If you hit it during a write, things > are really bad. > > >> bye >> David Ecker >> >> Rainer Gerhards schrieb: >> >>> Please use the version from git. I didn't realize that the bug affects >>> normal operations, but obviously it does. This is fixed and I'll see >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do this >>> today. >>> >>> Gitweb available at http://git.adiscon.com >>> >>> Rainer >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 9 15:31:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 9 Oct 2008 15:31:56 +0200 Subject: [rsyslog] Development of failsafe disk based queue In-Reply-To: <48EDFC72.2070404@ecker-software.de> References: <48E34A4F.308@ecker-software.de> <1222858206.2682.446.camel@rgf9dev.intern.adiscon.com> <1222863139.2682.460.camel@rgf9dev.intern.adiscon.com> <1222866669.2682.486.camel@rgf9dev.intern.adiscon.com> <48E3791D.3050304@ecker-software.de> <1222867464.2682.495.camel@rgf9dev.intern.adiscon.com> <48E37C96.3080904@ecker-software.de> <1222869574.2682.518.camel@rgf9dev.intern.adiscon.com> <48E38701.80707@ecker-software.de> <1222875861.2682.533.camel@rgf9dev.intern.adiscon.com> <48EA0775.7050105@ecker-software.de> <577465F99B41C842AAFBE9ED71E70ABA44F2A1@grfint2.intern.adiscon.com> <48EDC66E.1060202@ecker-software.de><1223553374.23821.6.camel@rgf9dev.intern.adiscon.com> <48EDFC72.2070404@ecker-software.de> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2F3@grfint2.intern.adiscon.com> Hi David, thanks for the quick feedback, much appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Ecker > Sent: Thursday, October 09, 2008 2:44 PM > To: rsyslog-users > Subject: Re: [rsyslog] Development of failsafe disk based queue > > Hi Rainer, > > with the same config it behaves similar to the HEAD version I tested in > the last few days. The assert message did not appear. > > bye > David Ecker > > Rainer Gerhards schrieb: > > Hi David, > > > > On Thu, 2008-10-09 at 10:53 +0200, David Ecker wrote: > > > >> Hi Rainer, > >> > >> the assert error seemed to be fixed in HEAD. > >> > > > > It is good to hear this. I begun to have some doubts when I reviewed > the > > code. I would really appreciate if you could download and test this > > version here: > > > > http://download.rsyslog.com/rsyslog/rsyslog-3.21.6-Test2.tar.gz > > > > I will probably release that tomorrow and so some indication if the > > problem is actually gone would be very good. The final 3.21.6 will > see > > one more patch, but nothing that affects the assert in question. > > > > > >> Mounting the ext3 filesystem with (noatime,sync,dirsync,rw) seemed > to > >> work a lot better. > >> > >> /etc/rsyslog.conf > >> ---------------- > >> $WorkDirectory /rsyslog/ > >> $ActionQueueFileName buffer > >> $ActionQueueMaxDiskSpace 1g > >> $ActionQueueSaveOnShutdown on > >> $ActionQueueType Disk > >> $ActionQueueMaxFileSize 100m > >> $ActionQueueSize 10000000 > >> $ActionQueueCheckpointInterval 1 > >> $ActionResumeRetryCount -1 > >> *.* @@10.8.0.1:514 > >> ----------------- > >> > >> I found out that the queue was limited to 1000 elements if you do > not > >> define ActionQueueSize inside the config. > >> > > > > Yes, that's the default for *action* queues (the main message queue > > default is different, I think 10,000). > > > > > >> One difference was that version 3.18.3 actually created one file per > msg > >> if I did set ActionQueueCheckpointInterval to 1. Right now only one > >> file in addition to the .qi file is created containing all messages. > I > >> haven't waited to see what will happen if I reach 100MB on the data > file. > >> > > > > That's probably a result of a bug fixed in the repo but not yet > > released. But I don't check if it is gone now. 3.18.x will only see > one > > more release, then we move on to 3.20.x. > > > > > >> If I shutdown rsyslog normally some status information seems to be > >> written to the .qi file (508 bytes -> 1024 bytes). After restarting > the > >> qi file actually shrinks to 508 bytes again. > >> > > > > That's right. > > > > > >> If I kill rsyslogd (SIGKILL) the qi file is not updated with this > >> information (as expected). But it looks like the queue is still > working > >> correctly after restarting rsyslogd at least it doesn't invalidate > the > >> queue or loosing messages. > >> > > > > I'd say it depends on when exactly it is doing when it is being > killed. > > > > > >> Turning the system off immediatly seems to be working most of the > times. > >> After 11 tries the last one failed. It actually looked like that I > >> turned the system off during a write. > >> > > > > This is also within what I expect. If you hit it during a write, > things > > are really bad. > > > > > >> bye > >> David Ecker > >> > >> Rainer Gerhards schrieb: > >> > >>> Please use the version from git. I didn't realize that the bug > affects > >>> normal operations, but obviously it does. This is fixed and I'll > see > >>> that I release 3.21.6 ASAP, but I am not sure if I manage to do > this > >>> today. > >>> > >>> Gitweb available at http://git.adiscon.com > >>> > >>> Rainer > >>> > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > From friedl at hq.adiscon.com Thu Oct 9 17:32:51 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 9 Oct 2008 17:32:51 +0200 Subject: [rsyslog] rsyslog 3.18.5 (v3-stable) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F2FD@grfint2.intern.adiscon.com> Hi all, we have just released rsyslog 3.18.5, a member of the v3-stable branch. This is most probably the last 3.18.x version, the current beta will soon become the new stable. Rsyslog 3.18.5 is a bug fixing release with some enhanced documentation. The most important bug fixes addresses some issues during HUP, which could lead to a segfault. This is a recommended update for all users of the v3-stable branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-133.phtml Changelog: http://www.rsyslog.com/Article281.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From kyuhwanzz at gmail.com Wed Oct 15 14:03:05 2008 From: kyuhwanzz at gmail.com (Kyu-Hwan Yun) Date: Wed, 15 Oct 2008 21:03:05 +0900 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? Message-ID: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Hi, all. My question is, How cound rsyslog recognize that remote server is alive? port open/close, network interface up/down, or etc... Which method is the answer? From rgerhards at hq.adiscon.com Wed Oct 15 15:08:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Oct 2008 15:08:43 +0200 Subject: [rsyslog] How can the rsyslog recognize remote server is alive? In-Reply-To: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> References: <8f4097740810150503n20030238u3bd4c29963b51417@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F378@grfint2.intern.adiscon.com> The simple answer is that we try a send() call and if it succeeds, that's a good indication the other side is alive. If it fails, the retry logic tries to restore the connection. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Kyu-Hwan Yun > Sent: Wednesday, October 15, 2008 2:03 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How can the rsyslog recognize remote server is > alive? > > Hi, all. > > My question is, > > How cound rsyslog recognize that remote server is alive? > > port open/close, network interface up/down, or etc... > > Which method is the answer? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Thu Oct 16 15:11:36 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 16 Oct 2008 15:11:36 +0200 Subject: [rsyslog] rsyslog 3.19.12 (beta) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F39F@grfint2.intern.adiscon.com> Hi all, rsyslog 3.12.19 has been released, a member of the beta branch. It is mainly a refresh of the current beta which provides all those bug-fixes that have been made to the stable releases. Other than that, it includes some minor documentation updates and a bugfix for a beta-specific issue. This is scheduled to become the new v3-stable soon. It is a recommended update for all beta branch users. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-134.phtml Changelog: http://www.rsyslog.com/Article283.phtml As always, feedback is appreciated. Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From david at lang.hm Sat Oct 18 06:40:22 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 17 Oct 2008 21:40:22 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 6 Oct 2008, Rainer Gerhards wrote: > So let me try to sum up where we are now: > > - it is considered useful to have a full daemon restart be done via HUP > - we can not sufficiently simply detect whether the configuration > has changed or not > - there are situations where it is useful to have the ability to > just close files, clear caches etc > - people do not like existing things be used in new ways > (least surprise principle) > > So I conclude: > > - SIGHUP, as ugly as it is, must stay with existing semantics > - a new signal can be defined to just do file closure etc > > Unfortunately, this means that most systems will still use the terribly > expensive during e.g. logrotation. However, this is considered > acceptable because a) it always was this way, b) a higher demand > environment then has options to avoid that. Over time, package > maintainers my get maintainers of logrotate involved to change the HUP > to the new signal. > > Am I grasping this right? I learned this week that on a HUP rsyslog will loose any messages that it has in it's memory queue (due to the fact that it basicly does a full shutdown and restart). I suspect that many other people would be surprised at this as well. It looks like we have two options. 1. HUP will notice config file changes, but can loose queued logs 2. HUP will not notice config file changes, but will not loose queued logs I think that there would be less surprise in the second case. all the scripts that use HUP to roll logs don't really care about the config files I guess there is a third case, make a HUP not drop the queue, but if the config file options for the queue change significantly, this can be a very difficult thing to do David Lang From rgerhards at hq.adiscon.com Sat Oct 18 10:32:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 18 Oct 2008 10:32:21 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > On Mon, 6 Oct 2008, Rainer Gerhards wrote: > > > So let me try to sum up where we are now: > > > > - it is considered useful to have a full daemon restart be done via HUP > > - we can not sufficiently simply detect whether the configuration > > has changed or not > > - there are situations where it is useful to have the ability to > > just close files, clear caches etc > > - people do not like existing things be used in new ways > > (least surprise principle) > > > > So I conclude: > > > > - SIGHUP, as ugly as it is, must stay with existing semantics > > - a new signal can be defined to just do file closure etc > > > > Unfortunately, this means that most systems will still use the terribly > > expensive during e.g. logrotation. However, this is considered > > acceptable because a) it always was this way, b) a higher demand > > environment then has options to avoid that. Over time, package > > maintainers my get maintainers of logrotate involved to change the HUP > > to the new signal. > > > > Am I grasping this right? > > I learned this week that on a HUP rsyslog will loose any messages that it > has in it's memory queue (due to the fact that it basicly does a full > shutdown and restart). I suspect that many other people would be surprised > at this as well. Well, actually it depends. In the default configuration, this is true. But the spirit of the default configuration (suitable for most simple cases) is "I don't care about message loss". If you care about message loss, queues need to be configured so that no already-gathered messages are lost between sessions. If done so, messages are written to files before a shutdown and read from it upon restart. Of course, messages are still lost while the system is doing the restart (new ones). For a HUP and a very busy system, this can mean it takes a couple of seconds, even minutes, to do a full restart. After I noticed that the "message loss on HUP if not configured to avoid it" is not obvious, I added some explicit doc to the man pages so that people are warned they do a full restart. > > It looks like we have two options. > > 1. HUP will notice config file changes, but can loose queued logs > > 2. HUP will not notice config file changes, but will not loose queued logs > > I think that there would be less surprise in the second case. see comments above > > all the scripts that use HUP to roll logs don't really care about the > config files that was my initial observation. I will define another signal to just close the file. I hope I can get this done next week. Unfortunately, the financial crisis begins to hit the real world and this causes me to currently spent less time on new additions to rsyslog and more on profit-generating projects, but that should be doable in a day or two (the output module interface needs to be extend, a new administrative linked list implemented and the signal handler be done - changes to outputs are also necessary). > > I guess there is a third case, make a HUP not drop the queue, but if the > config file options for the queue change significantly, this can be a very > difficult thing to do This is extremely difficult, but of course doable. I think, however, this is only needed for a very limited number of cases. So this is probably something that I will only work on if a sponsor for this work shows up. Rainer > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sat Oct 18 11:32:32 2008 From: david at lang.hm (david at lang.hm) Date: Sat, 18 Oct 2008 02:32:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: On Sat, 18 Oct 2008, Rainer Gerhards wrote: > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >> >>> So let me try to sum up where we are now: >>> >>> - it is considered useful to have a full daemon restart be done via HUP >>> - we can not sufficiently simply detect whether the configuration >>> has changed or not >>> - there are situations where it is useful to have the ability to >>> just close files, clear caches etc >>> - people do not like existing things be used in new ways >>> (least surprise principle) >>> >>> So I conclude: >>> >>> - SIGHUP, as ugly as it is, must stay with existing semantics >>> - a new signal can be defined to just do file closure etc >>> >>> Unfortunately, this means that most systems will still use the terribly >>> expensive during e.g. logrotation. However, this is considered >>> acceptable because a) it always was this way, b) a higher demand >>> environment then has options to avoid that. Over time, package >>> maintainers my get maintainers of logrotate involved to change the HUP >>> to the new signal. >>> >>> Am I grasping this right? >> >> I learned this week that on a HUP rsyslog will loose any messages that it >> has in it's memory queue (due to the fact that it basicly does a full >> shutdown and restart). I suspect that many other people would be surprised >> at this as well. > > Well, actually it depends. In the default configuration, this is true. > But the spirit of the default configuration (suitable for most simple > cases) is "I don't care about message loss". there's a big difference between "I don't care about message loss if a system crashed" and "I don't care about message loss at all" admins should evaluate the probability of the different message loss scenerios and decide on the appropriate cost/benifit for their situation. it's very possible that someone may not care about loosing logs when a system crashes (becouse it's a fairly rare event), but isn't willing to loose logs just to rotate log files. > If you care about message > loss, queues need to be configured so that no already-gathered messages > are lost between sessions. If done so, messages are written to files > before a shutdown and read from it upon restart. Of course, messages are > still lost while the system is doing the restart (new ones). > > For a HUP and a very busy system, this can mean it takes a couple of > seconds, even minutes, to do a full restart. and unfortunantly, the process of writing the pending queue to disk and then reading it back after a restart can greatly lengthen the restart, causing incoming logs to be missed (depending on how able the log sender is to queue the log messages for your) > After I noticed that the "message loss on HUP if not configured to avoid > it" is not obvious, I added some explicit doc to the man pages so that > people are warned they do a full restart. thanks. >> >> It looks like we have two options. >> >> 1. HUP will notice config file changes, but can loose queued logs >> >> 2. HUP will not notice config file changes, but will not loose queued logs >> >> I think that there would be less surprise in the second case. > > see comments above the purpose of this message was mostly to try and prompt the other people who commented in this thread to think about the issue and comment on their expectations. >> >> all the scripts that use HUP to roll logs don't really care about the >> config files > > that was my initial observation. I will define another signal to just > close the file. I hope I can get this done next week. Unfortunately, the > financial crisis begins to hit the real world and this causes me to > currently spent less time on new additions to rsyslog and more on > profit-generating projects, but that should be doable in a day or two > (the output module interface needs to be extend, a new administrative > linked list implemented and the signal handler be done - changes to > outputs are also necessary). I am not trying to preasure you into rushing on this, just to get opinions on which is the 'more expected' behavior for HUP >> >> I guess there is a third case, make a HUP not drop the queue, but if the >> config file options for the queue change significantly, this can be a very >> difficult thing to do > > This is extremely difficult, but of course doable. I think, however, > this is only needed for a very limited number of cases. So this is > probably something that I will only work on if a sponsor for this work > shows up. I would definantly not recommend this one becouse of the difficulty involved, but in the interest of completeness I thought it needed to be mentioned. David Lang From rgerhards at hq.adiscon.com Mon Oct 20 09:41:24 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 20 Oct 2008 09:41:24 +0200 Subject: [rsyslog] HUP action In-Reply-To: References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> Message-ID: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: > On Sat, 18 Oct 2008, Rainer Gerhards wrote: > > > On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: > >> On Mon, 6 Oct 2008, Rainer Gerhards wrote: > >> > >>> So let me try to sum up where we are now: > >>> > >>> - it is considered useful to have a full daemon restart be done via HUP > >>> - we can not sufficiently simply detect whether the configuration > >>> has changed or not > >>> - there are situations where it is useful to have the ability to > >>> just close files, clear caches etc > >>> - people do not like existing things be used in new ways > >>> (least surprise principle) > >>> > >>> So I conclude: > >>> > >>> - SIGHUP, as ugly as it is, must stay with existing semantics > >>> - a new signal can be defined to just do file closure etc > >>> > >>> Unfortunately, this means that most systems will still use the terribly > >>> expensive during e.g. logrotation. However, this is considered > >>> acceptable because a) it always was this way, b) a higher demand > >>> environment then has options to avoid that. Over time, package > >>> maintainers my get maintainers of logrotate involved to change the HUP > >>> to the new signal. > >>> > >>> Am I grasping this right? > >> > >> I learned this week that on a HUP rsyslog will loose any messages that it > >> has in it's memory queue (due to the fact that it basicly does a full > >> shutdown and restart). I suspect that many other people would be surprised > >> at this as well. > > > > Well, actually it depends. In the default configuration, this is true. > > But the spirit of the default configuration (suitable for most simple > > cases) is "I don't care about message loss". > > there's a big difference between "I don't care about message loss if a > system crashed" and "I don't care about message loss at all" That's right. But I was not talking about a crash condition. I was talking about a shutdown (or a restart). With the default settings, rsyslog does what sysklogd does, that is (mostly) not care about message loss. If you shut down the daemon, it tries to drain the queue for a short timeout. If all messages can be processed during that time, we are fine and terminate rsyslogd without message loss. However, if there are too many messages queued, the daemon terminates after the timeout has expired and discards whatever messages are left unprocessed. This is done in an effort to keep compatible with sysklogd. Also, in most scenarios it is highly unlikely that message loss will occur, because the queue will not hold extended data for an extended period of time. If, however, this is anticipated, there are ways to configure rsyslog not to loose messages. For example, the timeout can be increased and/or the queue can be made disk-assisted and data be written to disk if they can not be processed during the timeout. During queue drain, no new messages are accepted (otherwise, how should we finally shut down?). Given a busy system, there is unavoidable message loss. Either you use old messages you already had in the queue, but have a quick restart and loose few new messages. Or you save the already existing messages but experience loss of new messages. This is another reason for the default, which I consider the best compromise under the circumstances given. This is also a good indication why you should not unnecessarily restart rsyslgod (or any other syslogd, because this is not a project-specific problem but rather a general one). With fully reliable protocols, you can of course make the sender block while you are shutting down. But all of this requires very careful planning. I assume folks who want to do that either have the necessary knowledge themselvs or hire someone who does. So there are three scenarios: a) hup b) regular shutdown c) system abort I think you did not take into account b), which is a quite common case. Other than that, I fully agree to you post. Rainer From david at lang.hm Mon Oct 20 19:07:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 10:07:32 -0700 (PDT) Subject: [rsyslog] HUP action In-Reply-To: <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> References: <1223285045.2682.561.camel@rgf9dev.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AB@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44F2AE@grfint2.intern.adiscon.com> <1223311576.2682.571.camel@rgf9dev.intern.adiscon.com> <1224318741.23821.73.camel@rgf9dev.intern.adiscon.com> <1224488484.23821.83.camel@rgf9dev.intern.adiscon.com> Message-ID: On Mon, 20 Oct 2008, Rainer Gerhards wrote: > On Sat, 2008-10-18 at 02:32 -0700, david at lang.hm wrote: >> On Sat, 18 Oct 2008, Rainer Gerhards wrote: >> >>> On Fri, 2008-10-17 at 21:40 -0700, david at lang.hm wrote: >>>> On Mon, 6 Oct 2008, Rainer Gerhards wrote: >>>> >>>>> So let me try to sum up where we are now: >>>>> >>>>> - it is considered useful to have a full daemon restart be done via HUP >>>>> - we can not sufficiently simply detect whether the configuration >>>>> has changed or not >>>>> - there are situations where it is useful to have the ability to >>>>> just close files, clear caches etc >>>>> - people do not like existing things be used in new ways >>>>> (least surprise principle) >>>>> >>>>> So I conclude: >>>>> >>>>> - SIGHUP, as ugly as it is, must stay with existing semantics >>>>> - a new signal can be defined to just do file closure etc >>>>> >>>>> Unfortunately, this means that most systems will still use the terribly >>>>> expensive during e.g. logrotation. However, this is considered >>>>> acceptable because a) it always was this way, b) a higher demand >>>>> environment then has options to avoid that. Over time, package >>>>> maintainers my get maintainers of logrotate involved to change the HUP >>>>> to the new signal. >>>>> >>>>> Am I grasping this right? >>>> >>>> I learned this week that on a HUP rsyslog will loose any messages that it >>>> has in it's memory queue (due to the fact that it basicly does a full >>>> shutdown and restart). I suspect that many other people would be surprised >>>> at this as well. >>> >>> Well, actually it depends. In the default configuration, this is true. >>> But the spirit of the default configuration (suitable for most simple >>> cases) is "I don't care about message loss". >> >> there's a big difference between "I don't care about message loss if a >> system crashed" and "I don't care about message loss at all" > > That's right. But I was not talking about a crash condition. I was > talking about a shutdown (or a restart). With the default settings, > rsyslog does what sysklogd does, that is (mostly) not care about message > loss. If you shut down the daemon, it tries to drain the queue for a > short timeout. If all messages can be processed during that time, we are > fine and terminate rsyslogd without message loss. the big difference is that rsysklog does not maintain a queue, so when it shuts down it doesn't loose any messages that it has processed (even with async writes it has passed the logs to the OS and so those logs are not affected by the sysklog shutdown/HUP) > However, if there are too many messages queued, the daemon terminates > after the timeout has expired and discards whatever messages are left > unprocessed. This is done in an effort to keep compatible with sysklogd. > Also, in most scenarios it is highly unlikely that message loss will > occur, because the queue will not hold extended data for an extended > period of time. If, however, this is anticipated, there are ways to > configure rsyslog not to loose messages. For example, the timeout can be > increased and/or the queue can be made disk-assisted and data be written > to disk if they can not be processed during the timeout. > > During queue drain, no new messages are accepted (otherwise, how should > we finally shut down?). Given a busy system, there is unavoidable > message loss. Either you use old messages you already had in the queue, > but have a quick restart and loose few new messages. Or you save the > already existing messages but experience loss of new messages. This is > another reason for the default, which I consider the best compromise > under the circumstances given. This is also a good indication why you > should not unnecessarily restart rsyslgod (or any other syslogd, because > this is not a project-specific problem but rather a general one). this conflicts with the need to rotate the log files. while we have discussed the possibility of using a different signal to allow the logs to be rotated, I was bringing the subject up again becouse I think the HUP behavior is unexpected enough that it may be less surprise if a HUP doesn't re-read config files but does the close/open needed for log rotation than the alturnative of having HUP remain as-in with a different signal for log rotation. I'm not suggesting more work, I'm suggesting that it may make sense to just use HUP for the close/open and if someone needs the full config reload they do a stop/start instead. > With fully reliable protocols, you can of course make the sender block > while you are shutting down. But all of this requires very careful > planning. I assume folks who want to do that either have the necessary > knowledge themselvs or hire someone who does. > > So there are three scenarios: > > a) hup > b) regular shutdown > c) system abort > > I think you did not take into account b), which is a quite common case. there is case d) log rotation when someone does a stop/start (case b) they expect logs in a queue to be lost (you are fully stopping the process after all), I don't think that's what people would expect from a HUP. David Lang > Other than that, I fully agree to you post. > > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From pheller at me.com Mon Oct 20 19:32:51 2008 From: pheller at me.com (Phillip Heller) Date: Mon, 20 Oct 2008 13:32:51 -0400 Subject: [rsyslog] Expression based filters not working? Message-ID: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil From mikel at irontec.com Mon Oct 20 21:54:42 2008 From: mikel at irontec.com (Mikel Jimenez) Date: Mon, 20 Oct 2008 21:54:42 +0200 Subject: [rsyslog] db template how to Message-ID: <48FCE202.1070100@irontec.com> Hello How can I make a template of sql in rsyslog.conf that makes that? : hostname1 >database.hostname1 hostname2 >database.hostname2 hostanem3 >database.hostname3 ..... and hostnameaa hostnameab hostnamedd hostnamede hostnamea*>database.a hostnamed*>database.d Thanks From david at lang.hm Mon Oct 20 22:44:32 2008 From: david at lang.hm (david at lang.hm) Date: Mon, 20 Oct 2008 13:44:32 -0700 (PDT) Subject: [rsyslog] db template how to In-Reply-To: <48FCE202.1070100@irontec.com> References: <48FCE202.1070100@irontec.com> Message-ID: On Mon, 20 Oct 2008, Mikel Jimenez wrote: > Date: Mon, 20 Oct 2008 21:54:42 +0200 > From: Mikel Jimenez > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] db template how to > > Hello > How can I make a template of sql in rsyslog.conf that makes that? : > > hostname1 >database.hostname1 > hostname2 >database.hostname2 > hostanem3 >database.hostname3 > ..... first a disclaimer, I haven't used the database output yet, so I could be wrong or missing something easier that what I'm suggesting. putting them into different databases would be difficult and expensive, but possible. if you are putting them into different tables inside one database you could change from something like "insert into syslog(message) values ('%msg%')" to something like "insert into syslog||'.'||%hostname% (message) values ('%msg%')" since you are issuing SQL commands, you could do something along the lines of the following to do different databases "connect database||'.'||%hostname% ; insert into syslog(message) values ('%msg%'): to switch the the appropriate databases. (note that this is usually _very_ inefficiant to do) I don't know if you can do multiple database output modules going to different systems, but if you can you may be able to do something nicer by defining a different output module for each hostname and use filters to seperate the traffic (this will obviously only work for a limited number of hostnames, and they all have to be known ahead of time) > > and > > hostnameaa > hostnameab > hostnamedd > hostnamede > > hostnamea*>database.a > hostnamed*>database.d this could be similar, but you would have to do string manipulation of the hostname value to extract out just the character you need. David Lang From aoz.syn at gmail.com Mon Oct 20 23:12:23 2008 From: aoz.syn at gmail.com (RB) Date: Mon, 20 Oct 2008 15:12:23 -0600 Subject: [rsyslog] db template how to In-Reply-To: References: <48FCE202.1070100@irontec.com> Message-ID: <4255c2570810201412n2f8addbbqf94b6d78b1d7ccd8@mail.gmail.com> On Mon, Oct 20, 2008 at 14:44, wrote: > putting them into different databases would be difficult and expensive, > but possible. Hopefully we're not talking different databases, per se, maybe different tables? My suggestion wouldn't be to put any of that logic at all in your rsyslog configuration - the application & configurations are complex enough. Rather (if you have a proper database engine) use table partitioning, even constraint-based partitioning to do what you need. Much less horrifying on the client side and any competent DBA will know what you're doing and be able to handle it. Off the top of my head I know Oracle, PostgreSQL, and MySQL support partitioning and I'd be willing to bet MSSQL does as well. Perhaps a more appropriate question would be why? Other than your mental organization and maybe some access-control concerns, why would you want to divide your database by host? Not only does it make global correlation more complex, it's practically guaranteed to have an uneven distribution. When doing log tables, I always arrange the MERGE/partition constraint by date or the like so it's easier and often faster for limiting searches as well as to age data out (DROP TABLE versus DELETE FROM). From DGillies at fairfaxdigital.com.au Tue Oct 21 01:26:20 2008 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 21 Oct 2008 10:26:20 +1100 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> Message-ID: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Hi Phil, I found that ! didn't work for me in my expression style configs. I had to use 'not' instead: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then -?DailyLogs David Gillies Systems Engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller Sent: Tuesday, 21 October 2008 4:33 AM To: rsyslog at lists.adiscon.com Subject: [rsyslog] Expression based filters not working? Hello, Running 3.19.9 with the following old style config bits: :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs This works just fine, though the messages from the ASA are output to the DailyLogs template twice. I have not seen an example how one might use a logical AND with the old style config. So, I've tried the new expression style config: if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg !contains '%ASA-' then -?DailyLogs However, this style seems to not work at all. That is, no messages are recorded to DailyLogs or DailyCDR. Any suggestions? Regards, Phil _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From rgerhards at hq.adiscon.com Tue Oct 21 08:32:57 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 21 Oct 2008 08:32:57 +0200 Subject: [rsyslog] Expression based filters not working? In-Reply-To: <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> References: <9C986C75-8823-467E-9299-8EDA991DC21F@me.com> <4310250BC419AC46BB47F728902B0DD602AD64C0@EXCHDP3.ffx.jfh.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F3E8@grfint2.intern.adiscon.com> That's right ! is NOT part of the language. You need to use the word "not". Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of David Gillies > Sent: Tuesday, October 21, 2008 1:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] Expression based filters not working? > > Hi Phil, > > I found that ! didn't work for me in my expression style configs. I had > to use 'not' instead: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat > if $msg contains 'VOIPAAA' then -?DailyCDR > if not $msg contains '%VOIPAAA' and not $msg contains '%ASA-' then > -?DailyLogs > > David Gillies > Systems Engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Phillip Heller > Sent: Tuesday, 21 October 2008 4:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Expression based filters not working? > > Hello, > > Running 3.19.9 with the following old style config bits: > > :msg, contains, "%ASA-" -?DailyLogs;pix_format :msg, contains, > "%VOIPAAA" -?DailyCDR :msg, !contains, "%VOIPAAA" -?DailyLogs > > This works just fine, though the messages from the ASA are output to > the > DailyLogs template twice. I have not seen an example how one might use > a logical AND with the old style config. > > So, I've tried the new expression style config: > > if $msg contains '%ASA-' then -?DailyLogs;pixformat if $msg contains > 'VOIPAAA' then -?DailyCDR if $msg !contains '%VOIPAAA' and $msg > !contains '%ASA-' then -?DailyLogs > > However, this style seems to not work at all. That is, no messages are > recorded to DailyLogs or DailyCDR. > > Any suggestions? > > Regards, > > Phil > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended recipient, > any use, dissemination, reliance, forwarding, printing or copying of > this e-mail or any attached files is unauthorised. This e-mail is > subject to copyright. No part of it should be reproduced, adapted or > communicated without the written consent of the copyright owner. If you > have received this e-mail in error please advise the sender immediately > by return e-mail or telephone and delete all copies. Fairfax does not > guarantee the accuracy or completeness of any information contained in > this e-mail or attached files. Internet communications are not secure, > therefore Fairfax does not accept legal responsibility for the contents > of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 22 10:13:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 10:13:23 +0200 Subject: [rsyslog] rsyslog 3.21.6 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F414@grfint2.intern.adiscon.com> Hi all, rsyslog 3.21.6, a member of the development branch, has just been released. Most importantly, it fixes a segfault condition which could occur during processing. It also contains some other bugfixes, many imported from the stable and beta branches. Feature-wise, performance has been improved, permitting rsyslogd to handle an even higher number of messages per second. This is a recommended update for all devel branch users. This release is scheduled to become the new beta soon. Changelog: http://www.rsyslog.com/Article292.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-135.phtml As always, feedback is appreciated. Rainer From epiphani at gmail.com Wed Oct 22 18:05:06 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 12:05:06 -0400 Subject: [rsyslog] output compression... Message-ID: I've been looking for some method to automatically compress output files from a syslog service - rsyslogd seems to be the most flexible in terms of base architecture. Has anyone written a zlib-based (or some gzip implementation) output module for writing files to disk already compressed? Would it be something difficult to add to the roadmap? :) Thanks, -Aaron From aoz.syn at gmail.com Wed Oct 22 18:30:38 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 10:30:38 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <4255c2570810220930h3b63c09awbe6d993da0a5270b@mail.gmail.com> On Wed, Oct 22, 2008 at 10:05, Aaron Wiebe wrote: > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) It's on my personal roadmap, but fell behind circular logs (FBSD clog-compatible) and my screwing around with Windows .evt formats. Don't know if anyone else has looked at it. Even though it should be a well-received addition, the problem is that most users do well enough with logrotate or its ilk to manage log compression and don't count it too high on their priorities. From rgerhards at hq.adiscon.com Wed Oct 22 18:37:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Oct 2008 18:37:03 +0200 Subject: [rsyslog] output compression... In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Hi, I have not much to add to RB's remark. I agree to his analysis. I'd like to do it, but always higher priority things come in between... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 22, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] output compression... > > I've been looking for some method to automatically compress output > files from a syslog service - rsyslogd seems to be the most flexible > in terms of base architecture. Has anyone written a zlib-based (or > some gzip implementation) output module for writing files to disk > already compressed? > > Would it be something difficult to add to the roadmap? :) > > Thanks, > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Wed Oct 22 20:55:03 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 22 Oct 2008 14:55:03 -0400 Subject: [rsyslog] output compression... In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: Understood - thanks for the response. Can you point me at some docs for your modules API? I may see what I can do on my own here... Thanks, -Aaron On Wed, Oct 22, 2008 at 12:37 PM, Rainer Gerhards wrote: > Hi, > > I have not much to add to RB's remark. I agree to his analysis. I'd like > to do it, but always higher priority things come in between... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe >> Sent: Wednesday, October 22, 2008 6:05 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] output compression... >> >> I've been looking for some method to automatically compress output >> files from a syslog service - rsyslogd seems to be the most flexible >> in terms of base architecture. Has anyone written a zlib-based (or >> some gzip implementation) output module for writing files to disk >> already compressed? >> >> Would it be something difficult to add to the roadmap? :) >> >> Thanks, >> -Aaron >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 22 21:40:05 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 22 Oct 2008 13:40:05 -0600 Subject: [rsyslog] output compression... In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> Message-ID: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> > Can you point me at some docs for your modules API? I may see what I > can do on my own here... I would suggest starting with plugins/omtesting, plugins/imtemplate and tools/omfile.* - those helped me jump in PDQ. The following URLs were also helpful: http://www.rsyslog.com/doc-generic_design.html http://www.rsyslog.com/doc-modules.html From rgerhards at hq.adiscon.com Thu Oct 23 13:23:10 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 23 Oct 2008 13:23:10 +0200 Subject: [rsyslog] output compression... In-Reply-To: <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> Message-ID: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> On Wed, 2008-10-22 at 13:40 -0600, RB wrote: > > Can you point me at some docs for your modules API? I may see what I > > can do on my own here... > > I would suggest starting with plugins/omtesting, plugins/imtemplate > and tools/omfile.* - those helped me jump in PDQ. The following URLs > were also helpful: > > http://www.rsyslog.com/doc-generic_design.html > http://www.rsyslog.com/doc-modules.html That's a good suggestion. In this case, I would also tend to modifying omfile.c directly. This, too me, sounds much like just adding the ability to zip before doing the write() call. If you intend to contribute back the patch, I can lend you a helping hand (as much as time permits, unfortunately) during the process. Rainer From epiphani at gmail.com Thu Oct 23 17:12:49 2008 From: epiphani at gmail.com (Aaron Wiebe) Date: Thu, 23 Oct 2008 11:12:49 -0400 Subject: [rsyslog] output compression... In-Reply-To: <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F42C@grfint2.intern.adiscon.com> <4255c2570810221240v22b12e52g6a42185ebfe08649@mail.gmail.com> <1224760990.23821.107.camel@rgf9dev.intern.adiscon.com> Message-ID: On Thu, Oct 23, 2008 at 7:23 AM, Rainer Gerhards wrote: > > That's a good suggestion. In this case, I would also tend to modifying > omfile.c directly. This, too me, sounds much like just adding the > ability to zip before doing the write() call. Yes, I've read omfile.c - I will likely be following this route. > If you intend to contribute back the patch, I can lend you a helping > hand (as much as time permits, unfortunately) during the process. Unfortunately that is at the whim of my employer. I have a process started to try to clear such things, but for now I will have to work alone. Thanks, -Aaron From david at lang.hm Fri Oct 24 23:47:54 2008 From: david at lang.hm (david at lang.hm) Date: Fri, 24 Oct 2008 14:47:54 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? Message-ID: I'm constructing custom output formats and it looks like things are not gettng parsed as I would expect. the incoming logs look like 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 143 E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 plug-gw[13051]: disconnect host= /192.168.242.12 destination=179.50.100.52/14872 in=1069 out=71 duration=1 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG daemon.notice, length: 135 E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p plug-gw[10883]: disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 out=720 duration=1 what is unexpected is that tag is the hostname/IP and the plug-gw is part of the message the hostname field is getting populated with what I would expect to be in fromhost (the relay box that sent the message to me) the syslog daemons sending me the logs have been modified, so there is a possibility that I messed up on them and the format that's being sent isn't right, but if so I'm not seeing anything wrong with it. I am using the nextmaster branch. David Lang From rgerhards at hq.adiscon.com Sun Oct 26 20:41:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 26 Oct 2008 20:41:12 +0100 Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Could you get me the debug output while such a message is being processed? That would probably be useful... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 24, 2008 11:48 PM > To: rsyslog-users > Subject: [rsyslog] parsing error with imudp? > > I'm constructing custom output formats and it looks like > things are not > gettng parsed as I would expect. > > the incoming logs look like > > 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 143 > E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 > plug-gw[13051]: disconnect host= /192.168.242.12 > destination=179.50.100.52/14872 in=1069 out=71 duration=1 > > 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG > daemon.notice, length: 135 > E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p > plug-gw[10883]: > disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 > out=720 duration=1 > > > what is unexpected is that tag is the hostname/IP and the > plug-gw is part > of the message > > the hostname field is getting populated with what I would > expect to be in > fromhost (the relay box that sent the message to me) > > the syslog daemons sending me the logs have been modified, so > there is a > possibility that I messed up on them and the format that's being sent > isn't right, but if so I'm not seeing anything wrong with it. > > I am using the nextmaster branch. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Mon Oct 27 01:38:44 2008 From: david at lang.hm (david at lang.hm) Date: Sun, 26 Oct 2008 17:38:44 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, Rainer Gerhards wrote: > Could you get me the debug output while such a message is being > processed? That would probably be useful... I'll have to recompile to get a build with all the debug fetures enabled. I will do this tomorrow. I did check rsyslog->rsyslog relay and saw the same thing (actually it was sysklogd -> rsyslog -> rsyslog. the first rsyslog shows the correct thing, the second rsyslog shows the hostname of the first, with the hostname of the sysklog box in the message) David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Friday, October 24, 2008 11:48 PM >> To: rsyslog-users >> Subject: [rsyslog] parsing error with imudp? >> >> I'm constructing custom output formats and it looks like >> things are not >> gettng parsed as I would expect. >> >> the incoming logs look like >> >> 14:35:37.480815 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 143 >> E....j.. at ..z..............,.<29>Oct 24 14:35:37 179.50.100.86 >> plug-gw[13051]: disconnect host= /192.168.242.12 >> destination=179.50.100.52/14872 in=1069 out=71 duration=1 >> >> 14:35:37.480882 IP 192.168.210.6.32769 > 192.168.210.5.514: SYSLOG >> daemon.notice, length: 135 >> E....k.. at ..................|<29>Oct 24 14:35:37 happy1-p >> plug-gw[10883]: >> disconnect host= /10.201.7.120 destination=192.168.104.31/5667 in=132 >> out=720 duration=1 >> >> >> what is unexpected is that tag is the hostname/IP and the >> plug-gw is part >> of the message >> >> the hostname field is getting populated with what I would >> expect to be in >> fromhost (the relay box that sent the message to me) >> >> the syslog daemons sending me the logs have been modified, so >> there is a >> possibility that I messed up on them and the format that's being sent >> isn't right, but if so I'm not seeing anything wrong with it. >> >> I am using the nextmaster branch. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From luigi.perroti at googlemail.com Tue Oct 28 10:12:36 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 10:12:36 +0100 Subject: [rsyslog] New HUP directive effects on ommysql Message-ID: Hi, maybe the developers would like to add this finding to the ommysql module documentation. I found out that using '$HUPisRestart off' with 3.21.6 has an effect on the ommysql module too. More precisely I noted that changes made to $ActionOmmysqlServerPort become effective. This might be useful if it's needed to change the database port without doing a full configuration reload. Please correct me if you think I got this wrong, thanks. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 14:00:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 14:00:03 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Hi Luigi, could you please double-check your observation? This should not happen, as ommysql does not have any HUP handler. If that happens, we may have a new track to what is the bug in your environment. I will concentrate on a lab with MySQL enabled after I hear back from you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Luigi Perroti > Sent: Tuesday, October 28, 2008 10:13 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] New HUP directive effects on ommysql > > Hi, > > maybe the developers would like to add this finding to the ommysql > module documentation. > > I found out that using '$HUPisRestart off' with 3.21.6 has an effect > on the ommysql module too. > More precisely I noted that changes made to $ActionOmmysqlServerPort > become effective. This might be useful if it's needed to change the > database port without doing a full configuration reload. > > Please correct me if you think I got this wrong, thanks. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From luigi.perroti at googlemail.com Tue Oct 28 14:59:01 2008 From: luigi.perroti at googlemail.com (Luigi Perroti) Date: Tue, 28 Oct 2008 14:59:01 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: Hi, I looked into it more carefully and I get this by checking the configuration file: /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 rsyslogd: version 3.21.6, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: invalid or yet-unknown config file command - have you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 rsyslogd: End of config validation run. Bye. Indeed, line 31 is where I have the $HUPisRestart directive. Even if 3.21.6 is the latest released development version it occurred to me that this feature might only be in the repository at this time. If that's the case then I'm sorry about the noise. Regards, Luigi From rgerhards at hq.adiscon.com Tue Oct 28 15:12:25 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:12:25 +0100 Subject: [rsyslog] New HUP directive effects on ommysql In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4B5@grfint2.intern.adiscon.com> Message-ID: <1225203145.23821.131.camel@rgf9dev.intern.adiscon.com> Oh, sorry, I think we got the trouble spot ;) It is currently available as part of the "nextmaster" branch. I am right now in the process of switching stable/beta/master versions and nextmaster is what will be the new master when this is done. Rainer On Tue, 2008-10-28 at 14:59 +0100, Luigi Perroti wrote: > Hi, > > I looked into it more carefully and I get this by checking the > configuration file: > > /path/to/rsyslogd -c3 -f /etc/rsyslog.conf -N1 > > rsyslogd: version 3.21.6, config validation run (level 1), master > config /etc/rsyslog.conf > > rsyslogd: invalid or yet-unknown config file command - have you > forgotten to load a module? [try http://www.rsyslog.com/e/3003 ] > > rsyslogd: the last error occured in /etc/rsyslog.conf, line 31 > rsyslogd: End of config validation run. Bye. > > Indeed, line 31 is where I have the $HUPisRestart directive. > > Even if 3.21.6 is the latest released development version it occurred > to me that this feature might only be in the repository at this time. > If that's the case then I'm sorry about the noise. > > Regards, > Luigi > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Oct 28 15:55:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 15:55:15 +0100 Subject: [rsyslog] new v3-stable release candidate Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Hi all, I am about to switch the v3-stable release to 3.20, which means it will be based on the 3.19.x beta branch. I have created a test tarball, available at: http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz I would appreciate if some could try it out and report any potential issues they see. I intend to release this tomorrow or Thursday if nothing bad happens. Rainer From david at lang.hm Tue Oct 28 18:26:14 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 10:26:14 -0700 (PDT) Subject: [rsyslog] new v3-stable release candidate In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: On Tue, 28 Oct 2008, Rainer Gerhards wrote: > Hi all, > > I am about to switch the v3-stable release to 3.20, which means it will > be based on the 3.19.x beta branch. I have created a test tarball, > available at: > > http://download.rsyslog.com/rsyslog/rsyslog-3.20.0.tar.gz > > I would appreciate if some could try it out and report any potential > issues they see. I intend to release this tomorrow or Thursday if > nothing bad happens. one problem I have been seeing in the nextmaster branch is that with imudp on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the box rsyslog goes into a loop (eating 100% cpu, but not doing anything) when it recieves a IPv4 syslog packet. Rainer has not been able to duplicate the problem, could someone else test this? I've been seeing it on Debian amd64 boxes. David Lang From rgerhards at hq.adiscon.com Tue Oct 28 18:29:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Oct 2008 18:29:38 +0100 Subject: [rsyslog] new v3-stable release candidate In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4BB@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4BF@grfint2.intern.adiscon.com> > one problem I have been seeing in the nextmaster branch is that with > imudp > on a box that has IPv6 in the kernel, but not IPv6 IP addresses on the > box > rsyslog goes into a loop (eating 100% cpu, but not doing anything) when > it > recieves a IPv4 syslog packet. > > Rainer has not been able to duplicate the problem, could someone else > test > this? That would be really good. But please note that nextmaster will become 3.23.x, not 3.20. The upcoming 3.20 stable release is based on the current beta. I just wanted to make sure we are all talking about the right versions :) Rainer From david at lang.hm Wed Oct 29 02:44:04 2008 From: david at lang.hm (david at lang.hm) Date: Tue, 28 Oct 2008 18:44:04 -0700 (PDT) Subject: [rsyslog] parsing error with imudp? In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F493@grfint2.intern.adiscon.com> Message-ID: On Sun, 26 Oct 2008, david at lang.hm wrote: > On Sun, 26 Oct 2008, Rainer Gerhards wrote: > >> Could you get me the debug output while such a message is being >> processed? That would probably be useful... > > I'll have to recompile to get a build with all the debug fetures enabled. > I will do this tomorrow. 1264.971559438:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1264.971582626:{6000}imudp.c: main queue: entry added, size now 1 entries 1264.971595478:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1264.971611682:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1264.971627886:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1264.971649678:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1264.971664206:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1264.971684042:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1264.971701084:{6000}imudp.c: 1264.971717009:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000049 1264.971747461:{5997}main queue:Reg/w0: Message has legacy syslog format. 1264.971767577:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1264.971784898:{5997}main queue:Reg/w0: (/var/log/messages) 1264.971798867:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1265.971497411:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1265.971509424:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 1265.971527863:{6000}imudp.c: main queue: entry added, size now 1 entries 1265.971538480:{6000}imudp.c: wtpAdviseMaxWorkers signals busy 1265.971554125:{6000}imudp.c: main queue: EnqueueMsg advised worker start 1265.971569491:{5997}main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 1265.971587372:{5997}main queue:Reg/w0: dropped LF at very end of message (DropTrailingLF is set) 1265.971603017:{5997}main queue:Reg/w0: msg parser: flags 30, from '192.168.255.2', msg <13>Sep 18 00:44:18 secse c1 logger: this is a test message 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000050 1265.971617824:{5997}main queue:Reg/w0: Message has legacy syslog format. 1265.971633749:{6000}imudp.c: Listening on UDP syslogd socket 4 (IPv4/port 514). 1265.971647159:{6000}imudp.c: --------imUDP calling select, active file descriptors (max 4): 4 1265.971664760:{5997}main queue:Reg/w0: Called action, logging to builtin-file 1265.971681523:{6000}imudp.c: 1265.971696610:{5997}main queue:Reg/w0: (/var/log/messages) 1265.971712814:{5997}main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 1266.971494055:{6000}imudp.c: Host name for your address (192.168.255.2) unknown 1266.971506068:{6000}imudp.c: recv(4,266)/192.168.255.2,acl:1,msg:<13>Sep 18 00:44:18 secsec1 logger: this is a test message 00000000000000000000 From rgerhards at hq.adiscon.com Wed Oct 29 15:23:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:23:46 +0100 Subject: [rsyslog] regex help wanted Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Hi folks, I am tracking down a problem with a user here in this forum thread: http://kb.monitorware.com/regex-match-for-port-t8764.html#p14411 He claims that rsyslog does not correctly handle POSIX ERE regular expressions, but I now have written a simple program (tester) that works in the same way. I don't think I have set any options wrong. However, I am not at all an regex expert. Maybe some of you could shed some light on this? I would really appreciate some help, as the situation sounds sub-optimal, at least until we have an explanation for it... Thanks, Rainer From aoz.syn at gmail.com Wed Oct 29 15:47:23 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 08:47:23 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> > He claims that rsyslog does not correctly handle POSIX ERE regular > expressions, but I now have written a simple program (tester) that works > in the same way. I don't think I have set any options wrong. However, I > am not at all an regex expert. I've not carefully read the portion of the source where you do your matching; can you point me in a general direction? What library are you using? RB From rgerhards at hq.adiscon.com Wed Oct 29 15:50:20 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 15:50:20 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 3:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > He claims that rsyslog does not correctly handle POSIX ERE regular > > expressions, but I now have written a simple program (tester) that > works > > in the same way. I don't think I have set any options wrong. However, > I > > am not at all an regex expert. > > I've not carefully read the portion of the source where you do your > matching; can you point me in a general direction? What library are > you using? Sure. There is a very simple sample program inside the forum thread. This is a direct link: http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 Well... I'll also include it after my sig. I use the plain old clib regex library. The problem can fully be reproduced with the minimalistic sample, so I think it is better to look at it than at the actual rsyslog source (but most of the regex functionality is in msg.c). Thanks, Rainer #include #include #include #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 (198.39.187.236/54751)" int main() { regex_t preg; size_t nmatch = 10; regmatch_t pmatch[10]; char str[] = STR; int i; i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); printf("regcomp returns %d\n", i); i = regexec(&preg, str, nmatch, pmatch, 0); printf("regexec returns %d\n", i); if(i == REG_NOMATCH) { printf("found no match!\n"); return 1; } printf("returned substrings:\n"); for(i = 0 ; i < 10 ; i++) { printf("%d: so %d, eo %d", i, pmatch[i].rm_so, pmatch[i].rm_eo); if(pmatch[i].rm_so != -1) { int j; printf(", text: '"); for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) putchar(str[j]); putchar('\''); } putchar('\n'); } return 0; } From hks.private at gmail.com Wed Oct 29 16:07:18 2008 From: hks.private at gmail.com ((private) HKS) Date: Wed, 29 Oct 2008 11:07:18 -0400 Subject: [rsyslog] regex help wanted In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: The lazy quantifier (?) is not being respected, so rather than returning the first match of "stuff, a forward slash, more stuff, space", it returns the last one. I know zero C, so I'm afraid I can't be much help in figuring out why. You could try the universal regexp trick: if in doubt, escape it! -HKS On Wed, Oct 29, 2008 at 10:50 AM, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, October 29, 2008 3:47 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] regex help wanted >> >> > He claims that rsyslog does not correctly handle POSIX ERE regular >> > expressions, but I now have written a simple program (tester) that >> works >> > in the same way. I don't think I have set any options wrong. > However, >> I >> > am not at all an regex expert. >> >> I've not carefully read the portion of the source where you do your >> matching; can you point me in a general direction? What library are >> you using? > > Sure. There is a very simple sample program inside the forum thread. > This is a direct link: > > http://kb.monitorware.com/regex-match-for-port-t8764-15.html#p14423 > > Well... I'll also include it after my sig. I use the plain old clib > regex library. The problem can fully be reproduced with the minimalistic > sample, so I think it is better to look at it than at the actual rsyslog > source (but most of the regex functionality is in msg.c). > > Thanks, > Rainer > > #include > #include > #include > > #define STR "%ASA-6-302015: Built outbound UDP connection 25503427 > for outside:198.14.210.2/53 (198.14.210.2/53) to inside:12.66.8.80/61594 > (198.39.187.236/54751)" > int main() > { > regex_t preg; > size_t nmatch = 10; > regmatch_t pmatch[10]; > char str[] = STR; > int i; > > i = regcomp(&preg, "outside:.+?/(.+?)\\s", REG_EXTENDED); > printf("regcomp returns %d\n", i); > i = regexec(&preg, str, nmatch, pmatch, 0); > printf("regexec returns %d\n", i); > if(i == REG_NOMATCH) { > printf("found no match!\n"); > return 1; > } > > printf("returned substrings:\n"); > for(i = 0 ; i < 10 ; i++) { > printf("%d: so %d, eo %d", i, pmatch[i].rm_so, > pmatch[i].rm_eo); > if(pmatch[i].rm_so != -1) { > int j; > printf(", text: '"); > for(j = pmatch[i].rm_so ; j < pmatch[i].rm_eo ; ++j) > putchar(str[j]); > putchar('\''); > } > putchar('\n'); > } > return 0; > } > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From aoz.syn at gmail.com Wed Oct 29 16:32:39 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 29 Oct 2008 09:32:39 -0600 Subject: [rsyslog] regex help wanted In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com> <4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> Message-ID: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> > The lazy quantifier (?) is not being respected, so rather than > returning the first match of "stuff, a forward slash, more stuff, > space", it returns the last one. Correct, because the lazy quantifier is not POSIX and therefore does not work when using POSIX extended REs. I was wondering why the extra '?', but had forgotten about lazy notation, preferring to make my regexes a bit more strict. RB From rgerhards at hq.adiscon.com Wed Oct 29 16:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 29 Oct 2008 16:34:52 +0100 Subject: [rsyslog] regex help wanted In-Reply-To: <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> References: <577465F99B41C842AAFBE9ED71E70ABA44F4D1@grfint2.intern.adiscon.com><4255c2570810290747p24ef95e0v16d3195a9cb2aec2@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA44F4D4@grfint2.intern.adiscon.com> <4255c2570810290832s14b9568ar6fac06e3b2b0de01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA44F4DA@grfint2.intern.adiscon.com> Ah, thanks a lot to both of you. This greatly helps and makes me feel much better. Of course, we can add additional regex libs, but it has been proven that the current implementation works OK. Thanks again, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, October 29, 2008 4:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] regex help wanted > > > The lazy quantifier (?) is not being respected, so rather than > > returning the first match of "stuff, a forward slash, more stuff, > > space", it returns the last one. > > Correct, because the lazy quantifier is not POSIX and therefore does > not work when using POSIX extended REs. I was wondering why the extra > '?', but had forgotten about lazy notation, preferring to make my > regexes a bit more strict. > > > RB > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com