From tbergfeld at hq.adiscon.com Fri Feb 5 15:41:32 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 5 Feb 2010 15:41:32 +0100 Subject: [rsyslog] rsyslog 5.5.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103822@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.2, a member of the v5-devel branch. Rsyslog 5.5.2 primarily provides bug-fixes, but offers an enhanced functionality to escape 8-bit characters on reception. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-193.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From ryan.b.lynch at gmail.com Fri Feb 5 23:07:26 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 5 Feb 2010 17:07:26 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Under 5.3.6, I am trying to use property-based filtering to separate local messages (coming in via imuxsock, presumably) from remote messages (coming in via UDP). Initially, I tried using the 'inputname' property to distinguish between the two: :inputname, isequal, "imudp" /remote_udp.log :inputname, isequal, "imuxsock" /local_socket.log With this config, nothing ever ends up in the 2nd file (/local_socket.log)--instead, because the 'inputname' property is always set to 'imudp', regardless of where the input originates. The property replacer docs mention that the value of 'inputname' isn't guaranteed, so I guess this isn't necessarily a bug. Also, I found a comment in the source code for 'imuxsock.c' that makes this behavior seem to be intentional. Why does only imuxsock behave like this? The other input modules I've used (imrelp, imklog) provide the expected 'inputname' values ('imrelp' and 'imklog', respectively), which is intuitive and consistent. Is there a reason why imuxsock needs to be different, or is this a bug? (In case anyone is wondering, I could work around this by comparing the 'hostname' and '$myhostname' properties, in addition to checking 'inputname'. But that would force me to use expression-based filters (property-based filtering doesn't support AND/OR logic), which seems to increase the CPU activity per message. I really can't afford the lost efficiency, here, if it can be helped.) Ryan B. Lynch ryan.b.lynch at gmail.com From ryan.b.lynch at gmail.com Sat Feb 6 06:13:17 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 00:13:17 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. Message-ID: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> I'm trying to use multiple rulesets and omruleset actions to send a single incoming log message to two different log files, at the same time. I think my config file will help explain it: # BEGIN rsyslog.conf $ModLoad imuxsock $ModLoad imklog $ModLoad immark $ModLoad omruleset $MarkMessagePeriod 1 $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% %timereported:::date-rfc3339% %hostname% %fromhost% %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% %msg%\n" # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', each of which writes output to its own, separate file ('RS_Alfa.log' and 'RS_Bravo.log', respectively) using an action with a *.* selector. $Ruleset RS_Bravo $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset $Ruleset RS_Alfa $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset # SECOND: I create a pair of actions (in the default ruleset), both with *.* selectors and the action ':omruleset:', but preceded by different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and 'RS_Bravo', respectively $ActionOmrulesetRulesetName RS_Bravo $ActionResumeInterval 1 *.* :omruleset: $ActionOmrulesetRulesetName RS_Alfa $ActionResumeInterval 1 *.* :omruleset: # END rsyslog.conf Based on the documentation, I believe that incoming log messages should be processed by the default ruleset, which sends the message to both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I should see a copy of each log message written to each of the two output files, and each output file should have identical contents. But that isn't the result I'm getting, here. I get two copies of each message, but almost all of them are written to the file 'RS_Alfa'. Only the occasional lone random message is written to 'RS_Bravo'. (If I reverse the order of the :omrulset: actions, I see the oppositie pattern.) The exact proportion varies, but only about 1 in 20 messages makes it to the 1st (Bravo) output file. All different types of messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' file--again, it seems completely random to me. I never get more total messages logged than I should, though. If I run rsyslog long enough to let 10 incoming messages appear, I will see a total of 20 messages when I combine the contents of 'RS_Alfa' and 'RS_Bravo'--they just end up in the wrong files. I turned on debug logging during a short session and captured the results to a file, which I've attached, along with the rsyslog output files from that session. The host in question runs Centos Linux 5.4 as an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. If there is any more info I can provide, or any other debugging steps I should take, please let me know. Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Feb 6 12:00:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:00:59 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> I will check, but the most probably explanation is that it is a bug that so far nobody noticed ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Friday, February 05, 2010 11:07 PM > To: rsyslog-users > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > as'imudp'? > > Under 5.3.6, I am trying to use property-based filtering to separate > local messages (coming in via imuxsock, presumably) from remote > messages (coming in via UDP). Initially, I tried using the 'inputname' > property to distinguish between the two: > > :inputname, isequal, "imudp" /remote_udp.log > :inputname, isequal, "imuxsock" /local_socket.log > > With this config, nothing ever ends up in the 2nd file > (/local_socket.log)--instead, because the 'inputname' property is > always set to 'imudp', regardless of where the input originates. > > The property replacer docs mention that the value of 'inputname' isn't > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > comment in the source code for 'imuxsock.c' that makes this behavior > seem to be intentional. > > Why does only imuxsock behave like this? The other input modules I've > used (imrelp, imklog) provide the expected 'inputname' values > ('imrelp' and 'imklog', respectively), which is intuitive and > consistent. Is there a reason why imuxsock needs to be different, or > is this a bug? > > (In case anyone is wondering, I could work around this by comparing > the 'hostname' and '$myhostname' properties, in addition to checking > 'inputname'. But that would force me to use expression-based filters > (property-based filtering doesn't support AND/OR logic), which seems > to increase the CPU activity per message. I really can't afford the > lost efficiency, here, if it can be helped.) > > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sat Feb 6 12:04:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:04:18 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Indeed, it was a bug. Patch here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 d5305b1f6cc0fe22 Thanks for reporting the issue! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Saturday, February 06, 2010 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > propertyas'imudp'? > > I will check, but the most probably explanation is that it is a bug > that so > far nobody noticed ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Friday, February 05, 2010 11:07 PM > > To: rsyslog-users > > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > > as'imudp'? > > > > Under 5.3.6, I am trying to use property-based filtering to separate > > local messages (coming in via imuxsock, presumably) from remote > > messages (coming in via UDP). Initially, I tried using the > 'inputname' > > property to distinguish between the two: > > > > :inputname, isequal, "imudp" /remote_udp.log > > :inputname, isequal, "imuxsock" /local_socket.log > > > > With this config, nothing ever ends up in the 2nd file > > (/local_socket.log)--instead, because the 'inputname' property is > > always set to 'imudp', regardless of where the input originates. > > > > The property replacer docs mention that the value of 'inputname' > isn't > > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > > comment in the source code for 'imuxsock.c' that makes this behavior > > seem to be intentional. > > > > Why does only imuxsock behave like this? The other input modules I've > > used (imrelp, imklog) provide the expected 'inputname' values > > ('imrelp' and 'imklog', respectively), which is intuitive and > > consistent. Is there a reason why imuxsock needs to be different, or > > is this a bug? > > > > (In case anyone is wondering, I could work around this by comparing > > the 'hostname' and '$myhostname' properties, in addition to checking > > 'inputname'. But that would force me to use expression-based filters > > (property-based filtering doesn't support AND/OR logic), which seems > > to increase the CPU activity per message. I really can't afford the > > lost efficiency, here, if it can be helped.) > > > > Ryan B. Lynch > > ryan.b.lynch at gmail.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From ryan.b.lynch at gmail.com Sat Feb 6 19:27:52 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 13:27:52 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> No problem, and thank you for the patch. I'll try it out and let you know how it works. This is a huge help to me, BTW: I should be able to get my log server working properly, now, without having to worry about omruleset and multiple rulesets. (Hence my other problem report from last night.) Big sigh of relief. Not that I don't like omruleset. Like RELP, it feels really elegant, and it's easy to expressive complex log routing arrangements. I think it's one of your best ideas, to date, and I'm looking forward to trying out some crazy ideas with it. -Ryan On 2010-02-06, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 > > Thanks for reporting the issue! > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Saturday, February 06, 2010 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' >> propertyas'imudp'? >> >> I will check, but the most probably explanation is that it is a bug >> that so >> far nobody noticed ;) >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch >> > Sent: Friday, February 05, 2010 11:07 PM >> > To: rsyslog-users >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' property >> > as'imudp'? >> > >> > Under 5.3.6, I am trying to use property-based filtering to separate >> > local messages (coming in via imuxsock, presumably) from remote >> > messages (coming in via UDP). Initially, I tried using the >> 'inputname' >> > property to distinguish between the two: >> > >> > :inputname, isequal, "imudp" /remote_udp.log >> > :inputname, isequal, "imuxsock" /local_socket.log >> > >> > With this config, nothing ever ends up in the 2nd file >> > (/local_socket.log)--instead, because the 'inputname' property is >> > always set to 'imudp', regardless of where the input originates. >> > >> > The property replacer docs mention that the value of 'inputname' >> isn't >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found a >> > comment in the source code for 'imuxsock.c' that makes this behavior >> > seem to be intentional. >> > >> > Why does only imuxsock behave like this? The other input modules I've >> > used (imrelp, imklog) provide the expected 'inputname' values >> > ('imrelp' and 'imklog', respectively), which is intuitive and >> > consistent. Is there a reason why imuxsock needs to be different, or >> > is this a bug? >> > >> > (In case anyone is wondering, I could work around this by comparing >> > the 'hostname' and '$myhostname' properties, in addition to checking >> > 'inputname'. But that would force me to use expression-based filters >> > (property-based filtering doesn't support AND/OR logic), which seems >> > to increase the CPU activity per message. I really can't afford the >> > lost efficiency, here, if it can be helped.) >> > >> > Ryan B. Lynch >> > ryan.b.lynch at gmail.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Mon Feb 8 09:06:58 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 09:06:58 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103828@GRFEXC.intern.adiscon.com> Ryan, thanks for the feedback. I'll also try to look into the omruleset issue you reported. Looked too complex for the weekend ;) More later... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, February 06, 2010 7:28 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > property as'imudp'? > > No problem, and thank you for the patch. I'll try it out and let you > know how it works. > > This is a huge help to me, BTW: I should be able to get my log server > working properly, now, without having to worry about omruleset and > multiple rulesets. (Hence my other problem report from last night.) > Big sigh of relief. > > Not that I don't like omruleset. Like RELP, it feels really elegant, > and it's easy to expressive complex log routing arrangements. I think > it's one of your best ideas, to date, and I'm looking forward to > trying out some crazy ideas with it. > > -Ryan > > > On 2010-02-06, Rainer Gerhards wrote: > > Indeed, it was a bug. Patch here: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d0 > 54dac0 > > d5305b1f6cc0fe22 > > > > Thanks for reporting the issue! > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Saturday, February 06, 2010 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > >> propertyas'imudp'? > >> > >> I will check, but the most probably explanation is that it is a bug > >> that so > >> far nobody noticed ;) > >> > >> Rainer > >> > >> > -----Original Message----- > >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > >> > Sent: Friday, February 05, 2010 11:07 PM > >> > To: rsyslog-users > >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' > property > >> > as'imudp'? > >> > > >> > Under 5.3.6, I am trying to use property-based filtering to > separate > >> > local messages (coming in via imuxsock, presumably) from remote > >> > messages (coming in via UDP). Initially, I tried using the > >> 'inputname' > >> > property to distinguish between the two: > >> > > >> > :inputname, isequal, "imudp" /remote_udp.log > >> > :inputname, isequal, "imuxsock" /local_socket.log > >> > > >> > With this config, nothing ever ends up in the 2nd file > >> > (/local_socket.log)--instead, because the 'inputname' property is > >> > always set to 'imudp', regardless of where the input originates. > >> > > >> > The property replacer docs mention that the value of 'inputname' > >> isn't > >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found > a > >> > comment in the source code for 'imuxsock.c' that makes this > behavior > >> > seem to be intentional. > >> > > >> > Why does only imuxsock behave like this? The other input modules > I've > >> > used (imrelp, imklog) provide the expected 'inputname' values > >> > ('imrelp' and 'imklog', respectively), which is intuitive and > >> > consistent. Is there a reason why imuxsock needs to be different, > or > >> > is this a bug? > >> > > >> > (In case anyone is wondering, I could work around this by > comparing > >> > the 'hostname' and '$myhostname' properties, in addition to > checking > >> > 'inputname'. But that would force me to use expression-based > filters > >> > (property-based filtering doesn't support AND/OR logic), which > seems > >> > to increase the CPU activity per message. I really can't afford > the > >> > lost efficiency, here, if it can be helped.) > >> > > >> > Ryan B. Lynch > >> > ryan.b.lynch at gmail.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 10:03:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 10:03:14 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265619794.5427.490.camel@localhost> Hi Ryan, once again thanks for the bug report. I could confirm the problem and opend a bug tracker (it may make sense to subscribe to it): http://bugzilla.adiscon.com/show_bug.cgi?id=179 I am now digging into the details. Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From sergei at bslos.com Mon Feb 8 13:18:41 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 17:18:41 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 Message-ID: <201002081718.41670.sergei@bslos.com> Hi, I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit system): $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ --enable-omprog ... checking whether the compiler provides atomic builtins... no checking whether the compiler provides atomic builtins for 64 bit data types... no ... Configure script run without errors. $ make ... CC rsyslogd-omfile.o In file included from omfile.c:67: ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not available, using nul operations - rsyslogd will probably be racy!" omfile.c: In function 'getClockFileAccess': omfile.c:95: warning: implicit declaration of function 'ATOMIC_INC_AND_FETCH' CC rsyslogd-omdiscard.o CC rsyslogd-pmrfc5424.o CC rsyslogd-pmrfc3164.o CC rsyslogd-iminternal.o CC rsyslogd-pidfile.o pidfile.c: In function 'read_pid': pidfile.c:58: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result pidfile.c: In function 'write_pid': pidfile.c:116: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result CCLD rsyslogd rsyslogd-omfile.o: In function `getClockFileAccess': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function `SetGlobalInputTermination': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/glbl.c:136: undefined reference to `ATOMIC_STORE_1_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function `msgDestruct': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/msg.c:802: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function `wtiSetState': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:109: undefined reference to `ATOMIC_STORE_1_TO_INT' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:111: undefined reference to `ATOMIC_STORE_0_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function `DoDeleteBatchFromQStore': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: more undefined references to `ATOMIC_SUB' follow collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' make: *** [all] Error 2 Any help is appreciated. -- Regards, Sergei Butakov From rgerhards at hq.adiscon.com Mon Feb 8 13:53:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 13:53:55 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> this looks like you have generated code for a machine which does not support atomic instructions. rsyslog needs them to work correctly. I suggest generating code for i686. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > Sent: Monday, February 08, 2010 1:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > Hi, > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > system): > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ > --enable-omprog > ... > checking whether the compiler provides atomic builtins... no > checking whether the compiler provides atomic builtins for 64 bit data > types... no > ... > > Configure script run without errors. > > > > $ make > ... > CC rsyslogd-omfile.o > In file included from omfile.c:67: > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > available, > using nul operations - rsyslogd will probably be racy!" > omfile.c: In function 'getClockFileAccess': > omfile.c:95: warning: implicit declaration of function > 'ATOMIC_INC_AND_FETCH' > CC rsyslogd-omdiscard.o > CC rsyslogd-pmrfc5424.o > CC rsyslogd-pmrfc3164.o > CC rsyslogd-iminternal.o > CC rsyslogd-pidfile.o > pidfile.c: In function 'read_pid': > pidfile.c:58: warning: ignoring return value of 'fscanf', declared with > attribute warn_unused_result > pidfile.c: In function 'write_pid': > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > with > attribute warn_unused_result > CCLD rsyslogd > rsyslogd-omfile.o: In function `getClockFileAccess': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > `SetGlobalInputTermination': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/glbl.c:136: > undefined reference to `ATOMIC_STORE_1_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > `msgDestruct': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/msg.c:802: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > `wtiSetState': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:109: > undefined reference to `ATOMIC_STORE_1_TO_INT' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:111: > undefined reference to `ATOMIC_STORE_0_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > `DoDeleteBatchFromQStore': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > ../runtime/.libs/librsyslog.a(librsyslog_la- > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > more undefined references to `ATOMIC_SUB' follow > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > make: *** [all] Error 2 > > > > Any help is appreciated. > > > -- > Regards, > Sergei Butakov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 14:00:20 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 14:00:20 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265634020.5427.491.camel@localhost> Ryan, thanks again for the bug report. I managed to fix it. Details in the tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=179 Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From rgerhards at hq.adiscon.com Mon Feb 8 14:10:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 14:10:54 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> see also at the bottom of this page: http://www.rsyslog.com/doc-build_from_repo.html Raienr > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, February 08, 2010 1:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] can't compile rsyslog 5.3.7 > > this looks like you have generated code for a machine which does not > support > atomic instructions. rsyslog needs them to work correctly. I suggest > generating code for i686. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > > Sent: Monday, February 08, 2010 1:19 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > > > Hi, > > > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > > system): > > > > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile > \ > > --enable-omprog > > ... > > checking whether the compiler provides atomic builtins... no > > checking whether the compiler provides atomic builtins for 64 bit > data > > types... no > > ... > > > > Configure script run without errors. > > > > > > > > $ make > > ... > > CC rsyslogd-omfile.o > > In file included from omfile.c:67: > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > > available, > > using nul operations - rsyslogd will probably be racy!" > > omfile.c: In function 'getClockFileAccess': > > omfile.c:95: warning: implicit declaration of function > > 'ATOMIC_INC_AND_FETCH' > > CC rsyslogd-omdiscard.o > > CC rsyslogd-pmrfc5424.o > > CC rsyslogd-pmrfc3164.o > > CC rsyslogd-iminternal.o > > CC rsyslogd-pidfile.o > > pidfile.c: In function 'read_pid': > > pidfile.c:58: warning: ignoring return value of 'fscanf', declared > with > > attribute warn_unused_result > > pidfile.c: In function 'write_pid': > > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > > with > > attribute warn_unused_result > > CCLD rsyslogd > > rsyslogd-omfile.o: In function `getClockFileAccess': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > > `SetGlobalInputTermination': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/glbl.c:136: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > > `msgDestruct': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/msg.c:802: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > > `wtiSetState': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:109: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:111: > > undefined reference to `ATOMIC_STORE_0_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > > `DoDeleteBatchFromQStore': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > ../runtime/.libs/librsyslog.a(librsyslog_la- > > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > more undefined references to `ATOMIC_SUB' follow > > collect2: ld returned 1 exit status > > make[2]: *** [rsyslogd] Error 1 > > make[2]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > > make: *** [all] Error 2 > > > > > > > > Any help is appreciated. > > > > > > -- > > Regards, > > Sergei Butakov > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From sergei at bslos.com Mon Feb 8 15:11:32 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 19:11:32 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> Message-ID: <201002081911.33210.sergei@bslos.com> On Monday 08 February 2010, Rainer Gerhards wrote: > see also at the bottom of this page: > > http://www.rsyslog.com/doc-build_from_repo.html Thank you! The adding CFLAGS="-march=i686" to the configure script solved my trouble. -- Regards, Sergei Butakov From ryan.b.lynch at gmail.com Mon Feb 8 23:10:35 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:10:35 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Message-ID: <115906d11002081410v17596b5eg6a1fab390aad04d6@mail.gmail.com> On Sat, Feb 6, 2010 at 06:04, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 Works for me with 5.3.7, with different line alignment in the patch. That might be due to 'rpmbuild' and its built-in patch macros. In case it saves anyone else time, I've attached a copy of what works for me. -Ryan From ryan.b.lynch at gmail.com Mon Feb 8 23:16:25 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:16:25 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <1265634020.5427.491.camel@localhost> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> Message-ID: <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards wrote: > Ryan, > > thanks again for the bug report. I managed to fix it. Details in the > tracker: > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 Works fine for me with 5.3.7. Thanks! From rgerhards at hq.adiscon.com Tue Feb 9 07:43:48 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 9 Feb 2010 07:43:48 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710383F@GRFEXC.intern.adiscon.com> Thanks for the feedback. I think I can do the "official" merge today :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Monday, February 08, 2010 11:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] Possible bug in 5.3.6: Multiple > $Ruleset,:omruleset cause random output destination selection. > > On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards > wrote: > > Ryan, > > > > thanks again for the bug report. I managed to fix it. Details in the > > tracker: > > > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 > > Works fine for me with 5.3.7. Thanks! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikolaj at kucharski.name Wed Feb 10 13:02:30 2010 From: mikolaj at kucharski.name (Mikolaj Kucharski) Date: Wed, 10 Feb 2010 12:02:30 +0000 Subject: [rsyslog] last message repeated n times from remote hosts Message-ID: <20100210120230.GI3838@x40.openbsd.home.lan> Hi, I have few Linux based machines with sysklogd installed, and I have central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5. I have issue with missing hostname when sysklog sends "last message repeated N times" over the wire to rsyslog server. Let's have a look. I used logger(1) to repeatedly sent one message few times followed by one different message. Here is what I see in the log file on my central rsyslog server: Feb 10 11:39:46 10.101.43.124 root: remote test start Feb 10 11:39:54 last message repeated 14 times Feb 10 11:39:54 10.101.43.124 root: remote test end and here is tcpdump(8) log from the source (10.101.43.124) machine: 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28 0x0000: 4500 0038 0000 4000 4011 d1fb 0a65 2b7c E..8.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0024 68ef 3c31 333e .e(t.....$h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 7374 6172 740a t.start. 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35 0x0000: 4500 003f 0000 4000 4011 d1f4 0a65 2b7c E..?.. at .@....e+| 0x0010: 0a65 2874 0202 0202 002b 68f6 3c31 333e .e(t.....+h.<13> 0x0020: 6c61 7374 206d 6573 7361 6765 2072 6570 last.message.rep 0x0030: 6561 7465 6420 3134 2074 696d 6573 0a eated.14.times. 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26 0x0000: 4500 0036 0000 4000 4011 d1fd 0a65 2b7c E..6.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0022 68ed 3c31 333e .e(t....."h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 656e 640a t.end. I searched the list, and saw a comment which say the fault is on the sysklogd end as it never sends hostname in the repeated-n-times packet, but from above I cannot see that it ever sends the packet with hostname, so I think the issues is on rsyslog side, and not on the sysklogd. Could someone shed some light on my issue, as I would like to see all the time the source IP or hostname of incomming messages to rsyslog daemon. Is this missing source hostname/IP a bug of rsyslog? Is there any way to workaround that? Thanks. PS1. I cannot change client machines, I cannot reinstall them with different syslog implementation, the only machine where I have permission to do modifications is central rsyslog server. PS2. I know about DNS and RevDNS and yes, above server doesn't have revDNS setup. -- best regards q# From tbergfeld at hq.adiscon.com Wed Feb 10 15:31:17 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 10 Feb 2010 15:31:17 +0100 Subject: [rsyslog] rsyslog 4.5.8 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710385E@GRFEXC.intern.adiscon.com> Hi all, We have just release rsyslog 4.5.8, a member of the v4-beta branch. This is a bug-fixing release, fixing all those issues that have been reported the past weeks. Full details are available in the change log. This is scheduled to replace the current v4-stable, and conclude the v4 development efforts. So we would appreciate if you could give it a try and report any issues you see with it. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-194.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Feb 16 10:13:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 16 Feb 2010 10:13:14 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Hi folks, almost by accident, I learned today that rsyslog seems to be the default syslogd for SuSe as well: http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 I am a bit surprised I never heared about that change, but I have to admit I am quite happy that rsyslog now seems to be the default syslogd on (almost) all important distros... I just thought I share the good news. Rainer From mbiebl at gmail.com Tue Feb 16 15:29:57 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 16 Feb 2010 15:29:57 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/16 Rainer Gerhards : > Hi folks, > > almost by accident, I learned today that rsyslog seems to be the default > syslogd for SuSe as well: > > http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 > > I am a bit surprised I never heared about that change, but I have to admit I > am quite happy that rsyslog now seems to be the default syslogd on (almost) > all important distros... > > I just thought I share the good news. Congrats Rainer for your hard work and persistence! Let's welcome our openSUSE friends on board :-) Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Sun Feb 21 20:31:40 2010 From: reader at newsguy.com (Harry Putnam) Date: Sun, 21 Feb 2010 13:31:40 -0600 Subject: [rsyslog] [basic] No remote logs getting thru Message-ID: <87iq9qmllv.fsf@newsguy.com> Not sure if the localmachine is not sending or the remote not receiving. But it appears the logs expected to show up on the remote are not there. Both ends are gentoo linux, I'm not at all familiar with rsyslog but hoped I set it up right according to what I see on the example file and at: http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications Local and remote configs appear in full [inlined] at the end. Just a few lines (including commented ones) from local conf that I hoped were what I needed to send the logs: # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host The above last 2 commented lines would be less confusing of they used the same notation for remote host in both instances. remote host remote-host Appears to be talking about two different things *.* @@remote-host So is that really all it needs? ------- --------- ---=--- --------- -------- Now a few lines from the remote rsyslog.conf # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. That is all it appears to indicate is needed to receive logs. And this (the remote) instance of rsyslog is being started with -c3 -t514 flags. ------- --------- ---=--- --------- -------- Complete LOCAL rsyslog conf: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional # remote host is: # logsrv:512 # e.g. 192.168.0.1:514, port optional *.* @@192.168.0.26:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 ------- --------- ---=--- --------- -------- Complete remote rsyslog.conf $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 From mrdemeanour at jackpot.uk.net Mon Feb 22 07:27:28 2010 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 22 Feb 2010 06:27:28 +0000 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87iq9qmllv.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <4B8223D0.8040503@jackpot.uk.net> Harry Putnam wrote: > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. Hi, You seem to have all of the output ('action') configuration on the client machine - that's all the stuff starting with *.*, local1.* etc. As a consequence only the client machine is ever going to output anything. Please verify that the client machine is indeed logging to disk in /var/log. If you now set up a catch-all action line in the server's rsyslog.conf (and restart the service on both machines), I prophesy that output originating from the client will start appearing on the server. E.g.: *.* /var/log/catchall.log You can test using the logger command on the client, e.g. $ logger -p local1.info "Test log message from client" -- Jack. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 22 08:40:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 08:40:49 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Harry, I think this may contain useful information for you: http://cookbook.rsyslog.com/node13.html there is also a guide on receiving messages and putting them to different bins. Looking briefly at your config, I'd suggest that you use explicit port numbers (e.g. 10514) and make sure they are enabled in our firewall. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Sunday, February 21, 2010 8:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [basic] No remote logs getting thru > > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_-- > _starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support > it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Mon Feb 22 17:25:02 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 10:25:02 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Message-ID: <87wry5kzkx.fsf@newsguy.com> "Rainer Gerhards" writes: > Harry, > > I think this may contain useful information for you: > > http://cookbook.rsyslog.com/node13.html From:http://cookbook.rsyslog.com/node14.html *.* @@192.0.2.1:10514 # if you need to forward to other systems as well, just # add additional config lines: *.* @@other-server.example.net:10514 I must be missing something... (I am a novice with this) but I don't see anything different in the suggested config from what I've already posted in the client config. Except it shows a second remote server being sent logs... which I'm not trying to do. Also what is needed in the server to make rsyslog listen to port 514? As I've mentioned, that (server) instance of rsyslog is being started with -t514. I've seen nothing so far either in the stock rsyslog.conf example configs or at: http://cookbook.rsyslog.com/node13.html to indicate some better or more precise way to get the server to listen on 514 On the server: `ps wwaux' shows how the server is running: ps wwaux|grep rsyslog (wrapped for mail) root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ /var/run/rsyslogd.pid -f /etc/rsyslog.conf There are no firewalls involved here... its a local network behind a firewall router so all rsyslog traffic is only on the local side. I'm not sure if it means anything but trying: telnet server_host 514 from the client machine does fail to connect. And the attempt does not show up in the catchall ouput on server either. From rgerhards at hq.adiscon.com Mon Feb 22 18:51:36 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 18:51:36 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Monday, February 22, 2010 5:25 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] [basic] No remote logs getting thru > > "Rainer Gerhards" writes: > > > Harry, > > > > I think this may contain useful information for you: > > > > http://cookbook.rsyslog.com/node13.html > > From:http://cookbook.rsyslog.com/node14.html > > *.* @@192.0.2.1:10514 > # if you need to forward to other systems as well, just > # add additional config lines: > *.* @@other-server.example.net:10514 > > I must be missing something... (I am a novice with this) but I don't > see anything different in the suggested config from what I've already > posted in the client config. Except it shows a second remote server > being sent logs... which I'm not trying to do. > > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. > > I've seen nothing so far either in the stock rsyslog.conf example > configs or at: http://cookbook.rsyslog.com/node13.html to indicate > some better or more precise way to get the server to listen on 514 see http://cookbook.rsyslog.com/node24.html and http://www.rsyslog.com/doc-imtcp.html Rainer > > On the server: > > `ps wwaux' shows how the server is running: > > ps wwaux|grep rsyslog (wrapped for mail) > > root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ > 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ > /var/run/rsyslogd.pid -f /etc/rsyslog.conf > > There are no firewalls involved here... its a local network behind a > firewall router so all rsyslog traffic is only on the local side. > > I'm not sure if it means anything but trying: > telnet server_host 514 > > from the client machine does fail to connect. And the attempt does > not show up in the catchall ouput on server either. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Mon Feb 22 18:46:26 2010 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 22 Feb 2010 10:46:26 -0700 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87wry5kzkx.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. You loaded the module but I don't see anything explicitly starting inputTCP (not sure if this is required or not). In my config I have: $ModLoad imtcp.so # load module $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 I also have: # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 Generally - there's not much harm running both afaik and it makes it easier when you have a device that doesn't do TCP (network gear). > I'm not sure if it means anything but trying: > ?telnet server_host 514 > > from the client machine does fail to connect. ?And the attempt does > not show up in the catchall ouput on server either. Yeah, that's. bad. Try the above, and if you still cannot connect remotely run this on the syslog server: sudo netstat -alnp | grep 514 You may get some other stuff but you should get a line that looks like this: tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 29233/rsyslogd That tells you rsyslogd is listening on port 514 - if you did all of the above you should also see: udp 0 0 0.0.0.0:514 0.0.0.0:* 29233/rsyslogd Which tells you it's listening on udp 514. Hope this helps, Aaron From reader at newsguy.com Mon Feb 22 21:41:42 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 14:41:42 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> Message-ID: <87k4u5j94p.fsf@newsguy.com> "Rainer Gerhards" writes: [...] >> I must be missing something... (I am a novice with this) but I don't >> see anything different in the suggested config from what I've already >> posted in the client config. Except it shows a second remote server >> being sent logs... which I'm not trying to do. [...] >> I've seen nothing so far either in the stock rsyslog.conf example >> configs or at: http://cookbook.rsyslog.com/node13.html to indicate >> some better or more precise way to get the server to listen on 514 > > see http://cookbook.rsyslog.com/node24.html > and http://www.rsyslog.com/doc-imtcp.html Thanks for your patience Rainer... ok I get it now. But see comment at the end. Oh, and what about that dashed notation in the example file like: *.info;mail.none;authpriv.none;cron.none -/var/log/messages What role does the dash play? [...] Aaron Nichols writes: > On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: >> Also what is needed in the server to make rsyslog listen to port 514? >> As I've mentioned, that (server) instance of rsyslog is being started >> with -t514. > > You loaded the module but I don't see anything explicitly starting > inputTCP (not sure if this is required or not). In my config I have: > > $ModLoad imtcp.so # load module > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 Yup, those were right on the money, and of course I do see the info at `netstat -alnp | grep 514' you talked about... its listening now, and with a restart at both ends... logs info is now comming across. Many thanks... you input proved to be right on the money. Oh, do you know if there are many things that can only log by udp? From reader at newsguy.com Tue Feb 23 01:33:05 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 18:33:05 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> Message-ID: <87fx4skczi.fsf@newsguy.com> Harry Putnam writes: > Thanks for your patience Rainer... ok I get it now. But see comment > at the end. After saying that about something at the end.... I forgot to include it. I was just wondering why the example config that comes with the rsyslog distro leaves out such important stuff as what it takes to get the server listening for log messages. It has a commented section that claims to be about: `Receiving Messages from Remote Hosts' # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 But it doesn't bother to include the things that make tcp work, like the all important lines posted by Aaron N. $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 Again thank you both for your input and for website too Rainer. From ktm at rice.edu Tue Feb 23 03:03:19 2010 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 22 Feb 2010 20:03:19 -0600 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87fx4skczi.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> Message-ID: <20100223020319.GA21475@it.is.rice.edu> On Mon, Feb 22, 2010 at 06:33:05PM -0600, Harry Putnam wrote: > Harry Putnam writes: > > > Thanks for your patience Rainer... ok I get it now. But see comment > > at the end. > > After saying that about something at the end.... I forgot to include > it. > > I was just wondering why the example config that comes with the > rsyslog distro leaves out such important stuff as what it takes to get > the server listening for log messages. > > It has a commented section that claims to be about: > `Receiving Messages from Remote Hosts' > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > But it doesn't bother to include the things that make tcp work, like > the all important lines posted by Aaron N. > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 > > Again thank you both for your input and for website too Rainer. > Maybe the sample configuration files are for an earlier release of rsyslog. Cheers, Ken From reader at newsguy.com Tue Feb 23 16:02:24 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:02:24 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <87r5ochu67.fsf@newsguy.com> Kenneth Marshall writes: [...] Harry wrote: >> I was just wondering why the example config that comes with the >> rsyslog distro leaves out such important stuff as what it takes to get >> the server listening for log messages. [...] Harry wrote: >> But it doesn't bother to include the things that make tcp work, like >> the all important lines posted by Aaron N. >> >> $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) >> $InputTCPServerRun 514 # start up TCP listener at port 514 [...] Kenneth wrote: > Maybe the sample configuration files are for an earlier release of > rsyslog. You mean like maybe the version advanced and nobody bothered to update the example conf? Yeah that might be what happened but even so, it seems you'd want the example conf shipped with a version to match the version. What I'm looking at was installed with version 3.22.1 From reader at newsguy.com Tue Feb 23 16:11:18 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:11:18 -0600 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <87mxz0htrd.fsf@newsguy.com> Just getting started with rsyslog and finding the documentation pretty confusing. Probably not the fault of the documentation but more my lack of knowledge. What I'd like to do is have rsyslog create a directory for incoming foreign logs for each host that sends syslog info. >From there, its ok if it all goes in a single log, inside the directory for now, and I may work out some splitting scheme later. So far, with help from posters here I've managed to get the logs comming in from remote machines and logging to a file named after the host. ... I suspect I should be able to figure it out from the pages here (line wrapped for mail): http://wiki.rsyslog.com/index.php \ /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory $template DYNmessages,"/var/log/%HOSTNAME%/messages" But I'd want the directory named for the host and include all logs from that host, not just the `messages' log. Not quite getting how to edit that line to make it happen. From anichols at trumped.org Tue Feb 23 16:49:48 2010 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 23 Feb 2010 08:49:48 -0700 Subject: [rsyslog] How to make rsyslog create dir per host In-Reply-To: <87mxz0htrd.fsf@newsguy.com> References: <87mxz0htrd.fsf@newsguy.com> Message-ID: On Tue, Feb 23, 2010 at 8:11 AM, Harry Putnam wrote: > What I'd like to do is have rsyslog create a directory for incoming > foreign logs for each host that sends syslog info. [...] > I suspect I should be able to figure it out from the pages here (line > wrapped for mail): > ?http://wiki.rsyslog.com/index.php \ > ? /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > ?$template DYNmessages,"/var/log/%HOSTNAME%/messages" > > But I'd want the directory named for the host and include all logs from > that host, not just the `messages' log. You want to get familiar with templates and actions - there are two pieces to this. Rather than tell you exactly how to solve your problem you should read this: http://www.rsyslog.com/doc-rsyslog_conf_templates.html and this: http://www.rsyslog.com/doc-rsyslog_conf_actions.html And bookmark this: http://www.rsyslog.com/doc-property_replacer.html You are correct, the documentation isn't perfect. That said, the application works pretty good and solves a lot of problems that aren't otherwise easily solved, so people like to use it. I would prefer the developers put their few precious resources into making the application solid vs. creating explicit documentation. You can always come here for help. But for that reason I recommend you read the above thoroughly and also the other sections of the configuration documentation. You'll spend some time putting together a configuration that works for you but once you understand how the configs work I think you'll find it relatively easy to figure out how to do specific things. If you think you understand how the selectors, actions and templates work together and still have problems or questions, let us know. Aaron From rgerhards at hq.adiscon.com Tue Feb 23 18:20:51 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:20:51 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103930@GRFEXC.intern.adiscon.com> Hi Aaron, thanks for all your help - MUCH appreciated. One thing I'd like to mention: > You are correct, the documentation isn't perfect. That said, the > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree to your analysis, but the doc problem has even more facets. I have blogged about it (once again), so that I have the information right at hand and available for easy sharing. I'd really appreciate if you could take out a few minutes and read this: http://blog.gerhards.net/2010/02/why-is-writing-good-user-doc-such.html In short: it would be very useful if "the average user" would contribute somewhat to the doc (or wiki!), because other "average users" understand that probably much better than what I write... Rainer PS: as a motivation: the blog post also contains links to all the good documentation sources ;) From rgerhards at hq.adiscon.com Tue Feb 23 18:49:57 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:49:57 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com><87wry5kzkx.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com><87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> > Maybe the sample configuration files are for an earlier release of > rsyslog. Harry, please do me a favor and complain with your distribution - they probably have simply overlooked this (can easily happen). I have just checked, the sample rsyslog.conf I provide includes commented-out remote reception commands (for a reason ;)): http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Also, I have added some text to the rsyslog cookbook for remote reception. I would appreciate if you could have a look and let me know if it is useful. If not, some text suggestions would be most welcome: http://cookbook.rsyslog.com/node16.html Rainer > > Cheers, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rory at ooma.com Wed Feb 24 00:37:07 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 15:37:07 -0800 Subject: [rsyslog] amazon s3 as logstore Message-ID: <4B8466A3.8000304@ooma.com> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? From rory at ooma.com Wed Feb 24 01:05:20 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:05:20 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8466A3.8000304@ooma.com> References: <4B8466A3.8000304@ooma.com> Message-ID: <4B846D40.3070505@ooma.com> On 2/23/10 3:37 PM, Rory Toma wrote: > Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Or, generically, how would I take a group of rsyslog servers accepting logs and have them log to central storage, without using a single central logger, and preferably, not using SQL injection. From david at lang.hm Wed Feb 24 01:10:48 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:10:48 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Or, generically, how would I take a group of rsyslog servers accepting > logs and have them log to central storage, without using a single > central logger, and preferably, not using SQL injection. so you want the logs send to a central place but not to a central logger. what's the difference? as a security person SQL injection is a exploit of badly written code, not something anyone would ever want to use, so I don't understand what you are asking here. David Lang From pgollucci at p6m7g8.com Wed Feb 24 01:11:16 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Wed, 24 Feb 2010 00:11:16 +0000 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B846EA4.1090800@p6m7g8.com> On 02/24/10 00:05, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? The easiest way AFAIK, is to have your own central rsyslog server with a db (mysql). Then use tarnsap (http://tarsnap.com) to copy either log files or db dumps to it. It uses AES256 bit encryption among other things so it can even story CC related data if needed. FWIW, tarnsap uses s3/ec2 for storage et al. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From rory at ooma.com Wed Feb 24 01:20:22 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:20:22 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B8470C6.507@ooma.com> I would like several front end servers writing to a single file-system, but I would expect that, out-of-the-box I would get out of order files and would have locking issues. On 2/23/10 4:10 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > > >> On 2/23/10 3:37 PM, Rory Toma wrote: >> >>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Or, generically, how would I take a group of rsyslog servers accepting >> logs and have them log to central storage, without using a single >> central logger, and preferably, not using SQL injection. >> > so you want the logs send to a central place but not to a central logger. > > what's the difference? > > as a security person SQL injection is a exploit of badly written code, not > something anyone would ever want to use, so I don't understand what you > are asking here. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Feb 24 01:34:05 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:34:05 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8470C6.507@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > I would like several front end servers writing to a single file-system, but I > would expect that, out-of-the-box I would get out of order files and would > have locking issues. Yes, if you use a standard filesystem you can only have one system writing to it at a time. If you want multiple systems to be able to write to a single filesystem you need to select a cluster filesystem or a network filesystem such a filesystem will allow multiple machines to write to different files on the shared filesystem at once, but they cannot write to the same file from different machines. all of the options have very different performance, reliability, and cost trade-offs. You will need to spend a lot of time thinking though your particular requirements and the capabilities of the different systems to pick one. I'm not sure exactly what you mean by 'out of order files', but if you are thinking that everything would end up in the same file and be in a different order than they were 'really' generated, then you are correct. The problem is that packets going over the network may not arrive in the same order that they were sent. Thus it is impossible for any system to guarantee the log entries reamin in the same order. the best that anything can do it to record very accurate timestamps and re-sort the logs after they have arrived. Very few people go down this road for logs. Almost everyone relays the logs to a single server and lets that server write to a filesystem (and if they need to eliminate this as a single point of failure, they make another system able to take over this filesystem if the first one goes down) David Lang > On 2/23/10 4:10 PM, david at lang.hm wrote: >> On Tue, 23 Feb 2010, Rory Toma wrote: >> >> >>> On 2/23/10 3:37 PM, Rory Toma wrote: >>> >>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> Or, generically, how would I take a group of rsyslog servers accepting >>> logs and have them log to central storage, without using a single >>> central logger, and preferably, not using SQL injection. >>> >> so you want the logs send to a central place but not to a central logger. >> >> what's the difference? >> >> as a security person SQL injection is a exploit of badly written code, not >> something anyone would ever want to use, so I don't understand what you >> are asking here. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From rory at ooma.com Wed Feb 24 01:38:49 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:38:49 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: <4B847519.8060702@ooma.com> On 2/23/10 4:34 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > >> I would like several front end servers writing to a single >> file-system, but I would expect that, out-of-the-box I would get out >> of order files and would have locking issues. > > Yes, if you use a standard filesystem you can only have one system > writing to it at a time. > > If you want multiple systems to be able to write to a single > filesystem you need to select a cluster filesystem or a network > filesystem > > such a filesystem will allow multiple machines to write to different > files on the shared filesystem at once, but they cannot write to the > same file from different machines. > > all of the options have very different performance, reliability, and > cost trade-offs. You will need to spend a lot of time thinking though > your particular requirements and the capabilities of the different > systems to pick one. > > I'm not sure exactly what you mean by 'out of order files', but if you > are thinking that everything would end up in the same file and be in a > different order than they were 'really' generated, then you are > correct. The problem is that packets going over the network may not > arrive in the same order that they were sent. Thus it is impossible > for any system to guarantee the log entries reamin in the same order. > the best that anything can do it to record very accurate timestamps > and re-sort the logs after they have arrived. > > Very few people go down this road for logs. Almost everyone relays the > logs to a single server and lets that server write to a filesystem > (and if they need to eliminate this as a single point of failure, they > make another system able to take over this filesystem if the first one > goes down) > Yeah, I didn't think it was practical, but if someone had done something unorthodox, and it worked, I thought it might be nice to hear about it. From david at lang.hm Wed Feb 24 01:57:28 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:57:28 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, david at lang.hm wrote: > I'm not sure exactly what you mean by 'out of order files', but if you are > thinking that everything would end up in the same file and be in a different > order than they were 'really' generated, then you are correct. The problem is > that packets going over the network may not arrive in the same order that > they were sent. Thus it is impossible for any system to guarantee the log > entries reamin in the same order. the best that anything can do it to record > very accurate timestamps and re-sort the logs after they have arrived. By the way, if keeping the logs in order is critical to you, you need to be aware that rsyslog may not process the logs in order under all conditions. For example: if you have multiple worker threads, they each grab some logs off the queue and attempt to deliver/write them. This can cause the logs to be delivered/written in a different order than rsyslog received them. Since there are so many other things that can re-order logs slightly, this is not viewed as a significant problem. David Lang > >> On 2/23/10 4:10 PM, david at lang.hm wrote: >>> On Tue, 23 Feb 2010, Rory Toma wrote: >>> >>> >>>> On 2/23/10 3:37 PM, Rory Toma wrote: >>>> >>>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> Or, generically, how would I take a group of rsyslog servers accepting >>>> logs and have them log to central storage, without using a single >>>> central logger, and preferably, not using SQL injection. >>>> >>> so you want the logs send to a central place but not to a central logger. >>> >>> what's the difference? >>> >>> as a security person SQL injection is a exploit of badly written code, not >>> something anyone would ever want to use, so I don't understand what you >>> are asking here. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> > From rgerhards at hq.adiscon.com Wed Feb 24 07:28:15 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 07:28:15 +0100 Subject: [rsyslog] amazon s3 as logstore References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com><4B8470C6.507@ooma.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103933@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, February 24, 2010 1:57 AM > To: Rory Toma > Cc: rsyslog-users > Subject: Re: [rsyslog] amazon s3 as logstore > > On Tue, 23 Feb 2010, david at lang.hm wrote: > > By the way, if keeping the logs in order is critical to you, you need > to > be aware that rsyslog may not process the logs in order under all > conditions. > > For example: if you have multiple worker threads, they each grab some > logs > off the queue and attempt to deliver/write them. This can cause the > logs > to be delivered/written in a different order than rsyslog received > them. > > Since there are so many other things that can re-order logs slightly, > this > is not viewed as a significant problem. > David is very right here. If you rely on the sequence as recorded in the log file, you have a big problem. That problem is not caused by rsyslog's threading, but rather by the fact that syslog is usually used inside a remote system. In such a system, there is no guaranteed sequence: UDP get's reordered, TCP messages sit for different timespans in interim routers, syslog relay chains may delay messages even for hours (if a relay is down). I thought I had blogged about this, but can't find the post myself right now ;) But in short: you need to be very careful about your sequence expectation. Timestamps are much better, but of course not perfect. RFC5424 provides the plumbing for a better solution, but none is standardized yet and I know of no such implementation. I have Lamport clocks on my mind as the "ultimate" solution (and maybe I get time to implement that in a distant future...). Rainer From tbergfeld at hq.adiscon.com Wed Feb 24 14:57:16 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 24 Feb 2010 14:57:16 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From reader at newsguy.com Wed Feb 24 21:03:57 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:03:57 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> Message-ID: <871vgaieoi.fsf@newsguy.com> "Rainer Gerhards" writes: >> Maybe the sample configuration files are for an earlier release of >> rsyslog. > > Harry, > > please do me a favor and complain with your distribution - they probably have > simply overlooked this (can easily happen). I have just checked, the sample > rsyslog.conf I provide includes commented-out remote reception commands (for > a reason ;)): > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 > e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Thanks for showing that... and for a moment there I was afraid I was the dope that had somehow deleted the important missing lines. So just to make sure I checked my distros' (gentoo) rsyslog.conf shipped for installation by the distro maintainer and sure enough they have: # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 When the original shipped with rsyslog pkg has the lines below as you've shown. # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module #$InputTCPServerRun 514 # start up TCP listener at port 514 # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 Apparently my distro has removed that one all important line. `#$InputTCPServerRun 514 # start up TCP listener at port 514' And yes, I will make sure to contact the package maintainer about it. Thanks for the heads up and I'm sure the maintainer will want to get that back in there too. > Also, I have added some text to the rsyslog cookbook for remote reception. I > would appreciate if you could have a look and let me know if it is useful. If > not, some text suggestions would be most welcome: > > http://cookbook.rsyslog.com/node16.html > Thanks... that couldn't be more clear... brief and directly on point. Thanks. From reader at newsguy.com Wed Feb 24 21:11:14 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:11:14 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <87wry2gzrx.fsf@newsguy.com> Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. From rgerhards at hq.adiscon.com Wed Feb 24 22:09:01 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 22:09:01 +0100 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> At least for me, it would be usefuly if you could describe your needs in relatively precise words. It may be obvious for you, but that is what is hard to grasp for me... ----- Urspr?ngliche Nachricht ----- Von: "Harry Putnam" An: "rsyslog at lists.adiscon.com" Gesendet: 24.02.10 21:12 Betreff: Re: [rsyslog] How to make rsyslog create dir per host Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 01:45:35 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:45:35 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/24 Tom Bergfeld : > Hi all, > We have just released rsyslog 4.6.0. > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new > features of 4.5.x to the stable releases. This includes bug fixes as well as > many new features. Out of the many, I would like to highlight the ability to > have output files automatically zipped, multiple TCP listeners and overall > performance improvement. For all details, please check the 4.5.x change log > entries. > > This release has undergone rigorous testing in some very demanding > environments. We held release until all tests went out well. We are happy to > have reached this stage now and sincerely think that the version is ready for > prime time. Thanks to everyone who contributed test time and bug reports! > > We recommend that all v4-stable users have a look at 4.6.0 and consider > updating to it. Please note that 4.6.0 also does have some patches which > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available > via support contracts). Seems, the xconsole/pipe related fixes did not make it into this release. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:48:53 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:48:53 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Michael Biebl : > > Seems, the xconsole/pipe related fixes did not make it into this release. Running rsyslogd in debug mode and opening xconsole (as root) I get 8841.463601465:b6e86b70: Message from UNIX socket: #3 8841.463676615:b6e86b70: logmsg: flags 4, from 'pluto', msg Feb 25 01:47:21 sudo: michael : TTY=pts/4 ; PWD=/home/michael/debian/rsyslog ; USER=root ; COMMAND=/usr/bin/xconsole 8841.463705390:b6e86b70: Message has legacy syslog format. 8841.463735003:b6e86b70: main Q: entry added, size now 1 entries 8841.463762660:b6e86b70: wtpAdviseMaxWorkers signals busy 8841.463795905:b6e86b70: main Q: EnqueueMsg advised worker start 8841.463825239:b6e86b70: --------imuxsock calling select, active file descriptors (max 3): 3 8841.463884465:b7687b70: main Q: entry deleted, state 0, size now 0 entries 8841.463925812:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'NetworkManager': FALSE 8841.463959895:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'wpa_supplicant': FALSE 8841.463997330:b7687b70: Called action, logging to builtin-file 8841.464033369:b7687b70: file to log to: /var/log/auth.log 8841.464057674:b7687b70: doWrite, pData->pStrm 0x8a32360, lenBuf 124 8841.464081699:b7687b70: strm 0x8a32360: file 9 flush, buflen 124 8841.464150983:b7687b70: strm 0x8a32360: file 9 write wrote 124 bytes 8841.464196520:b7687b70: Called action, logging to builtin-file 8841.464224177:b7687b70: file to log to: /dev/xconsole 8841.464247086:b7687b70: doWrite, pData->pStrm 0x8a3c248, lenBuf 124 8841.464269714:b7687b70: strm 0x8a3c248: file -1 flush, buflen 494 8841.464298210:b7687b70: Note: stream '/dev/xconsole' is a named pipe, open with O_NONBLOCK 8841.464340674:b7687b70: file '/dev/xconsole' opened as #-1 with mode 416 8841.464380065:b7687b70: strm 0x8a3c248: open error 6, file '/dev/xconsole': No such device or address 8841.464413869:b7687b70: main Q:Reg/w0: worker IDLE, waiting for work. The relevant rsyslog.conf configuration is daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:50:45 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:50:45 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: As additional information: If i run rsyslogd first, then xconsole, I get the failure Running xconsole first, then rsyslogd, it works fine. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Thu Feb 25 01:58:52 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 18:58:52 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> Message-ID: <87fx4qgmgj.fsf@newsguy.com> "Rainer Gerhards" writes: > At least for me, it would be usefuly if you could describe your > needs in relatively precise words. It may be obvious for you, but > that is what is hard to grasp for me... Not sure what you are talking about here Rainer. I was just replying to someone who appeared to think I had made some disparaging remark about the documentation. I simply wanted to clear that up... Now if you mean for me to explain more precisely what I wanted from rsyslog, I'll do that. However, let me say first that with Aarons' pointers to URLS that I am now working my way thru and also having just watched one of your videos and the basics... I have to tell you that that caliber of the documentation is really quite high. There is in depth coverage and then very helpful outside resources such as the video and other postings of yours. I wish I had the command of a foreign language as well as you have of english. I'm a native english speaker... but am sadly nearly illiterate in my own language. The documentation really does out class the usual documentation one encounters in unix/linux tools. ------- --------- ---=--- --------- -------- What I sought to convey in my question about creating directories: Its really low level so sorry I didn't get it across very well. Once I got the clients writing across the network to the rsyslog server, and the server listening and logging the data, I thought for now it would be good to have rsyslog on the server create directories for each remote client, and write everything from each client to /var/log/%hostname%/everthing.log (as a beginning point. While I figure out how to set things up so that each incoming log from remote would be split up further under the top level directories. Kind of like the example shown at (wrapped for mail): http://wiki.rsyslog.com/index.php/ \ Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory Just one level deeper. So one level of dynamic directory is created for each client host. Later I'll probably want to spit that a bit more Once I have a more permanent plan figured out I would at least have the full output from each client host on hand. I needed to get the basics up some whet urgently in that I had one host failing silently or at least not leaving a trace I could find in its logs. It would just freeze up, where the mouse and keyboard became unresponsive and could not connect to the problem host remotely either. On mechanical reboot, it would look like the system logger froze up too but someone told me I might catch some log data that would be lost on a hard reboot if I had that host logging remotely...thru rsyslog. I doubted that would be true since it appears as though networking is froze. But I have gotten rsyslog logging remotely now and waiting for a freezeup to see if I can get any more clues as to what is causing the problem. From buraphalinuxserver at gmail.com Thu Feb 25 08:34:29 2010 From: buraphalinuxserver at gmail.com (BuraphaLinux Server) Date: Thu, 25 Feb 2010 14:34:29 +0700 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x Message-ID: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Will there be an rsyslog 4.4.3 release? I see references to changes from it in the website, but cannot find the 4.4.3 release itself, and the 4.6.0 release seems to say that there will not be any 4.x releases any more. The web page says 4.6.0 is a stable release and a beta at the same time, which is very confusing for me. http://www.rsyslog.com/Article443.phtml The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which means it is a beta, but then the text on the page also says "This is a new stable v4 version." and "Do not expect any more new developments in v4.". I'm a bit confused. Does this mean if we want bugfixes we have to run a 5.x version? From rgerhards at hq.adiscon.com Thu Feb 25 09:09:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:09:24 +0100 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x References: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of BuraphaLinux Server > Sent: Thursday, February 25, 2010 8:34 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x > > Will there be an rsyslog 4.4.3 release? I see references to changes > from it in the website, but cannot find the 4.4.3 release itself, No. I initially planned to have a 4.4.3, but Monday, when I tried to prepare it, I found some issues in the automated test suite. Whatever they were, they were fixed in 4.6.0. So I skipped that release, saving me the time to find what already is fixed (the original plan was to do a last 4.4.3 release followed by 4.6.0 two days later, so I only skipped the 4.4.3 step). > and > the 4.6.0 release seems to say that there will not be any 4.x releases > any more. The web page says 4.6.0 is a stable release and a beta at > the same time, which is very confusing for me. > > http://www.rsyslog.com/Article443.phtml Thanks for pointing that out. That's a simple typo, now corrected. > > The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which > means it is a beta, but then the text on the page also says "This is a > new stable v4 version." and "Do not expect any more new developments > in v4.". I'm a bit confused. Does this mean if we want bugfixes we > have to run a 5.x version? Bugfixes are not "new developments", at least in my point of view. But if you need some new feature, you need to run v5 if we implement it. That's the way it was always handled in rsyslog, with the exception of v4 and v5, which had two different development branches at the same time. There were good reasons for that, but it also caused a lot of work and some issues. I am very happy to be back to a single development version again. I won't split off a new second development branch again until there is *very* good reason. I hope this clarifies. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:25:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:25:18 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Hi all, on Debian, a discussion came up on the inconsistency of some config statements. The right cure would be to correct this, but that requires some time. I would appreciate your feedback on the issue. The details can be found here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 Thanks, Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:26:47 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:26:47 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:46 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/24 Tom Bergfeld : > > Hi all, > > We have just released rsyslog 4.6.0. > > > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings > all new > > features of 4.5.x to the stable releases. This includes bug fixes as > well as > > many new features. Out of the many, I would like to highlight the > ability to > > have output files automatically zipped, multiple TCP listeners and > overall > > performance improvement. For all details, please check the 4.5.x > change log > > entries. > > > > This release has undergone rigorous testing in some very demanding > > environments. We held release until all tests went out well. We are > happy to > > have reached this stage now and sincerely think that the version is > ready for > > prime time. Thanks to everyone who contributed test time and bug > reports! > > > > We recommend that all v4-stable users have a look at 4.6.0 and > consider > > updating to it. Please note that 4.6.0 also does have some patches > which > > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still > available > > via support contracts). > > Seems, the xconsole/pipe related fixes did not make it into this > release. mhhh... The delta between 4.5.8 and 4.6.0 is very slim. So I wonder what might have broken that fix... Will try to check. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:44:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:44:10 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Thursday, February 25, 2010 1:59 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] How to make rsyslog create dir per host > > "Rainer Gerhards" writes: > > > At least for me, it would be usefuly if you could describe your > > needs in relatively precise words. It may be obvious for you, but > > that is what is hard to grasp for me... > sorry, that was too brief. I was on a PDA and it was late ;) My intent is to see if I can write up a cookbook entry for what you need, but I have no clear understanding what's your intent. Now let me read through the rest of your mail and see where I come to... ;) > Not sure what you are talking about here Rainer. I was just replying > to someone who appeared to think I had made some disparaging remark > about the documentation. > > I simply wanted to clear that up... > > > Now if you mean for me to explain more precisely what I wanted from > rsyslog, I'll do that. > > However, let me say first that with Aarons' pointers to URLS that I am > now working my way thru and also having just watched one of your > videos and the basics... I have to tell you that that caliber of the > documentation is really quite high. > > There is in depth coverage and then very helpful outside resources such > as the video and other postings of yours. > > I wish I had the command of a foreign language as well as you have of > english. I'm a native english speaker... but am sadly nearly > illiterate in my own language. > > The documentation really does out class the usual documentation one > encounters in unix/linux tools. Thanks for the kind words, but you had a valid point in your initial remarks: the rsyslog doc may be good for someone who intends to understand the whole system and be an expert at that. However, most often people look for a solution to a specific problem. They don't want to understand how everything works together, just solve that immediate need quickly. I think this is a valid need, I have it myself with other softwares often enough ;) The rsyslog doc completely fails to address this use case. I hope the cookbook will somewhat improve that situation. I am not sure if you saw my longer blogpost, it has the details ;) > > ------- --------- ---=--- --------- -------- > What I sought to convey in my question about creating directories: > > Its really low level so sorry I didn't get it across very well. > > Once I got the clients writing across the network to the rsyslog > server, and the server listening and logging the data, I thought for > now it would be good to have rsyslog on the server create directories > for each remote client, and write everything from each client to > /var/log/%hostname%/everthing.log (as a beginning point. > > While I figure out how to set things up so that each incoming log > from remote would be split up further under the top level directories. > > Kind of like the example shown at (wrapped for mail): > > http://wiki.rsyslog.com/index.php/ \ > Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > Just one level deeper. So one level of dynamic directory is created > for each client host. Later I'll probably want to spit that a bit more ... and here is where some information is missing ;) *How* do you want to split these logs? I am not sure about this, and so I have no clear idea of how a configuration could look like. As there are some limits on the way filters can be combined, this is a very important point. As a side-note: I do not know your log volume, but if it is not overwhelmingly large to prevent storing logs to a database, I'd probably do that and analyze and search it with free GPLed Adiscon LogAnalyzer: http://loganalyzer.adiscon.com These days, we are preparing a new v3 release which also supports reporting (and you can schedule that via cron, too). (Adiscon LogAnalyzer also works with plain text files, but this is only meant for low-volume, few-system consolidated logs - otherwise the performance will be very bad). HTH Rainer > > Once I have a more permanent plan figured out I would at least have > the full output from each client host on hand. > > I needed to get the basics up some whet urgently in that I had one host > failing silently or at least not leaving a trace I could find in its > logs. It would just freeze up, where the mouse and keyboard became > unresponsive and could not connect to the problem host remotely > either. > > On mechanical reboot, it would look like the system logger froze up > too but someone told me I might catch some log data that would be lost > on a hard reboot if I had that host logging remotely...thru rsyslog. > > I doubted that would be true since it appears as though networking is > froze. But I have gotten rsyslog logging remotely now and waiting for > a freezeup to see if I can get any more clues as to what is causing > the problem. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 25 09:51:31 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:51:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. ah! I think I now understand the issue. This smells like a new one. Did this really work with 4.5.8? Rainer > > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 13:22:02 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 13:22:02 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, February 25, 2010 1:51 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. > > ah! I think I now understand the issue. This smells like a new one. Did this > really work with 4.5.8? I haven't tested 4.5.8. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 13:56:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 13:56:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> > -----Original Message----- > 2010/2/25 Rainer Gerhards : > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl > >> Sent: Thursday, February 25, 2010 1:51 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > >> > >> As additional information: > >> > >> If i run > >> rsyslogd first, then xconsole, I get the failure > >> Running xconsole first, then rsyslogd, it works fine. > > > > ah! I think I now understand the issue. This smells like a new one. > Did this > > really work with 4.5.8? > > I haven't tested 4.5.8. ah, ok, so which was the latest version you tested against? Rainer From mbiebl at gmail.com Thu Feb 25 14:01:19 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 14:01:19 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> 2010/2/25 Rainer Gerhards : >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> >> Sent: Thursday, February 25, 2010 1:51 AM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> >> >> As additional information: >> >> >> >> If i run >> >> rsyslogd first, then xconsole, I get the failure >> >> Running xconsole first, then rsyslogd, it works fine. >> > >> > ah! I think I now understand the issue. This smells like a new one. >> Did this >> > really work with 4.5.8? >> >> I haven't tested 4.5.8. > > ah, ok, so which was the latest version you tested against? > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 14:06:11 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 14:06:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103966@GRFEXC.intern.adiscon.com> > >> >> If i run > >> >> rsyslogd first, then xconsole, I get the failure > >> >> Running xconsole first, then rsyslogd, it works fine. > >> > > >> > ah! I think I now understand the issue. This smells like a new > one. > >> Did this > >> > really work with 4.5.8? > >> > >> I haven't tested 4.5.8. > > > > ah, ok, so which was the latest version you tested against? > > > > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. OK, misunderstanding on my side. I thought you had verified that the "xconsole" patch worked. I will try to dig again into the code, but it's some while since I crafted the patch, so it may take a while. Maybe I manage to do that today, if not it'll be monday - I have meetings tomorrow :( Rainer From rgerhards at hq.adiscon.com Thu Feb 25 15:27:44 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 15:27:44 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. > I just verified the case. I get the failure if xconsole is not active, what is perfectly OK IMHO (because otherwise it would block on the pipe). As soon as I start xconsole, everything works fine and messages are displayed in xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it everything works again. So I do not see an issue. I tested on Debian lenny with all current updates applied. Rainer > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Thu Feb 25 20:55:19 2010 From: reader at newsguy.com (Harry Putnam) Date: Thu, 25 Feb 2010 13:55:19 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> Message-ID: <87y6ihf5ug.fsf@newsguy.com> "Rainer Gerhards" writes: >> Just one level deeper. So one level of dynamic directory is created >> for each client host. Later I'll probably want to spit that a bit more > > ... and here is where some information is missing ;) *How* do you want to > split these logs? I am not sure about this, and so I have no clear idea of > how a configuration could look like. As there are some limits on the way > filters can be combined, this is a very important point. He he... yes it is.. but its missing because I don't have a solid plan for what I want yet. I really was only asking about the simple part. I now know how to do that, and when you see what I was after (for now) you will be disappointed because its probably to obvious and simple to need to be in your cook book. It just takes me a while to see how to get started. On the client machines: *.info;mail.none @@192.168.0.26:514 (192.168.0.26 is `logsrv' mentioned below) On the server: This template and action is all I was after (for now). (aside: DDF = Dynamic directory and filename) $template DDF,"/var/log/%hostname%/%hostname%.log" if \ $source != 'logsrv' \ then -?DDF I hope that is at least close to a correct formulation. I just used the actual host names because I wan't sure which property provided it .. just the simple alphabetic hostname with no domain. On thing I didn't see explained in what I've read so far is what role the question mark in -?DDF plays. I used it because I saw it used that way in the examples and so far the formulation seems to be working...(I actually haven't inspected the logs closely yet.... but a tail -f of the resulting log files would seem to indicate its working. At least each log appears to be only from a specific client, and to include everything incoming from that client.) From mbiebl at gmail.com Thu Feb 25 21:44:31 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 21:44:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. >> > > I just verified the case. I get the failure if xconsole is not active, what > is perfectly OK IMHO (because otherwise it would block on the pipe). As soon > as I start xconsole, everything works fine and messages are displayed in > xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it > everything works again. > > So I do not see an issue. I tested on Debian lenny with all current updates > applied. hm, it's perfectly reproducible here. starting rsyslogd, then xconsole, will cause xconsole to hang and no message is written to the xconsole window. 4.4.2 did not behave that way. See also the debug log I sent previously Is there anything else I can do to help you debug this? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Fri Feb 26 01:09:16 2010 From: lanas at securenet.net (lanas) Date: Thu, 25 Feb 2010 19:09:16 -0500 Subject: [rsyslog] Sending time in remote syslog Message-ID: <20100225190916.0a83b31d@mistral.stie> Hello ! Is it possible with rsyslog to somehow send the time of the local event as part of the log message when sending to a remote server ? Here's a use case. I have tried with syslogd. I enable syslogd on a unit to send log messages to a remote host running rsyslog (it so happens that it runs rsyslog because it's a Fedora host). Then I change the time on the unit by an offset of two hours. When the unit (syslogd) sends the log message, it is received by rsyslogd although the time stamp in the log msg shows the time on the host, not the time when the event happened on the unit itself. If I replace syslogd by rsyslog on the unit, will it be able to somehow send the local time of the unit as part of the log message ? Thanks, Al From david at lang.hm Fri Feb 26 08:46:21 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 25 Feb 2010 23:46:21 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100225190916.0a83b31d@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> Message-ID: On Thu, 25 Feb 2010, lanas wrote: > Hello ! > > Is it possible with rsyslog to somehow send the time of the local > event as part of the log message when sending to a remote server ? > > Here's a use case. I have tried with syslogd. I enable syslogd on a > unit to send log messages to a remote host running rsyslog (it so > happens that it runs rsyslog because it's a Fedora host). Then I > change the time on the unit by an offset of two hours. When the unit > (syslogd) sends the log message, it is received by rsyslogd although > the time stamp in the log msg shows the time on the host, not the time > when the event happened on the unit itself. > > If I replace syslogd by rsyslog on the unit, will it be able to > somehow send the local time of the unit as part of the log message ? each event has two timestamps, one is the timestamp in the message that was sent, the other is the time the message was received. I believe that the default template uses the recieved timestamp, but you can change it to timereportedto get the timestamp put on the message by the sender. David Lang From lanas at securenet.net Fri Feb 26 11:46:05 2010 From: lanas at securenet.net (lanas) Date: Fri, 26 Feb 2010 05:46:05 -0500 Subject: [rsyslog] Sending time in remote syslog In-Reply-To: References: <20100225190916.0a83b31d@mistral.stie> Message-ID: <20100226054605.3469f03a@mistral.stie> On Thu, 25 Feb 2010 23:46:21 -0800 (PST), david at lang.hm wrote : > On Thu, 25 Feb 2010, lanas wrote: > > Is it possible with rsyslog to somehow send the time of the local > > event as part of the log message when sending to a remote server ? > > each event has two timestamps, one is the timestamp in the message > that was sent, the other is the time the message was received. I > believe that the default template uses the recieved timestamp, but > you can change it to timereportedto get the timestamp put on the > message by the sender. Thanks, I will try it. Basically, this means that (if there's a syslog message standard ...) that syslog messages contains both timestamps, be them generated by syslogd or rsyslog, isn't it ? And it's at the receiving side that the proper display option should be chosen (in this case, does rsyslog have an advantage over others by providing more display options ?). Regards, Al From rgerhards at hq.adiscon.com Fri Feb 26 12:32:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:32:10 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> > > I just verified the case. I get the failure if xconsole is not > active, what > > is perfectly OK IMHO (because otherwise it would block on the pipe). > As soon > > as I start xconsole, everything works fine and messages are displayed > in > > xconsole. When I stop xconsole I get "broken pipe" errors and if I > restart it > > everything works again. > > > > So I do not see an issue. I tested on Debian lenny with all current > updates > > applied. > > hm, it's perfectly reproducible here. also on Debian lenny? Anything else I could do to set up my environment so that the problem occurs? > starting rsyslogd, then xconsole, will cause xconsole to hang and no > message is written to the xconsole window. > 4.4.2 did not behave that way. > > See also the debug log I sent previously The log tells me that it has a problem to write to the pipe, and thus ignores it. This also happens at my system, but only if xconsole is not running. So the debug log does no help very much this time. > Is there anything else I can do to help you debug this? When I get the platform information from you, I will see if I can record a session in my lab. Maybe you see that I do things differently from you. One thing I notice is that xconsole seems to be unable to access /dev/xconsole if I start it as non-root, thus resulting in no listener on the pipe and thus the error appears in rsyslog. However, this is not rsyslog-related, but I guess these permissions are set by xconsole. After all, there should be no change to 4.4.2 (and I don't see any difference in that regard to 4.4.2 on my system). Rainer > > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Fri Feb 26 12:55:11 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 12:55:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> > I just verified the case. I get the failure if xconsole is not >> active, what >> > is perfectly OK IMHO (because otherwise it would block on the pipe). >> As soon >> > as I start xconsole, everything works fine and messages are displayed >> in >> > xconsole. When I stop xconsole I get "broken pipe" errors and if I >> restart it >> > everything works again. >> > >> > So I do not see an issue. I tested on Debian lenny with all current >> updates >> > applied. >> >> hm, it's perfectly reproducible here. > > also on Debian lenny? Anything else I could do to set up my environment so > that the problem occurs? I'm can reproduce the problem, both on Debian lenny and Debian unstable (i386) >> starting rsyslogd, then xconsole, will cause xconsole to hang and no >> message is written to the xconsole window. >> 4.4.2 did not behave that way. >> >> See also the debug log I sent previously > > The log tells me that it has a problem to write to the pipe, and thus ignores > it. This also happens at my system, but only if xconsole is not running. So > the debug log does no help very much this time. The problem is, that xconsole is running at this point! To reiterate: I start rsyslogd then xconsole then try to log a message The result is the debug log and no message showing up in xconsole. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 12:56:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:56:49 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> > The problem is, that xconsole is running at this point! > To reiterate: > I start rsyslogd > then xconsole > then try to log a message > The result is the debug log and no message showing up in xconsole. that's what I am doing as well. I will try to record the session. Rainer From rgerhards at hq.adiscon.com Fri Feb 26 13:20:17 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 13:20:17 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103973@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, February 26, 2010 12:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > > The problem is, that xconsole is running at this point! > > To reiterate: > > I start rsyslogd > > then xconsole > > then try to log a message > > The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Please find a video of my session at http://www.rsyslog.com/download/rsyslog-xconsole.avi So I don't see the problem. Or did you mean something different? Rainer From mbiebl at gmail.com Fri Feb 26 13:43:23 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 13:43:23 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> The problem is, that xconsole is running at this point! >> To reiterate: >> I start rsyslogd >> then xconsole >> then try to log a message >> The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Wait a sec. With 4.4.2 (or any earlier version) I tried this: start rsyslog start xconsole xconsole shows the last two log messages immediately With 4.6.0 the xconsole window is blank (which I guess made me believe it blocked) With 4.6.0, I need to trigger a new log message (e.g. using logger foo), and then xconsole will show a new message. So, xconsole does indeed show messages, but the behaviour is different to 4.4 and earlier Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 15:19:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 15:19:55 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103975@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, February 26, 2010 1:43 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Rainer Gerhards : > >> The problem is, that xconsole is running at this point! > >> To reiterate: > >> I start rsyslogd > >> then xconsole > >> then try to log a message > >> The result is the debug log and no message showing up in xconsole. > > > > that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Mhhh... I don't think it is worth preserving "backward compatibility" here. I think these two (just two!) messages were an artifact of the way files were written to pipes that nobody cares about. The old file writer was *very* suboptimal. If I would need to retain that old "functionality", I would need to remove almost all optimizations. So it would make more sense than to craft a new output module specifically for pipes. Not sure if this is granted by that "use case". I'd appreciate comments from the community at large. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Feb 26 17:11:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 17:11:29 +0100 Subject: [rsyslog] Log Normalization effort Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Hi all, I have blogged about my quest for log normalization. I think there is some good information on the upcoming GPLed Adiscon LogAnalyzer and future directions for rsyslog in the blog post. So I thought I share the link: http://blog.gerhards.net/2010/02/syslog-normalization.html Please note that part of the effort requires community involvement. I would be very interested to learn if you think we could win enough support to make this a useful effort. I am asking for your feedback, because it will help me streamline my priorities for future rsyslog work. Thanks, Rainer From david at lang.hm Fri Feb 26 19:34:04 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 10:34:04 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100226054605.3469f03a@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> <20100226054605.3469f03a@mistral.stie> Message-ID: On Fri, 26 Feb 2010, lanas wrote: > On Thu, 25 Feb 2010 23:46:21 -0800 (PST), > david at lang.hm wrote : > >> On Thu, 25 Feb 2010, lanas wrote: >>> Is it possible with rsyslog to somehow send the time of the local >>> event as part of the log message when sending to a remote server ? >> >> each event has two timestamps, one is the timestamp in the message >> that was sent, the other is the time the message was received. I >> believe that the default template uses the recieved timestamp, but >> you can change it to timereportedto get the timestamp put on the >> message by the sender. > > Thanks, I will try it. Basically, this means that (if there's a syslog > message standard ...) that syslog messages contains both timestamps, be > them generated by syslogd or rsyslog, isn't it ? And it's at the > receiving side that the proper display option should be chosen (in this > case, does rsyslog have an advantage over others by providing more > display options ?). the messages sent over the wire only have one timestamp in them, but when rsyslog receives a message it records that time as well. This gives rsyslog the option of using either timestamp when it then writes (or sends) the log. David Lang From david at lang.hm Fri Feb 26 22:32:56 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 13:32:56 -0800 (PST) Subject: [rsyslog] Log Normalization effort In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 26 Feb 2010, Rainer Gerhards wrote: > Hi all, > > I have blogged about my quest for log normalization. I think there is some > good information on the upcoming GPLed Adiscon LogAnalyzer and future > directions for rsyslog in the blog post. So I thought I share the link: > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > Please note that part of the effort requires community involvement. I would > be very interested to learn if you think we could win enough support to make > this a useful effort. I am asking for your feedback, because it will help me > streamline my priorities for future rsyslog work. a few comments (but remember that I am usually dealing with high data rates, so my concerns are biased in that direction) log analysis is usually done in batches as opposed to in real-time. some of this is due to the difficulty in doing it in real time, but a lot of it is the processing overhead (you don't want to take so long to process an individual request that you miss the next one to arrive) at low volumes the idea of name-value pairs in the logs makes a lot of sense, but there is significantly more overhead in parsing a log with name-value pairs in arbitrary orders than there is in using a tree parsing approach to analyze known log formats in a fixed order. The message size can also increase significantly. As a result, at high traffic volumes this starts to be a bad (or at least questionable) idea. I would love to see rsyslog gain the ability to efficiently do tree-based parsing instead of regex parsing. regex parsing is easy to understand and tinker with, but very expensive to implement. it may be that having something that 'compiles' a list of regex parsers into a tree parser is the right answer for usability. I would save several hours of processing a day if I could easily (and efficiently) make rsyslog write different logs to different files (at high data rates and with a few hundred conditions based variations in the syslog tag) While there are some common events across different types of logs (logins for example) they almost always contain slightly different data in them. I also have no faith at all that anyone is going to make much effort to clean up their logs to make them nicely parseable, and if they do I see even less chance that they will end up using the same terms for the same thing. As such I see more value in trying to get samples of logs and what they mean than in trying to define a normalized version to shoehorn the logs into. It is worth doing this for some events (logins, failed logins for example), but I think it's a mistake to think that this will end up covering all, or even the majority of log messages. There's also a problem in that the ideal format for the output depends on what you are doing with the output. If I could wave a magic wand and get the result I would look for something like this the parser starts at the beginning of the message (at the priority) and can branch on priority/faclilty, timestamp, host, syslogtag, message and indicate if the message should be parsed into name-value pairs, or split based on a character (or character sequence like the perl split command allows) into individually addressable elements (defaulting to whitespace separated elements), then the format (and if needed dynafile path/file components) could be constructed from these variables. At any point in the parsing it should be possible to jump to another parser tree (so that you could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end up using the same parser for the message without having to redefine the rules for each one) With this capability, people could start writing parser 'branches' to understand a specific log type and output a 'standard' format (as such a format can start to be defined). This can be done in rsyslog today, but it is fairly difficult to define, and as I understand it, inefficient enough that it's not practical to do in real-time under heavy load. If this is fast enough, then the next step would be to add the ability to have the format/action be 'increment a counter for log type X' and a signal to rsyslog could generate a report on these counters. Although at some point it becomes better to feed the message into another opensource tool (SEC, Simple Event Correlator for example) instead of trying to do everything in rsyslog. parsing the file to know what to do with it, and be able to re-format log messages is very defiantly something that can fit into the rsyslog model of receiving, formatting, and delivering logs. Alerting on specific log entries, counting the number of times one thing shows up in logs, and this sort of thing start pushing beyond the core of rsyslog, and it may be better to feed other tools instead. David Lang From lanas at securenet.net Sat Feb 27 12:38:07 2010 From: lanas at securenet.net (lanas) Date: Sat, 27 Feb 2010 06:38:07 -0500 Subject: [rsyslog] rsyslog migration and syslogd.conf Message-ID: <20100227063807.5918fafc@mistral.stie> Hello, First, thanks to David for the reply on time stamps. I have a question about perhaps being able to replace syslogd by rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I reckon with limited possibilities) with a standard syslog config file ? That would enable a basic set of functionality as a first step toward using rsyslog's features. Al From david at lang.hm Sun Feb 28 01:02:16 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:02:16 -0800 (PST) Subject: [rsyslog] rsyslog migration and syslogd.conf In-Reply-To: <20100227063807.5918fafc@mistral.stie> References: <20100227063807.5918fafc@mistral.stie> Message-ID: On Sat, 27 Feb 2010, lanas wrote: > Hello, > > First, thanks to David for the reply on time stamps. > > I have a question about perhaps being able to replace syslogd by > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > reckon with limited possibilities) with a standard syslog config > file ? That would enable a basic set of functionality as a first step > toward using rsyslog's features. no, rsyslog will do nothing if you just feed it a syslog.conf file. rsyslog needs a few additional instructions at the top of the file to load the modules, but below that you can paste in the syslog.conf file. another thing that you can do is to make a generic rsyslog.conf file that includes the syslog.conf after the rsyslog specific headers. My rsyslog.conf (on my ubuntu 9.10 laptop) has $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog $RepeatedMsgReduction on $RepeatedMsgContainsOriginalMsg on David Lang From david at lang.hm Sun Feb 28 01:45:14 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:45:14 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 25 Feb 2010, Rainer Gerhards wrote: > Hi all, > > on Debian, a discussion came up on the inconsistency of some config > statements. The right cure would be to correct this, but that requires some > time. I would appreciate your feedback on the issue. The details can be found > here: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 First off, the config language really has two parts. The first part is the configuration of rsyslog, the second part is the instructions to rsyslog telling it what to do with the log messages it has received.. I think you are wildly optimistic if you think that "a year from now nobody will be using the old config language". I expect that 3-5 years from now users will still be using 3.x versions (because they are what was released in 'enterprise' releases) even for new versions there will need to be a transition path (including the ability to easily revert to an old version) I fully understand the desire to scrap the existing evelved config language and implement a new, designed version instead (without needing to support both), but I don't see that as being realistic. A new designed config language should be able to have all the existing configs map to it, and I expect that you will have to maintain a shim/transition layer for many years that will read the old configs and map it to the new config. for the first part of the configs (the configuration options for rsyslog itself, having nothing to do with processing the log messages) As such, I think making the infrastructure to log 'this config option has been replaced by that config option' could actually end up cleaning things up and making it easier to modify the config language going forward. If the logging of changed is done one option at a time it will be a mess, but if it's done as something like a mapping layer it could actually be better. I'm thinking of a table along the lines of config_option, type, destination where type could be 'boolean', 'string', 'filesystem path', 'number', or 'depriciated', 'ignore' if the type is depriciated then the destination would be another config_option (which you would then look up) if the type is 'ignore' then it would just be skipped over (possibly with a warning being logged that the config line is being ignored) this would also clean up some of the current cases where a boolean option doesn't honor on/off and true/false. For the second half of the config language (telling rsyslog what to do with the logs it has received) also has several variations in place and is showing issues. However I think that the right answer here is to end up implementing something like the parsing trees and then compile the other config language options into that tree to be consistant (and fast) under the covers, no matter which way the instructions are written (except when you have to use regex matches) David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:11:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:11:55 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Hi David, thanks for the thoughtful comments. But I seem to have incorrectly phrased my question, leaving out too many of those things that I obviously took for granted and thus not mentioned :(... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:45 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Thu, 25 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > on Debian, a discussion came up on the inconsistency of some config > > statements. The right cure would be to correct this, but that > requires some > > time. I would appreciate your feedback on the issue. The details can > be found > > here: > > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 > > First off, the config language really has two parts. > > The first part is the configuration of rsyslog, the second part is the > instructions to rsyslog telling it what to do with the log messages it > has > received.. > > I think you are wildly optimistic if you think that "a year from now > nobody will be using the old config language". I expect that 3-5 years > from now users will still be using 3.x versions (because they are what > was > released in 'enterprise' releases) > That was probably the most important thing I did not mention: I do NOT think that removing inconsistencies in the config language is a bug fix. And if it is not a bug fix, it is a feature request. But if it is a feature request, it will go into the recent devel. And as such it will be a v5-exclusive thing. As such, what is with v3 and v4 doesn't really matter, because it would not be changed (and, yes, I totally agree that v3/3 will be around for a long time). One may argue if this feature is so essential that it should be rolled back into the old versions. However, I became very skeptic of doing so. Almost always in the past years, when I put some non-essential change into the stable releases directly, big problems were the result. I could probably open up new devel/beta branches for v3/v4 as well. But that would a) cause a lot of extra work b) people would most probably not try it out so we would end up with the same issue when it is then named stable c) the experience with two v4/v5 devel versions was a bad one as well. It is too easy to mix up where the regression is And now to the time slot: I guess, when a clean config language is there, most people would tend to use the new format. This is why I assumed that after a year the old language (in v5!) will most probably not be used much longer. To be precise, I fear that I take out time to implement this in v5, then have the new format (let's hope) 6 month later available then wonder if that would really make a difference over not doing the first step. If I had lots of time, it would probably still make sense. But unfortunately, I currently have far less time than I wish I had. Thankfully, the newer versions get more and more adopted, but that also means more bug fixes to do. So I sadly see that my available time is currently eaten up by those bug fixes and clarifying things inside the doc so that people can use the features. That is almost painful to me, because I have so many cool things on my mind, but do not find time to implement a single one of them :( > even for new versions there will need to be a transition path > (including > the ability to easily revert to an old version) > > I fully understand the desire to scrap the existing evelved config > language and implement a new, designed version instead (without needing > to > support both), but I don't see that as being realistic. > > A new designed config language should be able to have all the existing > configs map to it, and I expect that you will have to maintain a > shim/transition layer for many years that will read the old configs and > map it to the new config. The new config language doesn't mean that the old will no longer work. It is a primary design goal to support both in parallel. I always tried hard to make rsyslog as backwards-compatible as possible and it turns out that this is a major advantage, at least at the low end. I do not intend to drop that (one need also consider existing know-how: people should be able to get rsyslog immediately up and running, for simple cases, without the need to learn anything new). > > for the first part of the configs (the configuration options for > rsyslog > itself, having nothing to do with processing the log messages) > > As such, I think making the infrastructure to log 'this config option > has > been replaced by that config option' could actually end up cleaning > things > up and making it easier to modify the config language going forward. > > If the logging of changed is done one option at a time it will be a > mess, > but if it's done as something like a mapping layer it could actually be > better. > > I'm thinking of a table along the lines of > > config_option, type, destination > > where type could be 'boolean', 'string', 'filesystem path', 'number', > or > 'depriciated', 'ignore' This *is* how the system works today! But I don't have a deprecated flag and aliases, so I would need to rewrite that whole system. BTW: this system is the reason why I avoided writing a new system until absolutely necessary: it is too easy to add a new directive as all you need to do is add a new "table entry". > > if the type is depriciated then the destination would be another > config_option (which you would then look up) > > if the type is 'ignore' then it would just be skipped over (possibly > with > a warning being logged that the config line is being ignored) > > this would also clean up some of the current cases where a boolean > option > doesn't honor on/off and true/false. True/false is NOT a valid boolean option. See http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > For the second half of the config language (telling rsyslog what to do > with the logs it has received) also has several variations in place and > is > showing issues. > > However I think that the right answer here is to end up implementing > something like the parsing trees and then compile the other config > language options into that tree to be consistant (and fast) under the > covers, no matter which way the instructions are written (except when > you > have to use regex matches) I don't fully agree here with you. For example, the traditional PRI based filter is lightning fast. I don't see any way it could be nearly as fast with any unified approach. Right now, we have a "unified" filter structure, but it contains three filter cases, each one adding potential power at the price of decreased speed. I think we need to keep different types of filters in order to have some lightning-fast ones. But more of this could be done under the hood. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 11:15:21 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:15:21 +0100 Subject: [rsyslog] rsyslog migration and syslogd.conf References: <20100227063807.5918fafc@mistral.stie> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:02 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog migration and syslogd.conf > > On Sat, 27 Feb 2010, lanas wrote: > > > Hello, > > > > First, thanks to David for the reply on time stamps. > > > > I have a question about perhaps being able to replace syslogd by > > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > > reckon with limited possibilities) with a standard syslog config > > file ? That would enable a basic set of functionality as a first > step > > toward using rsyslog's features. > > no, rsyslog will do nothing if you just feed it a syslog.conf file. Actually, rsyslog is designed as a drop-in replacement. So it should work if you just feed it a regular rsyslog.conf. This is controlled via the -c option, which sets the compatibility level. If you call rsyslogd like you called sysklogd (without the -c option), it will fall back to sysklogd compatibility and generate the necessary config commands before parsing the the config file. Then, -r starts up a UDP listener as it did in sysklogd. ... At least this is how it is meant to work, I have to admit I have not tested it lately. For various reasons, I could not add this to the automated testbench, so I might not have noticed a regression. But if it does not work, this is a bug and not how things are designed to work. Rainer > rsyslog needs a few additional instructions at the top of the file to > load > the modules, but below that you can paste in the syslog.conf file. > > another thing that you can do is to make a generic rsyslog.conf file > that > includes the syslog.conf after the rsyslog specific headers. > > My rsyslog.conf (on my ubuntu 9.10 laptop) has > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser syslog > $PrivDropToGroup syslog > $RepeatedMsgReduction on > $RepeatedMsgContainsOriginalMsg on > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sun Feb 28 11:28:18 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 02:28:18 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:30:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:30:24 +0100 Subject: [rsyslog] Log Normalization effort References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397E@GRFEXC.intern.adiscon.com> David, thanks again for the good feedback. I have a couple of points: First of all: I agree that a "one size fits all approach" does not exist. Thus the idea to craft a library, which than can be used at various places: e.g. inside message parsers, output modules or even processing runs outside of rsyslog. Secondly, an efficient normalizing parser tree must not be much slower than the regular parser. I think that the parser overhead will be very acceptable for average messages. Another story, however, is the normalized data that has been gathered. In short words, that is extra data, so copy overhead is much higher everywhere. Also, accessing the properties take some time. I guess that's the primary problem inside a real-time solution, even if very efficient lookup methods are used. On the normalized properties: I think it is really worth the effort to try to define an as-broad as possible set of normalized properties. But that does not imply all needs to be done at once. It can be an evolutionary approach. First of all, one would look for pretty obvious things like traffic flow data (one may even reuse existing data models like from ipfix!) and user login/logoff activity. The key point IMHO is that we would just need to gather what people so far have used and try to get folks to re-use these fields that already exist. If we have success with this approach, I think, we will have huge benefit in the reporting and analysis area where programs could then work on standard property sets (of normalized syslog data) rather than on the raw data itself. So you do not need to write an analyzer for each of the myriad vendor/device/version formats but rather only once for the normalized data and then create a one-line (!) parse template for each of the vendor/device/version formats (of course, for each message there, so it would be multiple lines, but very easy and intuitively to do). However, all of that does not really work without community involvement. None (expect large commercial entities) can create the necessary mass of parse templates alone. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 26, 2010 10:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Log Normalization effort > > On Fri, 26 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > I have blogged about my quest for log normalization. I think there is > some > > good information on the upcoming GPLed Adiscon LogAnalyzer and future > > directions for rsyslog in the blog post. So I thought I share the > link: > > > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > > > Please note that part of the effort requires community involvement. I > would > > be very interested to learn if you think we could win enough support > to make > > this a useful effort. I am asking for your feedback, because it will > help me > > streamline my priorities for future rsyslog work. > > a few comments (but remember that I am usually dealing with high data > rates, so my concerns are biased in that direction) > > log analysis is usually done in batches as opposed to in real-time. > some of this is due to the difficulty in doing it in real time, but a > lot > of it is the processing overhead (you don't want to take so long to > process an individual request that you miss the next one to arrive) > > at low volumes the idea of name-value pairs in the logs makes a lot of > sense, but there is significantly more overhead in parsing a log with > name-value pairs in arbitrary orders than there is in using a tree > parsing > approach to analyze known log formats in a fixed order. The message > size > can also increase significantly. As a result, at high traffic volumes > this > starts to be a bad (or at least questionable) idea. > > I would love to see rsyslog gain the ability to efficiently do tree- > based > parsing instead of regex parsing. regex parsing is easy to understand > and > tinker with, but very expensive to implement. it may be that having > something that 'compiles' a list of regex parsers into a tree parser is > the right answer for usability. I would save several hours of > processing > a day if I could easily (and efficiently) make rsyslog write different > logs to different files (at high data rates and with a few hundred > conditions based variations in the syslog tag) > > > While there are some common events across different types of logs > (logins > for example) they almost always contain slightly different data in > them. I > also have no faith at all that anyone is going to make much effort to > clean up their logs to make them nicely parseable, and if they do I see > even less chance that they will end up using the same terms for the > same > thing. As such I see more value in trying to get samples of logs and > what > they mean than in trying to define a normalized version to shoehorn the > logs into. It is worth doing this for some events (logins, failed > logins > for example), but I think it's a mistake to think that this will end up > covering all, or even the majority of log messages. > > There's also a problem in that the ideal format for the output depends > on > what you are doing with the output. > > > If I could wave a magic wand and get the result I would look for > something > like this > > the parser starts at the beginning of the message (at the priority) and > can branch on priority/faclilty, timestamp, host, syslogtag, message > and > indicate if the message should be parsed into name-value pairs, or > split > based on a character (or character sequence like the perl split command > allows) into individually addressable elements (defaulting to > whitespace > separated elements), then the format (and if needed dynafile path/file > components) could be constructed from these variables. At any point in > the > parsing it should be possible to jump to another parser tree (so that > you > could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end > up > using the same parser for the message without having to redefine the > rules > for each one) > > With this capability, people could start writing parser 'branches' to > understand a specific log type and output a 'standard' format (as such > a > format can start to be defined). > > This can be done in rsyslog today, but it is fairly difficult to > define, > and as I understand it, inefficient enough that it's not practical to > do > in real-time under heavy load. > > > If this is fast enough, then the next step would be to add the ability > to > have the format/action be 'increment a counter for log type X' and a > signal to rsyslog could generate a report on these counters. Although > at > some point it becomes better to feed the message into another > opensource > tool (SEC, Simple Event Correlator for example) instead of trying to do > everything in rsyslog. > > parsing the file to know what to do with it, and be able to re-format > log > messages is very defiantly something that can fit into the rsyslog > model > of receiving, formatting, and delivering logs. Alerting on specific log > entries, counting the number of times one thing shows up in logs, and > this sort of thing start pushing > beyond the core of rsyslog, and it may be better to feed other tools > instead. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 12:25:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 12:25:14 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 28.02.10 11:28 Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sun Feb 28 14:01:58 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 05:01:58 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <001201cab868$d148da7c$100013ac@intern.adiscon.com> References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: > Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. > > I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? I am probably using the wrong terminology here. say you have a set of rules that say if programname startswith abc log to /abc if programname startswith acc log to /acc if programname startswith acc log to @1.1.1.1 if programname startswith bcd log to /bcd and assuming that these were the only possible programnames for simplicity in the explination. the filtering logic would go somthing like this the rules would compile into a tree -a-b -> /abc -c -> /acc, at 1.1.1.1 -b -> /bcd receive programname abc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "b", There are no other decisions to make, invoke the action /abc receive progranmane bcd I have no facility/PRI filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's a "b", There are no other decisions to make, invoke the action /bcd receive programname acc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "c", There are no other decisions to make, invoke the action /acc and the action @1.1.1.1 similarly the facility/pri filtering rules would be either compiled into a tree, or (since it's 256 entries total) into a lookup table with pointers to the list of actions to take for that entry the idea is to spend the time at startup to create the tree that represents the ruleset in order to speed up the processing of each individual message. the real tree would be a bit more complicated than I describe here as it needs to have 'anything else' entries between the known entries (which means that it would not be able to shortcut quite as much as I describe), and have provision for 'do a more complicated thing (like regex) here and if it matches continue'. but except for regex matches, this would make processing the rules pretty much independant of how many rules there were or how complex the ruleset is. there doe snot need to be one single programname filter tree, if you had a rule that said if pri = info and programname startswith def then def the pri tree/table would have an entry for info that would point to a filter tree for programname that just had the check for def in it am I makeing any more sense now? David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 28.02.10 11:28 > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > >>> >>> if the type is depriciated then the destination would be another >>> config_option (which you would then look up) >>> >>> if the type is 'ignore' then it would just be skipped over (possibly >>> with >>> a warning being logged that the config line is being ignored) >>> >>> this would also clean up some of the current cases where a boolean >>> option >>> doesn't honor on/off and true/false. >> >> True/false is NOT a valid boolean option. See >> >> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > I was not remembering correctly then, maby it was yes/no vs on/off. I know > I ran into something where what was documented didn't work and I changed > it to another version of a boolean answer and it fixed the problem. > >>> For the second half of the config language (telling rsyslog what to do >>> with the logs it has received) also has several variations in place and >>> is >>> showing issues. >>> >>> However I think that the right answer here is to end up implementing >>> something like the parsing trees and then compile the other config >>> language options into that tree to be consistant (and fast) under the >>> covers, no matter which way the instructions are written (except when >>> you >>> have to use regex matches) >> >> I don't fully agree here with you. For example, the traditional PRI based >> filter is lightning fast. I don't see any way it could be nearly as fast with >> any unified approach. Right now, we have a "unified" filter structure, but it >> contains three filter cases, each one adding potential power at the price of >> decreased speed. I think we need to keep different types of filters in order >> to have some lightning-fast ones. But more of this could be done under the >> hood. > > My guess/expectation is that the tree-based parsing would be about the > same speed as the traditional PRI based filter for choices made based on > PRI, slowing down for other types of conditions only in that more of the > message may need to be scanned (and if there is no selection at a level, > skipping that level should be lightning fast). As a result, a set of > filteres based soley on programname would be as fast as the current PRI > filters. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Sun Feb 28 14:15:46 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Sun, 28 Feb 2010 14:15:46 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Michael Biebl : > 2010/2/26 Rainer Gerhards : >>> The problem is, that xconsole is running at this point! >>> To reiterate: >>> I start rsyslogd >>> then xconsole >>> then try to log a message >>> The result is the debug log and no message showing up in xconsole. >> >> that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Hm, it just happened again. I rebooted my system, logged into my system, ran "sudo xconsole", "logger foo", but nothing shows up in the xconsole window. I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, "logger foo" and this time it works. Weird. There is definitely something fishy here. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Sun Feb 28 14:56:30 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 14:56:30 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com> Hi David, yes, that makes much more sense to me. Let's call that filter expression tree for the time being (the parse tree used for normalization has some similar ideas, but a totally different scope and semantics). While I now see where you are heading, I fail to see how this could sufficiently easy be implemented for a real (and complex) configuration. Let's look at this example conf: mail.* /Var/log/mail.log if programname startswith abc log to /abc if programname startswith 'acc' and MSG contains 'error' log to /acc # we don't want debug messages from here on *.debug ~ if programname startswith 'acc' or (programname contains 'bde' and (facility > 1 and facility < 5)) log to @1.1.1.1 if programname startswith bcd or MSG startswith '123' or MSG contains 'error2' log to /bcd *.* /var/log/catchall I simply cannot envision how you would store this as a tree that does not look like what we have today. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 2:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > Quickly from the mobile: pri based filters require *excatly* one > single word (!) lookup currently. You can't beat that. > > > > I am not totally sure how you think about parse trees. In my view, > parsin and filteing are two different things./stages. Maybe we have > different concepts on our minds? > > I am probably using the wrong terminology here. > > say you have a set of rules that say > > if programname startswith abc log to /abc > if programname startswith acc log to /acc > if programname startswith acc log to @1.1.1.1 > if programname startswith bcd log to /bcd > > and assuming that these were the only possible programnames for > simplicity > in the explination. > > the filtering logic would go somthing like this > > the rules would compile into a tree > > -a-b -> /abc > -c -> /acc, at 1.1.1.1 > -b -> /bcd > > receive programname abc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "b", There are no other decisions to make, > invoke the action /abc > > receive progranmane bcd > I have no facility/PRI filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's a "b", There are no other decisions to make, > invoke the action /bcd > > receive programname acc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "c", There are no other decisions to make, > invoke the action /acc and the action @1.1.1.1 > > similarly the facility/pri filtering rules would be either compiled > into a > tree, or (since it's 256 entries total) into a lookup table with > pointers > to the list of actions to take for that entry > > the idea is to spend the time at startup to create the tree that > represents the ruleset in order to speed up the processing of each > individual message. > > the real tree would be a bit more complicated than I describe here as > it > needs to have 'anything else' entries between the known entries (which > means that it would not be able to shortcut quite as much as I > describe), > and have provision for 'do a more complicated thing (like regex) here > and > if it matches continue'. but except for regex matches, this would make > processing the rules pretty much independant of how many rules there > were > or how complex the ruleset is. > > there doe snot need to be one single programname filter tree, if you > had a > rule that said > if pri = info and programname startswith def then def > > the pri tree/table would have an entry for info that would point to a > filter tree for programname that just had the check for def in it > > am I makeing any more sense now? > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 28.02.10 11:28 > > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > >>> > >>> if the type is depriciated then the destination would be another > >>> config_option (which you would then look up) > >>> > >>> if the type is 'ignore' then it would just be skipped over > (possibly > >>> with > >>> a warning being logged that the config line is being ignored) > >>> > >>> this would also clean up some of the current cases where a boolean > >>> option > >>> doesn't honor on/off and true/false. > >> > >> True/false is NOT a valid boolean option. See > >> > >> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18 > 4c0d87 > >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > > I was not remembering correctly then, maby it was yes/no vs on/off. I > know > > I ran into something where what was documented didn't work and I > changed > > it to another version of a boolean answer and it fixed the problem. > > > >>> For the second half of the config language (telling rsyslog what to > do > >>> with the logs it has received) also has several variations in place > and > >>> is > >>> showing issues. > >>> > >>> However I think that the right answer here is to end up > implementing > >>> something like the parsing trees and then compile the other config > >>> language options into that tree to be consistant (and fast) under > the > >>> covers, no matter which way the instructions are written (except > when > >>> you > >>> have to use regex matches) > >> > >> I don't fully agree here with you. For example, the traditional PRI > based > >> filter is lightning fast. I don't see any way it could be nearly as > fast with > >> any unified approach. Right now, we have a "unified" filter > structure, but it > >> contains three filter cases, each one adding potential power at the > price of > >> decreased speed. I think we need to keep different types of filters > in order > >> to have some lightning-fast ones. But more of this could be done > under the > >> hood. > > > > My guess/expectation is that the tree-based parsing would be about > the > > same speed as the traditional PRI based filter for choices made based > on > > PRI, slowing down for other types of conditions only in that more of > the > > message may need to be scanned (and if there is no selection at a > level, > > skipping that level should be lightning fast). As a result, a set of > > filteres based soley on programname would be as fast as the current > PRI > > filters. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 15:01:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 15:01:54 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Sunday, February 28, 2010 2:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Michael Biebl : > > 2010/2/26 Rainer Gerhards : > >>> The problem is, that xconsole is running at this point! > >>> To reiterate: > >>> I start rsyslogd > >>> then xconsole > >>> then try to log a message > >>> The result is the debug log and no message showing up in xconsole. > >> > >> that's what I am doing as well. I will try to record the session. > > > > Wait a sec. > > With 4.4.2 (or any earlier version) I tried this: > > start rsyslog > > start xconsole > > xconsole shows the last two log messages immediately > > > > With 4.6.0 the xconsole window is blank (which I guess made me > believe > > it blocked) > > > > With 4.6.0, I need to trigger a new log message (e.g. using logger > > foo), and then xconsole will show a new message. > > > > So, xconsole does indeed show messages, but the behaviour is > different > > to 4.4 and earlier > > Hm, it just happened again. > I rebooted my system, logged into my system, ran "sudo xconsole", > "logger foo", but nothing shows up in the xconsole window. > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > "logger foo" and this time it works. > > Weird. There is definitely something fishy here. OK, rather than trying to figure out what may be going on here, I'll see that I copy the old omfile code and create a new ompipe. So that will behave exactly like it did before. The new optimizations are not so important for pipes. Rainer From rgerhards at hq.adiscon.com Sun Feb 28 17:33:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 17:33:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com> > > Hm, it just happened again. > > I rebooted my system, logged into my system, ran "sudo xconsole", > > "logger foo", but nothing shows up in the xconsole window. > > > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > > "logger foo" and this time it works. > > > > Weird. There is definitely something fishy here. > > OK, rather than trying to figure out what may be going on here, I'll > see that > I copy the old omfile code and create a new ompipe. So that will behave > exactly like it did before. The new optimizations are not so important > for > pipes. Michael, I have done this now. I would appreciate if you could checkout the v4-stable git head and try with that one. It should now have the exact same behavior as in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked like before. Thinking about it, I guess it helps ease development in the future, because I now do not need to think about pipes when I work on omfile and vice versa. Rainer From tbergfeld at hq.adiscon.com Fri Feb 5 15:41:32 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 5 Feb 2010 15:41:32 +0100 Subject: [rsyslog] rsyslog 5.5.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103822@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.2, a member of the v5-devel branch. Rsyslog 5.5.2 primarily provides bug-fixes, but offers an enhanced functionality to escape 8-bit characters on reception. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-193.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From ryan.b.lynch at gmail.com Fri Feb 5 23:07:26 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 5 Feb 2010 17:07:26 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Under 5.3.6, I am trying to use property-based filtering to separate local messages (coming in via imuxsock, presumably) from remote messages (coming in via UDP). Initially, I tried using the 'inputname' property to distinguish between the two: :inputname, isequal, "imudp" /remote_udp.log :inputname, isequal, "imuxsock" /local_socket.log With this config, nothing ever ends up in the 2nd file (/local_socket.log)--instead, because the 'inputname' property is always set to 'imudp', regardless of where the input originates. The property replacer docs mention that the value of 'inputname' isn't guaranteed, so I guess this isn't necessarily a bug. Also, I found a comment in the source code for 'imuxsock.c' that makes this behavior seem to be intentional. Why does only imuxsock behave like this? The other input modules I've used (imrelp, imklog) provide the expected 'inputname' values ('imrelp' and 'imklog', respectively), which is intuitive and consistent. Is there a reason why imuxsock needs to be different, or is this a bug? (In case anyone is wondering, I could work around this by comparing the 'hostname' and '$myhostname' properties, in addition to checking 'inputname'. But that would force me to use expression-based filters (property-based filtering doesn't support AND/OR logic), which seems to increase the CPU activity per message. I really can't afford the lost efficiency, here, if it can be helped.) Ryan B. Lynch ryan.b.lynch at gmail.com From ryan.b.lynch at gmail.com Sat Feb 6 06:13:17 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 00:13:17 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. Message-ID: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> I'm trying to use multiple rulesets and omruleset actions to send a single incoming log message to two different log files, at the same time. I think my config file will help explain it: # BEGIN rsyslog.conf $ModLoad imuxsock $ModLoad imklog $ModLoad immark $ModLoad omruleset $MarkMessagePeriod 1 $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% %timereported:::date-rfc3339% %hostname% %fromhost% %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% %msg%\n" # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', each of which writes output to its own, separate file ('RS_Alfa.log' and 'RS_Bravo.log', respectively) using an action with a *.* selector. $Ruleset RS_Bravo $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset $Ruleset RS_Alfa $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset # SECOND: I create a pair of actions (in the default ruleset), both with *.* selectors and the action ':omruleset:', but preceded by different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and 'RS_Bravo', respectively $ActionOmrulesetRulesetName RS_Bravo $ActionResumeInterval 1 *.* :omruleset: $ActionOmrulesetRulesetName RS_Alfa $ActionResumeInterval 1 *.* :omruleset: # END rsyslog.conf Based on the documentation, I believe that incoming log messages should be processed by the default ruleset, which sends the message to both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I should see a copy of each log message written to each of the two output files, and each output file should have identical contents. But that isn't the result I'm getting, here. I get two copies of each message, but almost all of them are written to the file 'RS_Alfa'. Only the occasional lone random message is written to 'RS_Bravo'. (If I reverse the order of the :omrulset: actions, I see the oppositie pattern.) The exact proportion varies, but only about 1 in 20 messages makes it to the 1st (Bravo) output file. All different types of messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' file--again, it seems completely random to me. I never get more total messages logged than I should, though. If I run rsyslog long enough to let 10 incoming messages appear, I will see a total of 20 messages when I combine the contents of 'RS_Alfa' and 'RS_Bravo'--they just end up in the wrong files. I turned on debug logging during a short session and captured the results to a file, which I've attached, along with the rsyslog output files from that session. The host in question runs Centos Linux 5.4 as an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. If there is any more info I can provide, or any other debugging steps I should take, please let me know. Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Feb 6 12:00:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:00:59 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> I will check, but the most probably explanation is that it is a bug that so far nobody noticed ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Friday, February 05, 2010 11:07 PM > To: rsyslog-users > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > as'imudp'? > > Under 5.3.6, I am trying to use property-based filtering to separate > local messages (coming in via imuxsock, presumably) from remote > messages (coming in via UDP). Initially, I tried using the 'inputname' > property to distinguish between the two: > > :inputname, isequal, "imudp" /remote_udp.log > :inputname, isequal, "imuxsock" /local_socket.log > > With this config, nothing ever ends up in the 2nd file > (/local_socket.log)--instead, because the 'inputname' property is > always set to 'imudp', regardless of where the input originates. > > The property replacer docs mention that the value of 'inputname' isn't > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > comment in the source code for 'imuxsock.c' that makes this behavior > seem to be intentional. > > Why does only imuxsock behave like this? The other input modules I've > used (imrelp, imklog) provide the expected 'inputname' values > ('imrelp' and 'imklog', respectively), which is intuitive and > consistent. Is there a reason why imuxsock needs to be different, or > is this a bug? > > (In case anyone is wondering, I could work around this by comparing > the 'hostname' and '$myhostname' properties, in addition to checking > 'inputname'. But that would force me to use expression-based filters > (property-based filtering doesn't support AND/OR logic), which seems > to increase the CPU activity per message. I really can't afford the > lost efficiency, here, if it can be helped.) > > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sat Feb 6 12:04:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:04:18 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Indeed, it was a bug. Patch here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 d5305b1f6cc0fe22 Thanks for reporting the issue! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Saturday, February 06, 2010 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > propertyas'imudp'? > > I will check, but the most probably explanation is that it is a bug > that so > far nobody noticed ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Friday, February 05, 2010 11:07 PM > > To: rsyslog-users > > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > > as'imudp'? > > > > Under 5.3.6, I am trying to use property-based filtering to separate > > local messages (coming in via imuxsock, presumably) from remote > > messages (coming in via UDP). Initially, I tried using the > 'inputname' > > property to distinguish between the two: > > > > :inputname, isequal, "imudp" /remote_udp.log > > :inputname, isequal, "imuxsock" /local_socket.log > > > > With this config, nothing ever ends up in the 2nd file > > (/local_socket.log)--instead, because the 'inputname' property is > > always set to 'imudp', regardless of where the input originates. > > > > The property replacer docs mention that the value of 'inputname' > isn't > > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > > comment in the source code for 'imuxsock.c' that makes this behavior > > seem to be intentional. > > > > Why does only imuxsock behave like this? The other input modules I've > > used (imrelp, imklog) provide the expected 'inputname' values > > ('imrelp' and 'imklog', respectively), which is intuitive and > > consistent. Is there a reason why imuxsock needs to be different, or > > is this a bug? > > > > (In case anyone is wondering, I could work around this by comparing > > the 'hostname' and '$myhostname' properties, in addition to checking > > 'inputname'. But that would force me to use expression-based filters > > (property-based filtering doesn't support AND/OR logic), which seems > > to increase the CPU activity per message. I really can't afford the > > lost efficiency, here, if it can be helped.) > > > > Ryan B. Lynch > > ryan.b.lynch at gmail.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From ryan.b.lynch at gmail.com Sat Feb 6 19:27:52 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 13:27:52 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> No problem, and thank you for the patch. I'll try it out and let you know how it works. This is a huge help to me, BTW: I should be able to get my log server working properly, now, without having to worry about omruleset and multiple rulesets. (Hence my other problem report from last night.) Big sigh of relief. Not that I don't like omruleset. Like RELP, it feels really elegant, and it's easy to expressive complex log routing arrangements. I think it's one of your best ideas, to date, and I'm looking forward to trying out some crazy ideas with it. -Ryan On 2010-02-06, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 > > Thanks for reporting the issue! > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Saturday, February 06, 2010 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' >> propertyas'imudp'? >> >> I will check, but the most probably explanation is that it is a bug >> that so >> far nobody noticed ;) >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch >> > Sent: Friday, February 05, 2010 11:07 PM >> > To: rsyslog-users >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' property >> > as'imudp'? >> > >> > Under 5.3.6, I am trying to use property-based filtering to separate >> > local messages (coming in via imuxsock, presumably) from remote >> > messages (coming in via UDP). Initially, I tried using the >> 'inputname' >> > property to distinguish between the two: >> > >> > :inputname, isequal, "imudp" /remote_udp.log >> > :inputname, isequal, "imuxsock" /local_socket.log >> > >> > With this config, nothing ever ends up in the 2nd file >> > (/local_socket.log)--instead, because the 'inputname' property is >> > always set to 'imudp', regardless of where the input originates. >> > >> > The property replacer docs mention that the value of 'inputname' >> isn't >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found a >> > comment in the source code for 'imuxsock.c' that makes this behavior >> > seem to be intentional. >> > >> > Why does only imuxsock behave like this? The other input modules I've >> > used (imrelp, imklog) provide the expected 'inputname' values >> > ('imrelp' and 'imklog', respectively), which is intuitive and >> > consistent. Is there a reason why imuxsock needs to be different, or >> > is this a bug? >> > >> > (In case anyone is wondering, I could work around this by comparing >> > the 'hostname' and '$myhostname' properties, in addition to checking >> > 'inputname'. But that would force me to use expression-based filters >> > (property-based filtering doesn't support AND/OR logic), which seems >> > to increase the CPU activity per message. I really can't afford the >> > lost efficiency, here, if it can be helped.) >> > >> > Ryan B. Lynch >> > ryan.b.lynch at gmail.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Mon Feb 8 09:06:58 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 09:06:58 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103828@GRFEXC.intern.adiscon.com> Ryan, thanks for the feedback. I'll also try to look into the omruleset issue you reported. Looked too complex for the weekend ;) More later... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, February 06, 2010 7:28 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > property as'imudp'? > > No problem, and thank you for the patch. I'll try it out and let you > know how it works. > > This is a huge help to me, BTW: I should be able to get my log server > working properly, now, without having to worry about omruleset and > multiple rulesets. (Hence my other problem report from last night.) > Big sigh of relief. > > Not that I don't like omruleset. Like RELP, it feels really elegant, > and it's easy to expressive complex log routing arrangements. I think > it's one of your best ideas, to date, and I'm looking forward to > trying out some crazy ideas with it. > > -Ryan > > > On 2010-02-06, Rainer Gerhards wrote: > > Indeed, it was a bug. Patch here: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d0 > 54dac0 > > d5305b1f6cc0fe22 > > > > Thanks for reporting the issue! > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Saturday, February 06, 2010 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > >> propertyas'imudp'? > >> > >> I will check, but the most probably explanation is that it is a bug > >> that so > >> far nobody noticed ;) > >> > >> Rainer > >> > >> > -----Original Message----- > >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > >> > Sent: Friday, February 05, 2010 11:07 PM > >> > To: rsyslog-users > >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' > property > >> > as'imudp'? > >> > > >> > Under 5.3.6, I am trying to use property-based filtering to > separate > >> > local messages (coming in via imuxsock, presumably) from remote > >> > messages (coming in via UDP). Initially, I tried using the > >> 'inputname' > >> > property to distinguish between the two: > >> > > >> > :inputname, isequal, "imudp" /remote_udp.log > >> > :inputname, isequal, "imuxsock" /local_socket.log > >> > > >> > With this config, nothing ever ends up in the 2nd file > >> > (/local_socket.log)--instead, because the 'inputname' property is > >> > always set to 'imudp', regardless of where the input originates. > >> > > >> > The property replacer docs mention that the value of 'inputname' > >> isn't > >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found > a > >> > comment in the source code for 'imuxsock.c' that makes this > behavior > >> > seem to be intentional. > >> > > >> > Why does only imuxsock behave like this? The other input modules > I've > >> > used (imrelp, imklog) provide the expected 'inputname' values > >> > ('imrelp' and 'imklog', respectively), which is intuitive and > >> > consistent. Is there a reason why imuxsock needs to be different, > or > >> > is this a bug? > >> > > >> > (In case anyone is wondering, I could work around this by > comparing > >> > the 'hostname' and '$myhostname' properties, in addition to > checking > >> > 'inputname'. But that would force me to use expression-based > filters > >> > (property-based filtering doesn't support AND/OR logic), which > seems > >> > to increase the CPU activity per message. I really can't afford > the > >> > lost efficiency, here, if it can be helped.) > >> > > >> > Ryan B. Lynch > >> > ryan.b.lynch at gmail.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 10:03:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 10:03:14 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265619794.5427.490.camel@localhost> Hi Ryan, once again thanks for the bug report. I could confirm the problem and opend a bug tracker (it may make sense to subscribe to it): http://bugzilla.adiscon.com/show_bug.cgi?id=179 I am now digging into the details. Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From sergei at bslos.com Mon Feb 8 13:18:41 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 17:18:41 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 Message-ID: <201002081718.41670.sergei@bslos.com> Hi, I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit system): $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ --enable-omprog ... checking whether the compiler provides atomic builtins... no checking whether the compiler provides atomic builtins for 64 bit data types... no ... Configure script run without errors. $ make ... CC rsyslogd-omfile.o In file included from omfile.c:67: ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not available, using nul operations - rsyslogd will probably be racy!" omfile.c: In function 'getClockFileAccess': omfile.c:95: warning: implicit declaration of function 'ATOMIC_INC_AND_FETCH' CC rsyslogd-omdiscard.o CC rsyslogd-pmrfc5424.o CC rsyslogd-pmrfc3164.o CC rsyslogd-iminternal.o CC rsyslogd-pidfile.o pidfile.c: In function 'read_pid': pidfile.c:58: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result pidfile.c: In function 'write_pid': pidfile.c:116: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result CCLD rsyslogd rsyslogd-omfile.o: In function `getClockFileAccess': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function `SetGlobalInputTermination': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/glbl.c:136: undefined reference to `ATOMIC_STORE_1_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function `msgDestruct': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/msg.c:802: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function `wtiSetState': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:109: undefined reference to `ATOMIC_STORE_1_TO_INT' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:111: undefined reference to `ATOMIC_STORE_0_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function `DoDeleteBatchFromQStore': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: more undefined references to `ATOMIC_SUB' follow collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' make: *** [all] Error 2 Any help is appreciated. -- Regards, Sergei Butakov From rgerhards at hq.adiscon.com Mon Feb 8 13:53:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 13:53:55 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> this looks like you have generated code for a machine which does not support atomic instructions. rsyslog needs them to work correctly. I suggest generating code for i686. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > Sent: Monday, February 08, 2010 1:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > Hi, > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > system): > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ > --enable-omprog > ... > checking whether the compiler provides atomic builtins... no > checking whether the compiler provides atomic builtins for 64 bit data > types... no > ... > > Configure script run without errors. > > > > $ make > ... > CC rsyslogd-omfile.o > In file included from omfile.c:67: > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > available, > using nul operations - rsyslogd will probably be racy!" > omfile.c: In function 'getClockFileAccess': > omfile.c:95: warning: implicit declaration of function > 'ATOMIC_INC_AND_FETCH' > CC rsyslogd-omdiscard.o > CC rsyslogd-pmrfc5424.o > CC rsyslogd-pmrfc3164.o > CC rsyslogd-iminternal.o > CC rsyslogd-pidfile.o > pidfile.c: In function 'read_pid': > pidfile.c:58: warning: ignoring return value of 'fscanf', declared with > attribute warn_unused_result > pidfile.c: In function 'write_pid': > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > with > attribute warn_unused_result > CCLD rsyslogd > rsyslogd-omfile.o: In function `getClockFileAccess': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > `SetGlobalInputTermination': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/glbl.c:136: > undefined reference to `ATOMIC_STORE_1_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > `msgDestruct': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/msg.c:802: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > `wtiSetState': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:109: > undefined reference to `ATOMIC_STORE_1_TO_INT' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:111: > undefined reference to `ATOMIC_STORE_0_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > `DoDeleteBatchFromQStore': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > ../runtime/.libs/librsyslog.a(librsyslog_la- > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > more undefined references to `ATOMIC_SUB' follow > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > make: *** [all] Error 2 > > > > Any help is appreciated. > > > -- > Regards, > Sergei Butakov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 14:00:20 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 14:00:20 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265634020.5427.491.camel@localhost> Ryan, thanks again for the bug report. I managed to fix it. Details in the tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=179 Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From rgerhards at hq.adiscon.com Mon Feb 8 14:10:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 14:10:54 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> see also at the bottom of this page: http://www.rsyslog.com/doc-build_from_repo.html Raienr > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, February 08, 2010 1:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] can't compile rsyslog 5.3.7 > > this looks like you have generated code for a machine which does not > support > atomic instructions. rsyslog needs them to work correctly. I suggest > generating code for i686. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > > Sent: Monday, February 08, 2010 1:19 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > > > Hi, > > > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > > system): > > > > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile > \ > > --enable-omprog > > ... > > checking whether the compiler provides atomic builtins... no > > checking whether the compiler provides atomic builtins for 64 bit > data > > types... no > > ... > > > > Configure script run without errors. > > > > > > > > $ make > > ... > > CC rsyslogd-omfile.o > > In file included from omfile.c:67: > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > > available, > > using nul operations - rsyslogd will probably be racy!" > > omfile.c: In function 'getClockFileAccess': > > omfile.c:95: warning: implicit declaration of function > > 'ATOMIC_INC_AND_FETCH' > > CC rsyslogd-omdiscard.o > > CC rsyslogd-pmrfc5424.o > > CC rsyslogd-pmrfc3164.o > > CC rsyslogd-iminternal.o > > CC rsyslogd-pidfile.o > > pidfile.c: In function 'read_pid': > > pidfile.c:58: warning: ignoring return value of 'fscanf', declared > with > > attribute warn_unused_result > > pidfile.c: In function 'write_pid': > > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > > with > > attribute warn_unused_result > > CCLD rsyslogd > > rsyslogd-omfile.o: In function `getClockFileAccess': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > > `SetGlobalInputTermination': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/glbl.c:136: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > > `msgDestruct': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/msg.c:802: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > > `wtiSetState': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:109: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:111: > > undefined reference to `ATOMIC_STORE_0_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > > `DoDeleteBatchFromQStore': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > ../runtime/.libs/librsyslog.a(librsyslog_la- > > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > more undefined references to `ATOMIC_SUB' follow > > collect2: ld returned 1 exit status > > make[2]: *** [rsyslogd] Error 1 > > make[2]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > > make: *** [all] Error 2 > > > > > > > > Any help is appreciated. > > > > > > -- > > Regards, > > Sergei Butakov > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From sergei at bslos.com Mon Feb 8 15:11:32 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 19:11:32 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> Message-ID: <201002081911.33210.sergei@bslos.com> On Monday 08 February 2010, Rainer Gerhards wrote: > see also at the bottom of this page: > > http://www.rsyslog.com/doc-build_from_repo.html Thank you! The adding CFLAGS="-march=i686" to the configure script solved my trouble. -- Regards, Sergei Butakov From ryan.b.lynch at gmail.com Mon Feb 8 23:10:35 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:10:35 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Message-ID: <115906d11002081410v17596b5eg6a1fab390aad04d6@mail.gmail.com> On Sat, Feb 6, 2010 at 06:04, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 Works for me with 5.3.7, with different line alignment in the patch. That might be due to 'rpmbuild' and its built-in patch macros. In case it saves anyone else time, I've attached a copy of what works for me. -Ryan From ryan.b.lynch at gmail.com Mon Feb 8 23:16:25 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:16:25 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <1265634020.5427.491.camel@localhost> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> Message-ID: <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards wrote: > Ryan, > > thanks again for the bug report. I managed to fix it. Details in the > tracker: > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 Works fine for me with 5.3.7. Thanks! From rgerhards at hq.adiscon.com Tue Feb 9 07:43:48 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 9 Feb 2010 07:43:48 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710383F@GRFEXC.intern.adiscon.com> Thanks for the feedback. I think I can do the "official" merge today :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Monday, February 08, 2010 11:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] Possible bug in 5.3.6: Multiple > $Ruleset,:omruleset cause random output destination selection. > > On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards > wrote: > > Ryan, > > > > thanks again for the bug report. I managed to fix it. Details in the > > tracker: > > > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 > > Works fine for me with 5.3.7. Thanks! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikolaj at kucharski.name Wed Feb 10 13:02:30 2010 From: mikolaj at kucharski.name (Mikolaj Kucharski) Date: Wed, 10 Feb 2010 12:02:30 +0000 Subject: [rsyslog] last message repeated n times from remote hosts Message-ID: <20100210120230.GI3838@x40.openbsd.home.lan> Hi, I have few Linux based machines with sysklogd installed, and I have central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5. I have issue with missing hostname when sysklog sends "last message repeated N times" over the wire to rsyslog server. Let's have a look. I used logger(1) to repeatedly sent one message few times followed by one different message. Here is what I see in the log file on my central rsyslog server: Feb 10 11:39:46 10.101.43.124 root: remote test start Feb 10 11:39:54 last message repeated 14 times Feb 10 11:39:54 10.101.43.124 root: remote test end and here is tcpdump(8) log from the source (10.101.43.124) machine: 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28 0x0000: 4500 0038 0000 4000 4011 d1fb 0a65 2b7c E..8.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0024 68ef 3c31 333e .e(t.....$h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 7374 6172 740a t.start. 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35 0x0000: 4500 003f 0000 4000 4011 d1f4 0a65 2b7c E..?.. at .@....e+| 0x0010: 0a65 2874 0202 0202 002b 68f6 3c31 333e .e(t.....+h.<13> 0x0020: 6c61 7374 206d 6573 7361 6765 2072 6570 last.message.rep 0x0030: 6561 7465 6420 3134 2074 696d 6573 0a eated.14.times. 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26 0x0000: 4500 0036 0000 4000 4011 d1fd 0a65 2b7c E..6.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0022 68ed 3c31 333e .e(t....."h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 656e 640a t.end. I searched the list, and saw a comment which say the fault is on the sysklogd end as it never sends hostname in the repeated-n-times packet, but from above I cannot see that it ever sends the packet with hostname, so I think the issues is on rsyslog side, and not on the sysklogd. Could someone shed some light on my issue, as I would like to see all the time the source IP or hostname of incomming messages to rsyslog daemon. Is this missing source hostname/IP a bug of rsyslog? Is there any way to workaround that? Thanks. PS1. I cannot change client machines, I cannot reinstall them with different syslog implementation, the only machine where I have permission to do modifications is central rsyslog server. PS2. I know about DNS and RevDNS and yes, above server doesn't have revDNS setup. -- best regards q# From tbergfeld at hq.adiscon.com Wed Feb 10 15:31:17 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 10 Feb 2010 15:31:17 +0100 Subject: [rsyslog] rsyslog 4.5.8 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710385E@GRFEXC.intern.adiscon.com> Hi all, We have just release rsyslog 4.5.8, a member of the v4-beta branch. This is a bug-fixing release, fixing all those issues that have been reported the past weeks. Full details are available in the change log. This is scheduled to replace the current v4-stable, and conclude the v4 development efforts. So we would appreciate if you could give it a try and report any issues you see with it. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-194.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Feb 16 10:13:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 16 Feb 2010 10:13:14 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Hi folks, almost by accident, I learned today that rsyslog seems to be the default syslogd for SuSe as well: http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 I am a bit surprised I never heared about that change, but I have to admit I am quite happy that rsyslog now seems to be the default syslogd on (almost) all important distros... I just thought I share the good news. Rainer From mbiebl at gmail.com Tue Feb 16 15:29:57 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 16 Feb 2010 15:29:57 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/16 Rainer Gerhards : > Hi folks, > > almost by accident, I learned today that rsyslog seems to be the default > syslogd for SuSe as well: > > http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 > > I am a bit surprised I never heared about that change, but I have to admit I > am quite happy that rsyslog now seems to be the default syslogd on (almost) > all important distros... > > I just thought I share the good news. Congrats Rainer for your hard work and persistence! Let's welcome our openSUSE friends on board :-) Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Sun Feb 21 20:31:40 2010 From: reader at newsguy.com (Harry Putnam) Date: Sun, 21 Feb 2010 13:31:40 -0600 Subject: [rsyslog] [basic] No remote logs getting thru Message-ID: <87iq9qmllv.fsf@newsguy.com> Not sure if the localmachine is not sending or the remote not receiving. But it appears the logs expected to show up on the remote are not there. Both ends are gentoo linux, I'm not at all familiar with rsyslog but hoped I set it up right according to what I see on the example file and at: http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications Local and remote configs appear in full [inlined] at the end. Just a few lines (including commented ones) from local conf that I hoped were what I needed to send the logs: # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host The above last 2 commented lines would be less confusing of they used the same notation for remote host in both instances. remote host remote-host Appears to be talking about two different things *.* @@remote-host So is that really all it needs? ------- --------- ---=--- --------- -------- Now a few lines from the remote rsyslog.conf # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. That is all it appears to indicate is needed to receive logs. And this (the remote) instance of rsyslog is being started with -c3 -t514 flags. ------- --------- ---=--- --------- -------- Complete LOCAL rsyslog conf: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional # remote host is: # logsrv:512 # e.g. 192.168.0.1:514, port optional *.* @@192.168.0.26:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 ------- --------- ---=--- --------- -------- Complete remote rsyslog.conf $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 From mrdemeanour at jackpot.uk.net Mon Feb 22 07:27:28 2010 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 22 Feb 2010 06:27:28 +0000 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87iq9qmllv.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <4B8223D0.8040503@jackpot.uk.net> Harry Putnam wrote: > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. Hi, You seem to have all of the output ('action') configuration on the client machine - that's all the stuff starting with *.*, local1.* etc. As a consequence only the client machine is ever going to output anything. Please verify that the client machine is indeed logging to disk in /var/log. If you now set up a catch-all action line in the server's rsyslog.conf (and restart the service on both machines), I prophesy that output originating from the client will start appearing on the server. E.g.: *.* /var/log/catchall.log You can test using the logger command on the client, e.g. $ logger -p local1.info "Test log message from client" -- Jack. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 22 08:40:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 08:40:49 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Harry, I think this may contain useful information for you: http://cookbook.rsyslog.com/node13.html there is also a guide on receiving messages and putting them to different bins. Looking briefly at your config, I'd suggest that you use explicit port numbers (e.g. 10514) and make sure they are enabled in our firewall. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Sunday, February 21, 2010 8:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [basic] No remote logs getting thru > > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_-- > _starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support > it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Mon Feb 22 17:25:02 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 10:25:02 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Message-ID: <87wry5kzkx.fsf@newsguy.com> "Rainer Gerhards" writes: > Harry, > > I think this may contain useful information for you: > > http://cookbook.rsyslog.com/node13.html From:http://cookbook.rsyslog.com/node14.html *.* @@192.0.2.1:10514 # if you need to forward to other systems as well, just # add additional config lines: *.* @@other-server.example.net:10514 I must be missing something... (I am a novice with this) but I don't see anything different in the suggested config from what I've already posted in the client config. Except it shows a second remote server being sent logs... which I'm not trying to do. Also what is needed in the server to make rsyslog listen to port 514? As I've mentioned, that (server) instance of rsyslog is being started with -t514. I've seen nothing so far either in the stock rsyslog.conf example configs or at: http://cookbook.rsyslog.com/node13.html to indicate some better or more precise way to get the server to listen on 514 On the server: `ps wwaux' shows how the server is running: ps wwaux|grep rsyslog (wrapped for mail) root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ /var/run/rsyslogd.pid -f /etc/rsyslog.conf There are no firewalls involved here... its a local network behind a firewall router so all rsyslog traffic is only on the local side. I'm not sure if it means anything but trying: telnet server_host 514 from the client machine does fail to connect. And the attempt does not show up in the catchall ouput on server either. From rgerhards at hq.adiscon.com Mon Feb 22 18:51:36 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 18:51:36 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Monday, February 22, 2010 5:25 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] [basic] No remote logs getting thru > > "Rainer Gerhards" writes: > > > Harry, > > > > I think this may contain useful information for you: > > > > http://cookbook.rsyslog.com/node13.html > > From:http://cookbook.rsyslog.com/node14.html > > *.* @@192.0.2.1:10514 > # if you need to forward to other systems as well, just > # add additional config lines: > *.* @@other-server.example.net:10514 > > I must be missing something... (I am a novice with this) but I don't > see anything different in the suggested config from what I've already > posted in the client config. Except it shows a second remote server > being sent logs... which I'm not trying to do. > > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. > > I've seen nothing so far either in the stock rsyslog.conf example > configs or at: http://cookbook.rsyslog.com/node13.html to indicate > some better or more precise way to get the server to listen on 514 see http://cookbook.rsyslog.com/node24.html and http://www.rsyslog.com/doc-imtcp.html Rainer > > On the server: > > `ps wwaux' shows how the server is running: > > ps wwaux|grep rsyslog (wrapped for mail) > > root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ > 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ > /var/run/rsyslogd.pid -f /etc/rsyslog.conf > > There are no firewalls involved here... its a local network behind a > firewall router so all rsyslog traffic is only on the local side. > > I'm not sure if it means anything but trying: > telnet server_host 514 > > from the client machine does fail to connect. And the attempt does > not show up in the catchall ouput on server either. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Mon Feb 22 18:46:26 2010 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 22 Feb 2010 10:46:26 -0700 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87wry5kzkx.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. You loaded the module but I don't see anything explicitly starting inputTCP (not sure if this is required or not). In my config I have: $ModLoad imtcp.so # load module $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 I also have: # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 Generally - there's not much harm running both afaik and it makes it easier when you have a device that doesn't do TCP (network gear). > I'm not sure if it means anything but trying: > ?telnet server_host 514 > > from the client machine does fail to connect. ?And the attempt does > not show up in the catchall ouput on server either. Yeah, that's. bad. Try the above, and if you still cannot connect remotely run this on the syslog server: sudo netstat -alnp | grep 514 You may get some other stuff but you should get a line that looks like this: tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 29233/rsyslogd That tells you rsyslogd is listening on port 514 - if you did all of the above you should also see: udp 0 0 0.0.0.0:514 0.0.0.0:* 29233/rsyslogd Which tells you it's listening on udp 514. Hope this helps, Aaron From reader at newsguy.com Mon Feb 22 21:41:42 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 14:41:42 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> Message-ID: <87k4u5j94p.fsf@newsguy.com> "Rainer Gerhards" writes: [...] >> I must be missing something... (I am a novice with this) but I don't >> see anything different in the suggested config from what I've already >> posted in the client config. Except it shows a second remote server >> being sent logs... which I'm not trying to do. [...] >> I've seen nothing so far either in the stock rsyslog.conf example >> configs or at: http://cookbook.rsyslog.com/node13.html to indicate >> some better or more precise way to get the server to listen on 514 > > see http://cookbook.rsyslog.com/node24.html > and http://www.rsyslog.com/doc-imtcp.html Thanks for your patience Rainer... ok I get it now. But see comment at the end. Oh, and what about that dashed notation in the example file like: *.info;mail.none;authpriv.none;cron.none -/var/log/messages What role does the dash play? [...] Aaron Nichols writes: > On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: >> Also what is needed in the server to make rsyslog listen to port 514? >> As I've mentioned, that (server) instance of rsyslog is being started >> with -t514. > > You loaded the module but I don't see anything explicitly starting > inputTCP (not sure if this is required or not). In my config I have: > > $ModLoad imtcp.so # load module > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 Yup, those were right on the money, and of course I do see the info at `netstat -alnp | grep 514' you talked about... its listening now, and with a restart at both ends... logs info is now comming across. Many thanks... you input proved to be right on the money. Oh, do you know if there are many things that can only log by udp? From reader at newsguy.com Tue Feb 23 01:33:05 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 18:33:05 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> Message-ID: <87fx4skczi.fsf@newsguy.com> Harry Putnam writes: > Thanks for your patience Rainer... ok I get it now. But see comment > at the end. After saying that about something at the end.... I forgot to include it. I was just wondering why the example config that comes with the rsyslog distro leaves out such important stuff as what it takes to get the server listening for log messages. It has a commented section that claims to be about: `Receiving Messages from Remote Hosts' # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 But it doesn't bother to include the things that make tcp work, like the all important lines posted by Aaron N. $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 Again thank you both for your input and for website too Rainer. From ktm at rice.edu Tue Feb 23 03:03:19 2010 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 22 Feb 2010 20:03:19 -0600 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87fx4skczi.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> Message-ID: <20100223020319.GA21475@it.is.rice.edu> On Mon, Feb 22, 2010 at 06:33:05PM -0600, Harry Putnam wrote: > Harry Putnam writes: > > > Thanks for your patience Rainer... ok I get it now. But see comment > > at the end. > > After saying that about something at the end.... I forgot to include > it. > > I was just wondering why the example config that comes with the > rsyslog distro leaves out such important stuff as what it takes to get > the server listening for log messages. > > It has a commented section that claims to be about: > `Receiving Messages from Remote Hosts' > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > But it doesn't bother to include the things that make tcp work, like > the all important lines posted by Aaron N. > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 > > Again thank you both for your input and for website too Rainer. > Maybe the sample configuration files are for an earlier release of rsyslog. Cheers, Ken From reader at newsguy.com Tue Feb 23 16:02:24 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:02:24 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <87r5ochu67.fsf@newsguy.com> Kenneth Marshall writes: [...] Harry wrote: >> I was just wondering why the example config that comes with the >> rsyslog distro leaves out such important stuff as what it takes to get >> the server listening for log messages. [...] Harry wrote: >> But it doesn't bother to include the things that make tcp work, like >> the all important lines posted by Aaron N. >> >> $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) >> $InputTCPServerRun 514 # start up TCP listener at port 514 [...] Kenneth wrote: > Maybe the sample configuration files are for an earlier release of > rsyslog. You mean like maybe the version advanced and nobody bothered to update the example conf? Yeah that might be what happened but even so, it seems you'd want the example conf shipped with a version to match the version. What I'm looking at was installed with version 3.22.1 From reader at newsguy.com Tue Feb 23 16:11:18 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:11:18 -0600 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <87mxz0htrd.fsf@newsguy.com> Just getting started with rsyslog and finding the documentation pretty confusing. Probably not the fault of the documentation but more my lack of knowledge. What I'd like to do is have rsyslog create a directory for incoming foreign logs for each host that sends syslog info. >From there, its ok if it all goes in a single log, inside the directory for now, and I may work out some splitting scheme later. So far, with help from posters here I've managed to get the logs comming in from remote machines and logging to a file named after the host. ... I suspect I should be able to figure it out from the pages here (line wrapped for mail): http://wiki.rsyslog.com/index.php \ /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory $template DYNmessages,"/var/log/%HOSTNAME%/messages" But I'd want the directory named for the host and include all logs from that host, not just the `messages' log. Not quite getting how to edit that line to make it happen. From anichols at trumped.org Tue Feb 23 16:49:48 2010 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 23 Feb 2010 08:49:48 -0700 Subject: [rsyslog] How to make rsyslog create dir per host In-Reply-To: <87mxz0htrd.fsf@newsguy.com> References: <87mxz0htrd.fsf@newsguy.com> Message-ID: On Tue, Feb 23, 2010 at 8:11 AM, Harry Putnam wrote: > What I'd like to do is have rsyslog create a directory for incoming > foreign logs for each host that sends syslog info. [...] > I suspect I should be able to figure it out from the pages here (line > wrapped for mail): > ?http://wiki.rsyslog.com/index.php \ > ? /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > ?$template DYNmessages,"/var/log/%HOSTNAME%/messages" > > But I'd want the directory named for the host and include all logs from > that host, not just the `messages' log. You want to get familiar with templates and actions - there are two pieces to this. Rather than tell you exactly how to solve your problem you should read this: http://www.rsyslog.com/doc-rsyslog_conf_templates.html and this: http://www.rsyslog.com/doc-rsyslog_conf_actions.html And bookmark this: http://www.rsyslog.com/doc-property_replacer.html You are correct, the documentation isn't perfect. That said, the application works pretty good and solves a lot of problems that aren't otherwise easily solved, so people like to use it. I would prefer the developers put their few precious resources into making the application solid vs. creating explicit documentation. You can always come here for help. But for that reason I recommend you read the above thoroughly and also the other sections of the configuration documentation. You'll spend some time putting together a configuration that works for you but once you understand how the configs work I think you'll find it relatively easy to figure out how to do specific things. If you think you understand how the selectors, actions and templates work together and still have problems or questions, let us know. Aaron From rgerhards at hq.adiscon.com Tue Feb 23 18:20:51 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:20:51 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103930@GRFEXC.intern.adiscon.com> Hi Aaron, thanks for all your help - MUCH appreciated. One thing I'd like to mention: > You are correct, the documentation isn't perfect. That said, the > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree to your analysis, but the doc problem has even more facets. I have blogged about it (once again), so that I have the information right at hand and available for easy sharing. I'd really appreciate if you could take out a few minutes and read this: http://blog.gerhards.net/2010/02/why-is-writing-good-user-doc-such.html In short: it would be very useful if "the average user" would contribute somewhat to the doc (or wiki!), because other "average users" understand that probably much better than what I write... Rainer PS: as a motivation: the blog post also contains links to all the good documentation sources ;) From rgerhards at hq.adiscon.com Tue Feb 23 18:49:57 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:49:57 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com><87wry5kzkx.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com><87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> > Maybe the sample configuration files are for an earlier release of > rsyslog. Harry, please do me a favor and complain with your distribution - they probably have simply overlooked this (can easily happen). I have just checked, the sample rsyslog.conf I provide includes commented-out remote reception commands (for a reason ;)): http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Also, I have added some text to the rsyslog cookbook for remote reception. I would appreciate if you could have a look and let me know if it is useful. If not, some text suggestions would be most welcome: http://cookbook.rsyslog.com/node16.html Rainer > > Cheers, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rory at ooma.com Wed Feb 24 00:37:07 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 15:37:07 -0800 Subject: [rsyslog] amazon s3 as logstore Message-ID: <4B8466A3.8000304@ooma.com> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? From rory at ooma.com Wed Feb 24 01:05:20 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:05:20 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8466A3.8000304@ooma.com> References: <4B8466A3.8000304@ooma.com> Message-ID: <4B846D40.3070505@ooma.com> On 2/23/10 3:37 PM, Rory Toma wrote: > Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Or, generically, how would I take a group of rsyslog servers accepting logs and have them log to central storage, without using a single central logger, and preferably, not using SQL injection. From david at lang.hm Wed Feb 24 01:10:48 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:10:48 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Or, generically, how would I take a group of rsyslog servers accepting > logs and have them log to central storage, without using a single > central logger, and preferably, not using SQL injection. so you want the logs send to a central place but not to a central logger. what's the difference? as a security person SQL injection is a exploit of badly written code, not something anyone would ever want to use, so I don't understand what you are asking here. David Lang From pgollucci at p6m7g8.com Wed Feb 24 01:11:16 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Wed, 24 Feb 2010 00:11:16 +0000 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B846EA4.1090800@p6m7g8.com> On 02/24/10 00:05, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? The easiest way AFAIK, is to have your own central rsyslog server with a db (mysql). Then use tarnsap (http://tarsnap.com) to copy either log files or db dumps to it. It uses AES256 bit encryption among other things so it can even story CC related data if needed. FWIW, tarnsap uses s3/ec2 for storage et al. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From rory at ooma.com Wed Feb 24 01:20:22 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:20:22 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B8470C6.507@ooma.com> I would like several front end servers writing to a single file-system, but I would expect that, out-of-the-box I would get out of order files and would have locking issues. On 2/23/10 4:10 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > > >> On 2/23/10 3:37 PM, Rory Toma wrote: >> >>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Or, generically, how would I take a group of rsyslog servers accepting >> logs and have them log to central storage, without using a single >> central logger, and preferably, not using SQL injection. >> > so you want the logs send to a central place but not to a central logger. > > what's the difference? > > as a security person SQL injection is a exploit of badly written code, not > something anyone would ever want to use, so I don't understand what you > are asking here. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Feb 24 01:34:05 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:34:05 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8470C6.507@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > I would like several front end servers writing to a single file-system, but I > would expect that, out-of-the-box I would get out of order files and would > have locking issues. Yes, if you use a standard filesystem you can only have one system writing to it at a time. If you want multiple systems to be able to write to a single filesystem you need to select a cluster filesystem or a network filesystem such a filesystem will allow multiple machines to write to different files on the shared filesystem at once, but they cannot write to the same file from different machines. all of the options have very different performance, reliability, and cost trade-offs. You will need to spend a lot of time thinking though your particular requirements and the capabilities of the different systems to pick one. I'm not sure exactly what you mean by 'out of order files', but if you are thinking that everything would end up in the same file and be in a different order than they were 'really' generated, then you are correct. The problem is that packets going over the network may not arrive in the same order that they were sent. Thus it is impossible for any system to guarantee the log entries reamin in the same order. the best that anything can do it to record very accurate timestamps and re-sort the logs after they have arrived. Very few people go down this road for logs. Almost everyone relays the logs to a single server and lets that server write to a filesystem (and if they need to eliminate this as a single point of failure, they make another system able to take over this filesystem if the first one goes down) David Lang > On 2/23/10 4:10 PM, david at lang.hm wrote: >> On Tue, 23 Feb 2010, Rory Toma wrote: >> >> >>> On 2/23/10 3:37 PM, Rory Toma wrote: >>> >>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> Or, generically, how would I take a group of rsyslog servers accepting >>> logs and have them log to central storage, without using a single >>> central logger, and preferably, not using SQL injection. >>> >> so you want the logs send to a central place but not to a central logger. >> >> what's the difference? >> >> as a security person SQL injection is a exploit of badly written code, not >> something anyone would ever want to use, so I don't understand what you >> are asking here. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From rory at ooma.com Wed Feb 24 01:38:49 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:38:49 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: <4B847519.8060702@ooma.com> On 2/23/10 4:34 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > >> I would like several front end servers writing to a single >> file-system, but I would expect that, out-of-the-box I would get out >> of order files and would have locking issues. > > Yes, if you use a standard filesystem you can only have one system > writing to it at a time. > > If you want multiple systems to be able to write to a single > filesystem you need to select a cluster filesystem or a network > filesystem > > such a filesystem will allow multiple machines to write to different > files on the shared filesystem at once, but they cannot write to the > same file from different machines. > > all of the options have very different performance, reliability, and > cost trade-offs. You will need to spend a lot of time thinking though > your particular requirements and the capabilities of the different > systems to pick one. > > I'm not sure exactly what you mean by 'out of order files', but if you > are thinking that everything would end up in the same file and be in a > different order than they were 'really' generated, then you are > correct. The problem is that packets going over the network may not > arrive in the same order that they were sent. Thus it is impossible > for any system to guarantee the log entries reamin in the same order. > the best that anything can do it to record very accurate timestamps > and re-sort the logs after they have arrived. > > Very few people go down this road for logs. Almost everyone relays the > logs to a single server and lets that server write to a filesystem > (and if they need to eliminate this as a single point of failure, they > make another system able to take over this filesystem if the first one > goes down) > Yeah, I didn't think it was practical, but if someone had done something unorthodox, and it worked, I thought it might be nice to hear about it. From david at lang.hm Wed Feb 24 01:57:28 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:57:28 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, david at lang.hm wrote: > I'm not sure exactly what you mean by 'out of order files', but if you are > thinking that everything would end up in the same file and be in a different > order than they were 'really' generated, then you are correct. The problem is > that packets going over the network may not arrive in the same order that > they were sent. Thus it is impossible for any system to guarantee the log > entries reamin in the same order. the best that anything can do it to record > very accurate timestamps and re-sort the logs after they have arrived. By the way, if keeping the logs in order is critical to you, you need to be aware that rsyslog may not process the logs in order under all conditions. For example: if you have multiple worker threads, they each grab some logs off the queue and attempt to deliver/write them. This can cause the logs to be delivered/written in a different order than rsyslog received them. Since there are so many other things that can re-order logs slightly, this is not viewed as a significant problem. David Lang > >> On 2/23/10 4:10 PM, david at lang.hm wrote: >>> On Tue, 23 Feb 2010, Rory Toma wrote: >>> >>> >>>> On 2/23/10 3:37 PM, Rory Toma wrote: >>>> >>>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> Or, generically, how would I take a group of rsyslog servers accepting >>>> logs and have them log to central storage, without using a single >>>> central logger, and preferably, not using SQL injection. >>>> >>> so you want the logs send to a central place but not to a central logger. >>> >>> what's the difference? >>> >>> as a security person SQL injection is a exploit of badly written code, not >>> something anyone would ever want to use, so I don't understand what you >>> are asking here. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> > From rgerhards at hq.adiscon.com Wed Feb 24 07:28:15 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 07:28:15 +0100 Subject: [rsyslog] amazon s3 as logstore References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com><4B8470C6.507@ooma.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103933@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, February 24, 2010 1:57 AM > To: Rory Toma > Cc: rsyslog-users > Subject: Re: [rsyslog] amazon s3 as logstore > > On Tue, 23 Feb 2010, david at lang.hm wrote: > > By the way, if keeping the logs in order is critical to you, you need > to > be aware that rsyslog may not process the logs in order under all > conditions. > > For example: if you have multiple worker threads, they each grab some > logs > off the queue and attempt to deliver/write them. This can cause the > logs > to be delivered/written in a different order than rsyslog received > them. > > Since there are so many other things that can re-order logs slightly, > this > is not viewed as a significant problem. > David is very right here. If you rely on the sequence as recorded in the log file, you have a big problem. That problem is not caused by rsyslog's threading, but rather by the fact that syslog is usually used inside a remote system. In such a system, there is no guaranteed sequence: UDP get's reordered, TCP messages sit for different timespans in interim routers, syslog relay chains may delay messages even for hours (if a relay is down). I thought I had blogged about this, but can't find the post myself right now ;) But in short: you need to be very careful about your sequence expectation. Timestamps are much better, but of course not perfect. RFC5424 provides the plumbing for a better solution, but none is standardized yet and I know of no such implementation. I have Lamport clocks on my mind as the "ultimate" solution (and maybe I get time to implement that in a distant future...). Rainer From tbergfeld at hq.adiscon.com Wed Feb 24 14:57:16 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 24 Feb 2010 14:57:16 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From reader at newsguy.com Wed Feb 24 21:03:57 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:03:57 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> Message-ID: <871vgaieoi.fsf@newsguy.com> "Rainer Gerhards" writes: >> Maybe the sample configuration files are for an earlier release of >> rsyslog. > > Harry, > > please do me a favor and complain with your distribution - they probably have > simply overlooked this (can easily happen). I have just checked, the sample > rsyslog.conf I provide includes commented-out remote reception commands (for > a reason ;)): > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 > e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Thanks for showing that... and for a moment there I was afraid I was the dope that had somehow deleted the important missing lines. So just to make sure I checked my distros' (gentoo) rsyslog.conf shipped for installation by the distro maintainer and sure enough they have: # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 When the original shipped with rsyslog pkg has the lines below as you've shown. # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module #$InputTCPServerRun 514 # start up TCP listener at port 514 # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 Apparently my distro has removed that one all important line. `#$InputTCPServerRun 514 # start up TCP listener at port 514' And yes, I will make sure to contact the package maintainer about it. Thanks for the heads up and I'm sure the maintainer will want to get that back in there too. > Also, I have added some text to the rsyslog cookbook for remote reception. I > would appreciate if you could have a look and let me know if it is useful. If > not, some text suggestions would be most welcome: > > http://cookbook.rsyslog.com/node16.html > Thanks... that couldn't be more clear... brief and directly on point. Thanks. From reader at newsguy.com Wed Feb 24 21:11:14 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:11:14 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <87wry2gzrx.fsf@newsguy.com> Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. From rgerhards at hq.adiscon.com Wed Feb 24 22:09:01 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 22:09:01 +0100 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> At least for me, it would be usefuly if you could describe your needs in relatively precise words. It may be obvious for you, but that is what is hard to grasp for me... ----- Urspr?ngliche Nachricht ----- Von: "Harry Putnam" An: "rsyslog at lists.adiscon.com" Gesendet: 24.02.10 21:12 Betreff: Re: [rsyslog] How to make rsyslog create dir per host Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 01:45:35 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:45:35 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/24 Tom Bergfeld : > Hi all, > We have just released rsyslog 4.6.0. > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new > features of 4.5.x to the stable releases. This includes bug fixes as well as > many new features. Out of the many, I would like to highlight the ability to > have output files automatically zipped, multiple TCP listeners and overall > performance improvement. For all details, please check the 4.5.x change log > entries. > > This release has undergone rigorous testing in some very demanding > environments. We held release until all tests went out well. We are happy to > have reached this stage now and sincerely think that the version is ready for > prime time. Thanks to everyone who contributed test time and bug reports! > > We recommend that all v4-stable users have a look at 4.6.0 and consider > updating to it. Please note that 4.6.0 also does have some patches which > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available > via support contracts). Seems, the xconsole/pipe related fixes did not make it into this release. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:48:53 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:48:53 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Michael Biebl : > > Seems, the xconsole/pipe related fixes did not make it into this release. Running rsyslogd in debug mode and opening xconsole (as root) I get 8841.463601465:b6e86b70: Message from UNIX socket: #3 8841.463676615:b6e86b70: logmsg: flags 4, from 'pluto', msg Feb 25 01:47:21 sudo: michael : TTY=pts/4 ; PWD=/home/michael/debian/rsyslog ; USER=root ; COMMAND=/usr/bin/xconsole 8841.463705390:b6e86b70: Message has legacy syslog format. 8841.463735003:b6e86b70: main Q: entry added, size now 1 entries 8841.463762660:b6e86b70: wtpAdviseMaxWorkers signals busy 8841.463795905:b6e86b70: main Q: EnqueueMsg advised worker start 8841.463825239:b6e86b70: --------imuxsock calling select, active file descriptors (max 3): 3 8841.463884465:b7687b70: main Q: entry deleted, state 0, size now 0 entries 8841.463925812:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'NetworkManager': FALSE 8841.463959895:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'wpa_supplicant': FALSE 8841.463997330:b7687b70: Called action, logging to builtin-file 8841.464033369:b7687b70: file to log to: /var/log/auth.log 8841.464057674:b7687b70: doWrite, pData->pStrm 0x8a32360, lenBuf 124 8841.464081699:b7687b70: strm 0x8a32360: file 9 flush, buflen 124 8841.464150983:b7687b70: strm 0x8a32360: file 9 write wrote 124 bytes 8841.464196520:b7687b70: Called action, logging to builtin-file 8841.464224177:b7687b70: file to log to: /dev/xconsole 8841.464247086:b7687b70: doWrite, pData->pStrm 0x8a3c248, lenBuf 124 8841.464269714:b7687b70: strm 0x8a3c248: file -1 flush, buflen 494 8841.464298210:b7687b70: Note: stream '/dev/xconsole' is a named pipe, open with O_NONBLOCK 8841.464340674:b7687b70: file '/dev/xconsole' opened as #-1 with mode 416 8841.464380065:b7687b70: strm 0x8a3c248: open error 6, file '/dev/xconsole': No such device or address 8841.464413869:b7687b70: main Q:Reg/w0: worker IDLE, waiting for work. The relevant rsyslog.conf configuration is daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:50:45 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:50:45 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: As additional information: If i run rsyslogd first, then xconsole, I get the failure Running xconsole first, then rsyslogd, it works fine. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Thu Feb 25 01:58:52 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 18:58:52 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> Message-ID: <87fx4qgmgj.fsf@newsguy.com> "Rainer Gerhards" writes: > At least for me, it would be usefuly if you could describe your > needs in relatively precise words. It may be obvious for you, but > that is what is hard to grasp for me... Not sure what you are talking about here Rainer. I was just replying to someone who appeared to think I had made some disparaging remark about the documentation. I simply wanted to clear that up... Now if you mean for me to explain more precisely what I wanted from rsyslog, I'll do that. However, let me say first that with Aarons' pointers to URLS that I am now working my way thru and also having just watched one of your videos and the basics... I have to tell you that that caliber of the documentation is really quite high. There is in depth coverage and then very helpful outside resources such as the video and other postings of yours. I wish I had the command of a foreign language as well as you have of english. I'm a native english speaker... but am sadly nearly illiterate in my own language. The documentation really does out class the usual documentation one encounters in unix/linux tools. ------- --------- ---=--- --------- -------- What I sought to convey in my question about creating directories: Its really low level so sorry I didn't get it across very well. Once I got the clients writing across the network to the rsyslog server, and the server listening and logging the data, I thought for now it would be good to have rsyslog on the server create directories for each remote client, and write everything from each client to /var/log/%hostname%/everthing.log (as a beginning point. While I figure out how to set things up so that each incoming log from remote would be split up further under the top level directories. Kind of like the example shown at (wrapped for mail): http://wiki.rsyslog.com/index.php/ \ Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory Just one level deeper. So one level of dynamic directory is created for each client host. Later I'll probably want to spit that a bit more Once I have a more permanent plan figured out I would at least have the full output from each client host on hand. I needed to get the basics up some whet urgently in that I had one host failing silently or at least not leaving a trace I could find in its logs. It would just freeze up, where the mouse and keyboard became unresponsive and could not connect to the problem host remotely either. On mechanical reboot, it would look like the system logger froze up too but someone told me I might catch some log data that would be lost on a hard reboot if I had that host logging remotely...thru rsyslog. I doubted that would be true since it appears as though networking is froze. But I have gotten rsyslog logging remotely now and waiting for a freezeup to see if I can get any more clues as to what is causing the problem. From buraphalinuxserver at gmail.com Thu Feb 25 08:34:29 2010 From: buraphalinuxserver at gmail.com (BuraphaLinux Server) Date: Thu, 25 Feb 2010 14:34:29 +0700 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x Message-ID: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Will there be an rsyslog 4.4.3 release? I see references to changes from it in the website, but cannot find the 4.4.3 release itself, and the 4.6.0 release seems to say that there will not be any 4.x releases any more. The web page says 4.6.0 is a stable release and a beta at the same time, which is very confusing for me. http://www.rsyslog.com/Article443.phtml The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which means it is a beta, but then the text on the page also says "This is a new stable v4 version." and "Do not expect any more new developments in v4.". I'm a bit confused. Does this mean if we want bugfixes we have to run a 5.x version? From rgerhards at hq.adiscon.com Thu Feb 25 09:09:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:09:24 +0100 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x References: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of BuraphaLinux Server > Sent: Thursday, February 25, 2010 8:34 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x > > Will there be an rsyslog 4.4.3 release? I see references to changes > from it in the website, but cannot find the 4.4.3 release itself, No. I initially planned to have a 4.4.3, but Monday, when I tried to prepare it, I found some issues in the automated test suite. Whatever they were, they were fixed in 4.6.0. So I skipped that release, saving me the time to find what already is fixed (the original plan was to do a last 4.4.3 release followed by 4.6.0 two days later, so I only skipped the 4.4.3 step). > and > the 4.6.0 release seems to say that there will not be any 4.x releases > any more. The web page says 4.6.0 is a stable release and a beta at > the same time, which is very confusing for me. > > http://www.rsyslog.com/Article443.phtml Thanks for pointing that out. That's a simple typo, now corrected. > > The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which > means it is a beta, but then the text on the page also says "This is a > new stable v4 version." and "Do not expect any more new developments > in v4.". I'm a bit confused. Does this mean if we want bugfixes we > have to run a 5.x version? Bugfixes are not "new developments", at least in my point of view. But if you need some new feature, you need to run v5 if we implement it. That's the way it was always handled in rsyslog, with the exception of v4 and v5, which had two different development branches at the same time. There were good reasons for that, but it also caused a lot of work and some issues. I am very happy to be back to a single development version again. I won't split off a new second development branch again until there is *very* good reason. I hope this clarifies. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:25:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:25:18 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Hi all, on Debian, a discussion came up on the inconsistency of some config statements. The right cure would be to correct this, but that requires some time. I would appreciate your feedback on the issue. The details can be found here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 Thanks, Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:26:47 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:26:47 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:46 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/24 Tom Bergfeld : > > Hi all, > > We have just released rsyslog 4.6.0. > > > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings > all new > > features of 4.5.x to the stable releases. This includes bug fixes as > well as > > many new features. Out of the many, I would like to highlight the > ability to > > have output files automatically zipped, multiple TCP listeners and > overall > > performance improvement. For all details, please check the 4.5.x > change log > > entries. > > > > This release has undergone rigorous testing in some very demanding > > environments. We held release until all tests went out well. We are > happy to > > have reached this stage now and sincerely think that the version is > ready for > > prime time. Thanks to everyone who contributed test time and bug > reports! > > > > We recommend that all v4-stable users have a look at 4.6.0 and > consider > > updating to it. Please note that 4.6.0 also does have some patches > which > > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still > available > > via support contracts). > > Seems, the xconsole/pipe related fixes did not make it into this > release. mhhh... The delta between 4.5.8 and 4.6.0 is very slim. So I wonder what might have broken that fix... Will try to check. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:44:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:44:10 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Thursday, February 25, 2010 1:59 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] How to make rsyslog create dir per host > > "Rainer Gerhards" writes: > > > At least for me, it would be usefuly if you could describe your > > needs in relatively precise words. It may be obvious for you, but > > that is what is hard to grasp for me... > sorry, that was too brief. I was on a PDA and it was late ;) My intent is to see if I can write up a cookbook entry for what you need, but I have no clear understanding what's your intent. Now let me read through the rest of your mail and see where I come to... ;) > Not sure what you are talking about here Rainer. I was just replying > to someone who appeared to think I had made some disparaging remark > about the documentation. > > I simply wanted to clear that up... > > > Now if you mean for me to explain more precisely what I wanted from > rsyslog, I'll do that. > > However, let me say first that with Aarons' pointers to URLS that I am > now working my way thru and also having just watched one of your > videos and the basics... I have to tell you that that caliber of the > documentation is really quite high. > > There is in depth coverage and then very helpful outside resources such > as the video and other postings of yours. > > I wish I had the command of a foreign language as well as you have of > english. I'm a native english speaker... but am sadly nearly > illiterate in my own language. > > The documentation really does out class the usual documentation one > encounters in unix/linux tools. Thanks for the kind words, but you had a valid point in your initial remarks: the rsyslog doc may be good for someone who intends to understand the whole system and be an expert at that. However, most often people look for a solution to a specific problem. They don't want to understand how everything works together, just solve that immediate need quickly. I think this is a valid need, I have it myself with other softwares often enough ;) The rsyslog doc completely fails to address this use case. I hope the cookbook will somewhat improve that situation. I am not sure if you saw my longer blogpost, it has the details ;) > > ------- --------- ---=--- --------- -------- > What I sought to convey in my question about creating directories: > > Its really low level so sorry I didn't get it across very well. > > Once I got the clients writing across the network to the rsyslog > server, and the server listening and logging the data, I thought for > now it would be good to have rsyslog on the server create directories > for each remote client, and write everything from each client to > /var/log/%hostname%/everthing.log (as a beginning point. > > While I figure out how to set things up so that each incoming log > from remote would be split up further under the top level directories. > > Kind of like the example shown at (wrapped for mail): > > http://wiki.rsyslog.com/index.php/ \ > Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > Just one level deeper. So one level of dynamic directory is created > for each client host. Later I'll probably want to spit that a bit more ... and here is where some information is missing ;) *How* do you want to split these logs? I am not sure about this, and so I have no clear idea of how a configuration could look like. As there are some limits on the way filters can be combined, this is a very important point. As a side-note: I do not know your log volume, but if it is not overwhelmingly large to prevent storing logs to a database, I'd probably do that and analyze and search it with free GPLed Adiscon LogAnalyzer: http://loganalyzer.adiscon.com These days, we are preparing a new v3 release which also supports reporting (and you can schedule that via cron, too). (Adiscon LogAnalyzer also works with plain text files, but this is only meant for low-volume, few-system consolidated logs - otherwise the performance will be very bad). HTH Rainer > > Once I have a more permanent plan figured out I would at least have > the full output from each client host on hand. > > I needed to get the basics up some whet urgently in that I had one host > failing silently or at least not leaving a trace I could find in its > logs. It would just freeze up, where the mouse and keyboard became > unresponsive and could not connect to the problem host remotely > either. > > On mechanical reboot, it would look like the system logger froze up > too but someone told me I might catch some log data that would be lost > on a hard reboot if I had that host logging remotely...thru rsyslog. > > I doubted that would be true since it appears as though networking is > froze. But I have gotten rsyslog logging remotely now and waiting for > a freezeup to see if I can get any more clues as to what is causing > the problem. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 25 09:51:31 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:51:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. ah! I think I now understand the issue. This smells like a new one. Did this really work with 4.5.8? Rainer > > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 13:22:02 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 13:22:02 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, February 25, 2010 1:51 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. > > ah! I think I now understand the issue. This smells like a new one. Did this > really work with 4.5.8? I haven't tested 4.5.8. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 13:56:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 13:56:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> > -----Original Message----- > 2010/2/25 Rainer Gerhards : > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl > >> Sent: Thursday, February 25, 2010 1:51 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > >> > >> As additional information: > >> > >> If i run > >> rsyslogd first, then xconsole, I get the failure > >> Running xconsole first, then rsyslogd, it works fine. > > > > ah! I think I now understand the issue. This smells like a new one. > Did this > > really work with 4.5.8? > > I haven't tested 4.5.8. ah, ok, so which was the latest version you tested against? Rainer From mbiebl at gmail.com Thu Feb 25 14:01:19 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 14:01:19 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> 2010/2/25 Rainer Gerhards : >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> >> Sent: Thursday, February 25, 2010 1:51 AM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> >> >> As additional information: >> >> >> >> If i run >> >> rsyslogd first, then xconsole, I get the failure >> >> Running xconsole first, then rsyslogd, it works fine. >> > >> > ah! I think I now understand the issue. This smells like a new one. >> Did this >> > really work with 4.5.8? >> >> I haven't tested 4.5.8. > > ah, ok, so which was the latest version you tested against? > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 14:06:11 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 14:06:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103966@GRFEXC.intern.adiscon.com> > >> >> If i run > >> >> rsyslogd first, then xconsole, I get the failure > >> >> Running xconsole first, then rsyslogd, it works fine. > >> > > >> > ah! I think I now understand the issue. This smells like a new > one. > >> Did this > >> > really work with 4.5.8? > >> > >> I haven't tested 4.5.8. > > > > ah, ok, so which was the latest version you tested against? > > > > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. OK, misunderstanding on my side. I thought you had verified that the "xconsole" patch worked. I will try to dig again into the code, but it's some while since I crafted the patch, so it may take a while. Maybe I manage to do that today, if not it'll be monday - I have meetings tomorrow :( Rainer From rgerhards at hq.adiscon.com Thu Feb 25 15:27:44 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 15:27:44 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. > I just verified the case. I get the failure if xconsole is not active, what is perfectly OK IMHO (because otherwise it would block on the pipe). As soon as I start xconsole, everything works fine and messages are displayed in xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it everything works again. So I do not see an issue. I tested on Debian lenny with all current updates applied. Rainer > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Thu Feb 25 20:55:19 2010 From: reader at newsguy.com (Harry Putnam) Date: Thu, 25 Feb 2010 13:55:19 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> Message-ID: <87y6ihf5ug.fsf@newsguy.com> "Rainer Gerhards" writes: >> Just one level deeper. So one level of dynamic directory is created >> for each client host. Later I'll probably want to spit that a bit more > > ... and here is where some information is missing ;) *How* do you want to > split these logs? I am not sure about this, and so I have no clear idea of > how a configuration could look like. As there are some limits on the way > filters can be combined, this is a very important point. He he... yes it is.. but its missing because I don't have a solid plan for what I want yet. I really was only asking about the simple part. I now know how to do that, and when you see what I was after (for now) you will be disappointed because its probably to obvious and simple to need to be in your cook book. It just takes me a while to see how to get started. On the client machines: *.info;mail.none @@192.168.0.26:514 (192.168.0.26 is `logsrv' mentioned below) On the server: This template and action is all I was after (for now). (aside: DDF = Dynamic directory and filename) $template DDF,"/var/log/%hostname%/%hostname%.log" if \ $source != 'logsrv' \ then -?DDF I hope that is at least close to a correct formulation. I just used the actual host names because I wan't sure which property provided it .. just the simple alphabetic hostname with no domain. On thing I didn't see explained in what I've read so far is what role the question mark in -?DDF plays. I used it because I saw it used that way in the examples and so far the formulation seems to be working...(I actually haven't inspected the logs closely yet.... but a tail -f of the resulting log files would seem to indicate its working. At least each log appears to be only from a specific client, and to include everything incoming from that client.) From mbiebl at gmail.com Thu Feb 25 21:44:31 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 21:44:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. >> > > I just verified the case. I get the failure if xconsole is not active, what > is perfectly OK IMHO (because otherwise it would block on the pipe). As soon > as I start xconsole, everything works fine and messages are displayed in > xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it > everything works again. > > So I do not see an issue. I tested on Debian lenny with all current updates > applied. hm, it's perfectly reproducible here. starting rsyslogd, then xconsole, will cause xconsole to hang and no message is written to the xconsole window. 4.4.2 did not behave that way. See also the debug log I sent previously Is there anything else I can do to help you debug this? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Fri Feb 26 01:09:16 2010 From: lanas at securenet.net (lanas) Date: Thu, 25 Feb 2010 19:09:16 -0500 Subject: [rsyslog] Sending time in remote syslog Message-ID: <20100225190916.0a83b31d@mistral.stie> Hello ! Is it possible with rsyslog to somehow send the time of the local event as part of the log message when sending to a remote server ? Here's a use case. I have tried with syslogd. I enable syslogd on a unit to send log messages to a remote host running rsyslog (it so happens that it runs rsyslog because it's a Fedora host). Then I change the time on the unit by an offset of two hours. When the unit (syslogd) sends the log message, it is received by rsyslogd although the time stamp in the log msg shows the time on the host, not the time when the event happened on the unit itself. If I replace syslogd by rsyslog on the unit, will it be able to somehow send the local time of the unit as part of the log message ? Thanks, Al From david at lang.hm Fri Feb 26 08:46:21 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 25 Feb 2010 23:46:21 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100225190916.0a83b31d@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> Message-ID: On Thu, 25 Feb 2010, lanas wrote: > Hello ! > > Is it possible with rsyslog to somehow send the time of the local > event as part of the log message when sending to a remote server ? > > Here's a use case. I have tried with syslogd. I enable syslogd on a > unit to send log messages to a remote host running rsyslog (it so > happens that it runs rsyslog because it's a Fedora host). Then I > change the time on the unit by an offset of two hours. When the unit > (syslogd) sends the log message, it is received by rsyslogd although > the time stamp in the log msg shows the time on the host, not the time > when the event happened on the unit itself. > > If I replace syslogd by rsyslog on the unit, will it be able to > somehow send the local time of the unit as part of the log message ? each event has two timestamps, one is the timestamp in the message that was sent, the other is the time the message was received. I believe that the default template uses the recieved timestamp, but you can change it to timereportedto get the timestamp put on the message by the sender. David Lang From lanas at securenet.net Fri Feb 26 11:46:05 2010 From: lanas at securenet.net (lanas) Date: Fri, 26 Feb 2010 05:46:05 -0500 Subject: [rsyslog] Sending time in remote syslog In-Reply-To: References: <20100225190916.0a83b31d@mistral.stie> Message-ID: <20100226054605.3469f03a@mistral.stie> On Thu, 25 Feb 2010 23:46:21 -0800 (PST), david at lang.hm wrote : > On Thu, 25 Feb 2010, lanas wrote: > > Is it possible with rsyslog to somehow send the time of the local > > event as part of the log message when sending to a remote server ? > > each event has two timestamps, one is the timestamp in the message > that was sent, the other is the time the message was received. I > believe that the default template uses the recieved timestamp, but > you can change it to timereportedto get the timestamp put on the > message by the sender. Thanks, I will try it. Basically, this means that (if there's a syslog message standard ...) that syslog messages contains both timestamps, be them generated by syslogd or rsyslog, isn't it ? And it's at the receiving side that the proper display option should be chosen (in this case, does rsyslog have an advantage over others by providing more display options ?). Regards, Al From rgerhards at hq.adiscon.com Fri Feb 26 12:32:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:32:10 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> > > I just verified the case. I get the failure if xconsole is not > active, what > > is perfectly OK IMHO (because otherwise it would block on the pipe). > As soon > > as I start xconsole, everything works fine and messages are displayed > in > > xconsole. When I stop xconsole I get "broken pipe" errors and if I > restart it > > everything works again. > > > > So I do not see an issue. I tested on Debian lenny with all current > updates > > applied. > > hm, it's perfectly reproducible here. also on Debian lenny? Anything else I could do to set up my environment so that the problem occurs? > starting rsyslogd, then xconsole, will cause xconsole to hang and no > message is written to the xconsole window. > 4.4.2 did not behave that way. > > See also the debug log I sent previously The log tells me that it has a problem to write to the pipe, and thus ignores it. This also happens at my system, but only if xconsole is not running. So the debug log does no help very much this time. > Is there anything else I can do to help you debug this? When I get the platform information from you, I will see if I can record a session in my lab. Maybe you see that I do things differently from you. One thing I notice is that xconsole seems to be unable to access /dev/xconsole if I start it as non-root, thus resulting in no listener on the pipe and thus the error appears in rsyslog. However, this is not rsyslog-related, but I guess these permissions are set by xconsole. After all, there should be no change to 4.4.2 (and I don't see any difference in that regard to 4.4.2 on my system). Rainer > > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Fri Feb 26 12:55:11 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 12:55:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> > I just verified the case. I get the failure if xconsole is not >> active, what >> > is perfectly OK IMHO (because otherwise it would block on the pipe). >> As soon >> > as I start xconsole, everything works fine and messages are displayed >> in >> > xconsole. When I stop xconsole I get "broken pipe" errors and if I >> restart it >> > everything works again. >> > >> > So I do not see an issue. I tested on Debian lenny with all current >> updates >> > applied. >> >> hm, it's perfectly reproducible here. > > also on Debian lenny? Anything else I could do to set up my environment so > that the problem occurs? I'm can reproduce the problem, both on Debian lenny and Debian unstable (i386) >> starting rsyslogd, then xconsole, will cause xconsole to hang and no >> message is written to the xconsole window. >> 4.4.2 did not behave that way. >> >> See also the debug log I sent previously > > The log tells me that it has a problem to write to the pipe, and thus ignores > it. This also happens at my system, but only if xconsole is not running. So > the debug log does no help very much this time. The problem is, that xconsole is running at this point! To reiterate: I start rsyslogd then xconsole then try to log a message The result is the debug log and no message showing up in xconsole. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 12:56:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:56:49 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> > The problem is, that xconsole is running at this point! > To reiterate: > I start rsyslogd > then xconsole > then try to log a message > The result is the debug log and no message showing up in xconsole. that's what I am doing as well. I will try to record the session. Rainer From rgerhards at hq.adiscon.com Fri Feb 26 13:20:17 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 13:20:17 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103973@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, February 26, 2010 12:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > > The problem is, that xconsole is running at this point! > > To reiterate: > > I start rsyslogd > > then xconsole > > then try to log a message > > The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Please find a video of my session at http://www.rsyslog.com/download/rsyslog-xconsole.avi So I don't see the problem. Or did you mean something different? Rainer From mbiebl at gmail.com Fri Feb 26 13:43:23 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 13:43:23 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> The problem is, that xconsole is running at this point! >> To reiterate: >> I start rsyslogd >> then xconsole >> then try to log a message >> The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Wait a sec. With 4.4.2 (or any earlier version) I tried this: start rsyslog start xconsole xconsole shows the last two log messages immediately With 4.6.0 the xconsole window is blank (which I guess made me believe it blocked) With 4.6.0, I need to trigger a new log message (e.g. using logger foo), and then xconsole will show a new message. So, xconsole does indeed show messages, but the behaviour is different to 4.4 and earlier Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 15:19:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 15:19:55 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103975@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, February 26, 2010 1:43 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Rainer Gerhards : > >> The problem is, that xconsole is running at this point! > >> To reiterate: > >> I start rsyslogd > >> then xconsole > >> then try to log a message > >> The result is the debug log and no message showing up in xconsole. > > > > that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Mhhh... I don't think it is worth preserving "backward compatibility" here. I think these two (just two!) messages were an artifact of the way files were written to pipes that nobody cares about. The old file writer was *very* suboptimal. If I would need to retain that old "functionality", I would need to remove almost all optimizations. So it would make more sense than to craft a new output module specifically for pipes. Not sure if this is granted by that "use case". I'd appreciate comments from the community at large. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Feb 26 17:11:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 17:11:29 +0100 Subject: [rsyslog] Log Normalization effort Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Hi all, I have blogged about my quest for log normalization. I think there is some good information on the upcoming GPLed Adiscon LogAnalyzer and future directions for rsyslog in the blog post. So I thought I share the link: http://blog.gerhards.net/2010/02/syslog-normalization.html Please note that part of the effort requires community involvement. I would be very interested to learn if you think we could win enough support to make this a useful effort. I am asking for your feedback, because it will help me streamline my priorities for future rsyslog work. Thanks, Rainer From david at lang.hm Fri Feb 26 19:34:04 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 10:34:04 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100226054605.3469f03a@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> <20100226054605.3469f03a@mistral.stie> Message-ID: On Fri, 26 Feb 2010, lanas wrote: > On Thu, 25 Feb 2010 23:46:21 -0800 (PST), > david at lang.hm wrote : > >> On Thu, 25 Feb 2010, lanas wrote: >>> Is it possible with rsyslog to somehow send the time of the local >>> event as part of the log message when sending to a remote server ? >> >> each event has two timestamps, one is the timestamp in the message >> that was sent, the other is the time the message was received. I >> believe that the default template uses the recieved timestamp, but >> you can change it to timereportedto get the timestamp put on the >> message by the sender. > > Thanks, I will try it. Basically, this means that (if there's a syslog > message standard ...) that syslog messages contains both timestamps, be > them generated by syslogd or rsyslog, isn't it ? And it's at the > receiving side that the proper display option should be chosen (in this > case, does rsyslog have an advantage over others by providing more > display options ?). the messages sent over the wire only have one timestamp in them, but when rsyslog receives a message it records that time as well. This gives rsyslog the option of using either timestamp when it then writes (or sends) the log. David Lang From david at lang.hm Fri Feb 26 22:32:56 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 13:32:56 -0800 (PST) Subject: [rsyslog] Log Normalization effort In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 26 Feb 2010, Rainer Gerhards wrote: > Hi all, > > I have blogged about my quest for log normalization. I think there is some > good information on the upcoming GPLed Adiscon LogAnalyzer and future > directions for rsyslog in the blog post. So I thought I share the link: > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > Please note that part of the effort requires community involvement. I would > be very interested to learn if you think we could win enough support to make > this a useful effort. I am asking for your feedback, because it will help me > streamline my priorities for future rsyslog work. a few comments (but remember that I am usually dealing with high data rates, so my concerns are biased in that direction) log analysis is usually done in batches as opposed to in real-time. some of this is due to the difficulty in doing it in real time, but a lot of it is the processing overhead (you don't want to take so long to process an individual request that you miss the next one to arrive) at low volumes the idea of name-value pairs in the logs makes a lot of sense, but there is significantly more overhead in parsing a log with name-value pairs in arbitrary orders than there is in using a tree parsing approach to analyze known log formats in a fixed order. The message size can also increase significantly. As a result, at high traffic volumes this starts to be a bad (or at least questionable) idea. I would love to see rsyslog gain the ability to efficiently do tree-based parsing instead of regex parsing. regex parsing is easy to understand and tinker with, but very expensive to implement. it may be that having something that 'compiles' a list of regex parsers into a tree parser is the right answer for usability. I would save several hours of processing a day if I could easily (and efficiently) make rsyslog write different logs to different files (at high data rates and with a few hundred conditions based variations in the syslog tag) While there are some common events across different types of logs (logins for example) they almost always contain slightly different data in them. I also have no faith at all that anyone is going to make much effort to clean up their logs to make them nicely parseable, and if they do I see even less chance that they will end up using the same terms for the same thing. As such I see more value in trying to get samples of logs and what they mean than in trying to define a normalized version to shoehorn the logs into. It is worth doing this for some events (logins, failed logins for example), but I think it's a mistake to think that this will end up covering all, or even the majority of log messages. There's also a problem in that the ideal format for the output depends on what you are doing with the output. If I could wave a magic wand and get the result I would look for something like this the parser starts at the beginning of the message (at the priority) and can branch on priority/faclilty, timestamp, host, syslogtag, message and indicate if the message should be parsed into name-value pairs, or split based on a character (or character sequence like the perl split command allows) into individually addressable elements (defaulting to whitespace separated elements), then the format (and if needed dynafile path/file components) could be constructed from these variables. At any point in the parsing it should be possible to jump to another parser tree (so that you could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end up using the same parser for the message without having to redefine the rules for each one) With this capability, people could start writing parser 'branches' to understand a specific log type and output a 'standard' format (as such a format can start to be defined). This can be done in rsyslog today, but it is fairly difficult to define, and as I understand it, inefficient enough that it's not practical to do in real-time under heavy load. If this is fast enough, then the next step would be to add the ability to have the format/action be 'increment a counter for log type X' and a signal to rsyslog could generate a report on these counters. Although at some point it becomes better to feed the message into another opensource tool (SEC, Simple Event Correlator for example) instead of trying to do everything in rsyslog. parsing the file to know what to do with it, and be able to re-format log messages is very defiantly something that can fit into the rsyslog model of receiving, formatting, and delivering logs. Alerting on specific log entries, counting the number of times one thing shows up in logs, and this sort of thing start pushing beyond the core of rsyslog, and it may be better to feed other tools instead. David Lang From lanas at securenet.net Sat Feb 27 12:38:07 2010 From: lanas at securenet.net (lanas) Date: Sat, 27 Feb 2010 06:38:07 -0500 Subject: [rsyslog] rsyslog migration and syslogd.conf Message-ID: <20100227063807.5918fafc@mistral.stie> Hello, First, thanks to David for the reply on time stamps. I have a question about perhaps being able to replace syslogd by rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I reckon with limited possibilities) with a standard syslog config file ? That would enable a basic set of functionality as a first step toward using rsyslog's features. Al From david at lang.hm Sun Feb 28 01:02:16 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:02:16 -0800 (PST) Subject: [rsyslog] rsyslog migration and syslogd.conf In-Reply-To: <20100227063807.5918fafc@mistral.stie> References: <20100227063807.5918fafc@mistral.stie> Message-ID: On Sat, 27 Feb 2010, lanas wrote: > Hello, > > First, thanks to David for the reply on time stamps. > > I have a question about perhaps being able to replace syslogd by > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > reckon with limited possibilities) with a standard syslog config > file ? That would enable a basic set of functionality as a first step > toward using rsyslog's features. no, rsyslog will do nothing if you just feed it a syslog.conf file. rsyslog needs a few additional instructions at the top of the file to load the modules, but below that you can paste in the syslog.conf file. another thing that you can do is to make a generic rsyslog.conf file that includes the syslog.conf after the rsyslog specific headers. My rsyslog.conf (on my ubuntu 9.10 laptop) has $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog $RepeatedMsgReduction on $RepeatedMsgContainsOriginalMsg on David Lang From david at lang.hm Sun Feb 28 01:45:14 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:45:14 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 25 Feb 2010, Rainer Gerhards wrote: > Hi all, > > on Debian, a discussion came up on the inconsistency of some config > statements. The right cure would be to correct this, but that requires some > time. I would appreciate your feedback on the issue. The details can be found > here: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 First off, the config language really has two parts. The first part is the configuration of rsyslog, the second part is the instructions to rsyslog telling it what to do with the log messages it has received.. I think you are wildly optimistic if you think that "a year from now nobody will be using the old config language". I expect that 3-5 years from now users will still be using 3.x versions (because they are what was released in 'enterprise' releases) even for new versions there will need to be a transition path (including the ability to easily revert to an old version) I fully understand the desire to scrap the existing evelved config language and implement a new, designed version instead (without needing to support both), but I don't see that as being realistic. A new designed config language should be able to have all the existing configs map to it, and I expect that you will have to maintain a shim/transition layer for many years that will read the old configs and map it to the new config. for the first part of the configs (the configuration options for rsyslog itself, having nothing to do with processing the log messages) As such, I think making the infrastructure to log 'this config option has been replaced by that config option' could actually end up cleaning things up and making it easier to modify the config language going forward. If the logging of changed is done one option at a time it will be a mess, but if it's done as something like a mapping layer it could actually be better. I'm thinking of a table along the lines of config_option, type, destination where type could be 'boolean', 'string', 'filesystem path', 'number', or 'depriciated', 'ignore' if the type is depriciated then the destination would be another config_option (which you would then look up) if the type is 'ignore' then it would just be skipped over (possibly with a warning being logged that the config line is being ignored) this would also clean up some of the current cases where a boolean option doesn't honor on/off and true/false. For the second half of the config language (telling rsyslog what to do with the logs it has received) also has several variations in place and is showing issues. However I think that the right answer here is to end up implementing something like the parsing trees and then compile the other config language options into that tree to be consistant (and fast) under the covers, no matter which way the instructions are written (except when you have to use regex matches) David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:11:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:11:55 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Hi David, thanks for the thoughtful comments. But I seem to have incorrectly phrased my question, leaving out too many of those things that I obviously took for granted and thus not mentioned :(... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:45 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Thu, 25 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > on Debian, a discussion came up on the inconsistency of some config > > statements. The right cure would be to correct this, but that > requires some > > time. I would appreciate your feedback on the issue. The details can > be found > > here: > > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 > > First off, the config language really has two parts. > > The first part is the configuration of rsyslog, the second part is the > instructions to rsyslog telling it what to do with the log messages it > has > received.. > > I think you are wildly optimistic if you think that "a year from now > nobody will be using the old config language". I expect that 3-5 years > from now users will still be using 3.x versions (because they are what > was > released in 'enterprise' releases) > That was probably the most important thing I did not mention: I do NOT think that removing inconsistencies in the config language is a bug fix. And if it is not a bug fix, it is a feature request. But if it is a feature request, it will go into the recent devel. And as such it will be a v5-exclusive thing. As such, what is with v3 and v4 doesn't really matter, because it would not be changed (and, yes, I totally agree that v3/3 will be around for a long time). One may argue if this feature is so essential that it should be rolled back into the old versions. However, I became very skeptic of doing so. Almost always in the past years, when I put some non-essential change into the stable releases directly, big problems were the result. I could probably open up new devel/beta branches for v3/v4 as well. But that would a) cause a lot of extra work b) people would most probably not try it out so we would end up with the same issue when it is then named stable c) the experience with two v4/v5 devel versions was a bad one as well. It is too easy to mix up where the regression is And now to the time slot: I guess, when a clean config language is there, most people would tend to use the new format. This is why I assumed that after a year the old language (in v5!) will most probably not be used much longer. To be precise, I fear that I take out time to implement this in v5, then have the new format (let's hope) 6 month later available then wonder if that would really make a difference over not doing the first step. If I had lots of time, it would probably still make sense. But unfortunately, I currently have far less time than I wish I had. Thankfully, the newer versions get more and more adopted, but that also means more bug fixes to do. So I sadly see that my available time is currently eaten up by those bug fixes and clarifying things inside the doc so that people can use the features. That is almost painful to me, because I have so many cool things on my mind, but do not find time to implement a single one of them :( > even for new versions there will need to be a transition path > (including > the ability to easily revert to an old version) > > I fully understand the desire to scrap the existing evelved config > language and implement a new, designed version instead (without needing > to > support both), but I don't see that as being realistic. > > A new designed config language should be able to have all the existing > configs map to it, and I expect that you will have to maintain a > shim/transition layer for many years that will read the old configs and > map it to the new config. The new config language doesn't mean that the old will no longer work. It is a primary design goal to support both in parallel. I always tried hard to make rsyslog as backwards-compatible as possible and it turns out that this is a major advantage, at least at the low end. I do not intend to drop that (one need also consider existing know-how: people should be able to get rsyslog immediately up and running, for simple cases, without the need to learn anything new). > > for the first part of the configs (the configuration options for > rsyslog > itself, having nothing to do with processing the log messages) > > As such, I think making the infrastructure to log 'this config option > has > been replaced by that config option' could actually end up cleaning > things > up and making it easier to modify the config language going forward. > > If the logging of changed is done one option at a time it will be a > mess, > but if it's done as something like a mapping layer it could actually be > better. > > I'm thinking of a table along the lines of > > config_option, type, destination > > where type could be 'boolean', 'string', 'filesystem path', 'number', > or > 'depriciated', 'ignore' This *is* how the system works today! But I don't have a deprecated flag and aliases, so I would need to rewrite that whole system. BTW: this system is the reason why I avoided writing a new system until absolutely necessary: it is too easy to add a new directive as all you need to do is add a new "table entry". > > if the type is depriciated then the destination would be another > config_option (which you would then look up) > > if the type is 'ignore' then it would just be skipped over (possibly > with > a warning being logged that the config line is being ignored) > > this would also clean up some of the current cases where a boolean > option > doesn't honor on/off and true/false. True/false is NOT a valid boolean option. See http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > For the second half of the config language (telling rsyslog what to do > with the logs it has received) also has several variations in place and > is > showing issues. > > However I think that the right answer here is to end up implementing > something like the parsing trees and then compile the other config > language options into that tree to be consistant (and fast) under the > covers, no matter which way the instructions are written (except when > you > have to use regex matches) I don't fully agree here with you. For example, the traditional PRI based filter is lightning fast. I don't see any way it could be nearly as fast with any unified approach. Right now, we have a "unified" filter structure, but it contains three filter cases, each one adding potential power at the price of decreased speed. I think we need to keep different types of filters in order to have some lightning-fast ones. But more of this could be done under the hood. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 11:15:21 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:15:21 +0100 Subject: [rsyslog] rsyslog migration and syslogd.conf References: <20100227063807.5918fafc@mistral.stie> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:02 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog migration and syslogd.conf > > On Sat, 27 Feb 2010, lanas wrote: > > > Hello, > > > > First, thanks to David for the reply on time stamps. > > > > I have a question about perhaps being able to replace syslogd by > > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > > reckon with limited possibilities) with a standard syslog config > > file ? That would enable a basic set of functionality as a first > step > > toward using rsyslog's features. > > no, rsyslog will do nothing if you just feed it a syslog.conf file. Actually, rsyslog is designed as a drop-in replacement. So it should work if you just feed it a regular rsyslog.conf. This is controlled via the -c option, which sets the compatibility level. If you call rsyslogd like you called sysklogd (without the -c option), it will fall back to sysklogd compatibility and generate the necessary config commands before parsing the the config file. Then, -r starts up a UDP listener as it did in sysklogd. ... At least this is how it is meant to work, I have to admit I have not tested it lately. For various reasons, I could not add this to the automated testbench, so I might not have noticed a regression. But if it does not work, this is a bug and not how things are designed to work. Rainer > rsyslog needs a few additional instructions at the top of the file to > load > the modules, but below that you can paste in the syslog.conf file. > > another thing that you can do is to make a generic rsyslog.conf file > that > includes the syslog.conf after the rsyslog specific headers. > > My rsyslog.conf (on my ubuntu 9.10 laptop) has > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser syslog > $PrivDropToGroup syslog > $RepeatedMsgReduction on > $RepeatedMsgContainsOriginalMsg on > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sun Feb 28 11:28:18 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 02:28:18 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:30:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:30:24 +0100 Subject: [rsyslog] Log Normalization effort References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397E@GRFEXC.intern.adiscon.com> David, thanks again for the good feedback. I have a couple of points: First of all: I agree that a "one size fits all approach" does not exist. Thus the idea to craft a library, which than can be used at various places: e.g. inside message parsers, output modules or even processing runs outside of rsyslog. Secondly, an efficient normalizing parser tree must not be much slower than the regular parser. I think that the parser overhead will be very acceptable for average messages. Another story, however, is the normalized data that has been gathered. In short words, that is extra data, so copy overhead is much higher everywhere. Also, accessing the properties take some time. I guess that's the primary problem inside a real-time solution, even if very efficient lookup methods are used. On the normalized properties: I think it is really worth the effort to try to define an as-broad as possible set of normalized properties. But that does not imply all needs to be done at once. It can be an evolutionary approach. First of all, one would look for pretty obvious things like traffic flow data (one may even reuse existing data models like from ipfix!) and user login/logoff activity. The key point IMHO is that we would just need to gather what people so far have used and try to get folks to re-use these fields that already exist. If we have success with this approach, I think, we will have huge benefit in the reporting and analysis area where programs could then work on standard property sets (of normalized syslog data) rather than on the raw data itself. So you do not need to write an analyzer for each of the myriad vendor/device/version formats but rather only once for the normalized data and then create a one-line (!) parse template for each of the vendor/device/version formats (of course, for each message there, so it would be multiple lines, but very easy and intuitively to do). However, all of that does not really work without community involvement. None (expect large commercial entities) can create the necessary mass of parse templates alone. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 26, 2010 10:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Log Normalization effort > > On Fri, 26 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > I have blogged about my quest for log normalization. I think there is > some > > good information on the upcoming GPLed Adiscon LogAnalyzer and future > > directions for rsyslog in the blog post. So I thought I share the > link: > > > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > > > Please note that part of the effort requires community involvement. I > would > > be very interested to learn if you think we could win enough support > to make > > this a useful effort. I am asking for your feedback, because it will > help me > > streamline my priorities for future rsyslog work. > > a few comments (but remember that I am usually dealing with high data > rates, so my concerns are biased in that direction) > > log analysis is usually done in batches as opposed to in real-time. > some of this is due to the difficulty in doing it in real time, but a > lot > of it is the processing overhead (you don't want to take so long to > process an individual request that you miss the next one to arrive) > > at low volumes the idea of name-value pairs in the logs makes a lot of > sense, but there is significantly more overhead in parsing a log with > name-value pairs in arbitrary orders than there is in using a tree > parsing > approach to analyze known log formats in a fixed order. The message > size > can also increase significantly. As a result, at high traffic volumes > this > starts to be a bad (or at least questionable) idea. > > I would love to see rsyslog gain the ability to efficiently do tree- > based > parsing instead of regex parsing. regex parsing is easy to understand > and > tinker with, but very expensive to implement. it may be that having > something that 'compiles' a list of regex parsers into a tree parser is > the right answer for usability. I would save several hours of > processing > a day if I could easily (and efficiently) make rsyslog write different > logs to different files (at high data rates and with a few hundred > conditions based variations in the syslog tag) > > > While there are some common events across different types of logs > (logins > for example) they almost always contain slightly different data in > them. I > also have no faith at all that anyone is going to make much effort to > clean up their logs to make them nicely parseable, and if they do I see > even less chance that they will end up using the same terms for the > same > thing. As such I see more value in trying to get samples of logs and > what > they mean than in trying to define a normalized version to shoehorn the > logs into. It is worth doing this for some events (logins, failed > logins > for example), but I think it's a mistake to think that this will end up > covering all, or even the majority of log messages. > > There's also a problem in that the ideal format for the output depends > on > what you are doing with the output. > > > If I could wave a magic wand and get the result I would look for > something > like this > > the parser starts at the beginning of the message (at the priority) and > can branch on priority/faclilty, timestamp, host, syslogtag, message > and > indicate if the message should be parsed into name-value pairs, or > split > based on a character (or character sequence like the perl split command > allows) into individually addressable elements (defaulting to > whitespace > separated elements), then the format (and if needed dynafile path/file > components) could be constructed from these variables. At any point in > the > parsing it should be possible to jump to another parser tree (so that > you > could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end > up > using the same parser for the message without having to redefine the > rules > for each one) > > With this capability, people could start writing parser 'branches' to > understand a specific log type and output a 'standard' format (as such > a > format can start to be defined). > > This can be done in rsyslog today, but it is fairly difficult to > define, > and as I understand it, inefficient enough that it's not practical to > do > in real-time under heavy load. > > > If this is fast enough, then the next step would be to add the ability > to > have the format/action be 'increment a counter for log type X' and a > signal to rsyslog could generate a report on these counters. Although > at > some point it becomes better to feed the message into another > opensource > tool (SEC, Simple Event Correlator for example) instead of trying to do > everything in rsyslog. > > parsing the file to know what to do with it, and be able to re-format > log > messages is very defiantly something that can fit into the rsyslog > model > of receiving, formatting, and delivering logs. Alerting on specific log > entries, counting the number of times one thing shows up in logs, and > this sort of thing start pushing > beyond the core of rsyslog, and it may be better to feed other tools > instead. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 12:25:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 12:25:14 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 28.02.10 11:28 Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sun Feb 28 14:01:58 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 05:01:58 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <001201cab868$d148da7c$100013ac@intern.adiscon.com> References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: > Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. > > I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? I am probably using the wrong terminology here. say you have a set of rules that say if programname startswith abc log to /abc if programname startswith acc log to /acc if programname startswith acc log to @1.1.1.1 if programname startswith bcd log to /bcd and assuming that these were the only possible programnames for simplicity in the explination. the filtering logic would go somthing like this the rules would compile into a tree -a-b -> /abc -c -> /acc, at 1.1.1.1 -b -> /bcd receive programname abc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "b", There are no other decisions to make, invoke the action /abc receive progranmane bcd I have no facility/PRI filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's a "b", There are no other decisions to make, invoke the action /bcd receive programname acc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "c", There are no other decisions to make, invoke the action /acc and the action @1.1.1.1 similarly the facility/pri filtering rules would be either compiled into a tree, or (since it's 256 entries total) into a lookup table with pointers to the list of actions to take for that entry the idea is to spend the time at startup to create the tree that represents the ruleset in order to speed up the processing of each individual message. the real tree would be a bit more complicated than I describe here as it needs to have 'anything else' entries between the known entries (which means that it would not be able to shortcut quite as much as I describe), and have provision for 'do a more complicated thing (like regex) here and if it matches continue'. but except for regex matches, this would make processing the rules pretty much independant of how many rules there were or how complex the ruleset is. there doe snot need to be one single programname filter tree, if you had a rule that said if pri = info and programname startswith def then def the pri tree/table would have an entry for info that would point to a filter tree for programname that just had the check for def in it am I makeing any more sense now? David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 28.02.10 11:28 > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > >>> >>> if the type is depriciated then the destination would be another >>> config_option (which you would then look up) >>> >>> if the type is 'ignore' then it would just be skipped over (possibly >>> with >>> a warning being logged that the config line is being ignored) >>> >>> this would also clean up some of the current cases where a boolean >>> option >>> doesn't honor on/off and true/false. >> >> True/false is NOT a valid boolean option. See >> >> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > I was not remembering correctly then, maby it was yes/no vs on/off. I know > I ran into something where what was documented didn't work and I changed > it to another version of a boolean answer and it fixed the problem. > >>> For the second half of the config language (telling rsyslog what to do >>> with the logs it has received) also has several variations in place and >>> is >>> showing issues. >>> >>> However I think that the right answer here is to end up implementing >>> something like the parsing trees and then compile the other config >>> language options into that tree to be consistant (and fast) under the >>> covers, no matter which way the instructions are written (except when >>> you >>> have to use regex matches) >> >> I don't fully agree here with you. For example, the traditional PRI based >> filter is lightning fast. I don't see any way it could be nearly as fast with >> any unified approach. Right now, we have a "unified" filter structure, but it >> contains three filter cases, each one adding potential power at the price of >> decreased speed. I think we need to keep different types of filters in order >> to have some lightning-fast ones. But more of this could be done under the >> hood. > > My guess/expectation is that the tree-based parsing would be about the > same speed as the traditional PRI based filter for choices made based on > PRI, slowing down for other types of conditions only in that more of the > message may need to be scanned (and if there is no selection at a level, > skipping that level should be lightning fast). As a result, a set of > filteres based soley on programname would be as fast as the current PRI > filters. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Sun Feb 28 14:15:46 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Sun, 28 Feb 2010 14:15:46 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Michael Biebl : > 2010/2/26 Rainer Gerhards : >>> The problem is, that xconsole is running at this point! >>> To reiterate: >>> I start rsyslogd >>> then xconsole >>> then try to log a message >>> The result is the debug log and no message showing up in xconsole. >> >> that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Hm, it just happened again. I rebooted my system, logged into my system, ran "sudo xconsole", "logger foo", but nothing shows up in the xconsole window. I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, "logger foo" and this time it works. Weird. There is definitely something fishy here. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Sun Feb 28 14:56:30 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 14:56:30 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com> Hi David, yes, that makes much more sense to me. Let's call that filter expression tree for the time being (the parse tree used for normalization has some similar ideas, but a totally different scope and semantics). While I now see where you are heading, I fail to see how this could sufficiently easy be implemented for a real (and complex) configuration. Let's look at this example conf: mail.* /Var/log/mail.log if programname startswith abc log to /abc if programname startswith 'acc' and MSG contains 'error' log to /acc # we don't want debug messages from here on *.debug ~ if programname startswith 'acc' or (programname contains 'bde' and (facility > 1 and facility < 5)) log to @1.1.1.1 if programname startswith bcd or MSG startswith '123' or MSG contains 'error2' log to /bcd *.* /var/log/catchall I simply cannot envision how you would store this as a tree that does not look like what we have today. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 2:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > Quickly from the mobile: pri based filters require *excatly* one > single word (!) lookup currently. You can't beat that. > > > > I am not totally sure how you think about parse trees. In my view, > parsin and filteing are two different things./stages. Maybe we have > different concepts on our minds? > > I am probably using the wrong terminology here. > > say you have a set of rules that say > > if programname startswith abc log to /abc > if programname startswith acc log to /acc > if programname startswith acc log to @1.1.1.1 > if programname startswith bcd log to /bcd > > and assuming that these were the only possible programnames for > simplicity > in the explination. > > the filtering logic would go somthing like this > > the rules would compile into a tree > > -a-b -> /abc > -c -> /acc, at 1.1.1.1 > -b -> /bcd > > receive programname abc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "b", There are no other decisions to make, > invoke the action /abc > > receive progranmane bcd > I have no facility/PRI filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's a "b", There are no other decisions to make, > invoke the action /bcd > > receive programname acc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "c", There are no other decisions to make, > invoke the action /acc and the action @1.1.1.1 > > similarly the facility/pri filtering rules would be either compiled > into a > tree, or (since it's 256 entries total) into a lookup table with > pointers > to the list of actions to take for that entry > > the idea is to spend the time at startup to create the tree that > represents the ruleset in order to speed up the processing of each > individual message. > > the real tree would be a bit more complicated than I describe here as > it > needs to have 'anything else' entries between the known entries (which > means that it would not be able to shortcut quite as much as I > describe), > and have provision for 'do a more complicated thing (like regex) here > and > if it matches continue'. but except for regex matches, this would make > processing the rules pretty much independant of how many rules there > were > or how complex the ruleset is. > > there doe snot need to be one single programname filter tree, if you > had a > rule that said > if pri = info and programname startswith def then def > > the pri tree/table would have an entry for info that would point to a > filter tree for programname that just had the check for def in it > > am I makeing any more sense now? > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 28.02.10 11:28 > > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > >>> > >>> if the type is depriciated then the destination would be another > >>> config_option (which you would then look up) > >>> > >>> if the type is 'ignore' then it would just be skipped over > (possibly > >>> with > >>> a warning being logged that the config line is being ignored) > >>> > >>> this would also clean up some of the current cases where a boolean > >>> option > >>> doesn't honor on/off and true/false. > >> > >> True/false is NOT a valid boolean option. See > >> > >> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18 > 4c0d87 > >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > > I was not remembering correctly then, maby it was yes/no vs on/off. I > know > > I ran into something where what was documented didn't work and I > changed > > it to another version of a boolean answer and it fixed the problem. > > > >>> For the second half of the config language (telling rsyslog what to > do > >>> with the logs it has received) also has several variations in place > and > >>> is > >>> showing issues. > >>> > >>> However I think that the right answer here is to end up > implementing > >>> something like the parsing trees and then compile the other config > >>> language options into that tree to be consistant (and fast) under > the > >>> covers, no matter which way the instructions are written (except > when > >>> you > >>> have to use regex matches) > >> > >> I don't fully agree here with you. For example, the traditional PRI > based > >> filter is lightning fast. I don't see any way it could be nearly as > fast with > >> any unified approach. Right now, we have a "unified" filter > structure, but it > >> contains three filter cases, each one adding potential power at the > price of > >> decreased speed. I think we need to keep different types of filters > in order > >> to have some lightning-fast ones. But more of this could be done > under the > >> hood. > > > > My guess/expectation is that the tree-based parsing would be about > the > > same speed as the traditional PRI based filter for choices made based > on > > PRI, slowing down for other types of conditions only in that more of > the > > message may need to be scanned (and if there is no selection at a > level, > > skipping that level should be lightning fast). As a result, a set of > > filteres based soley on programname would be as fast as the current > PRI > > filters. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 15:01:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 15:01:54 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Sunday, February 28, 2010 2:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Michael Biebl : > > 2010/2/26 Rainer Gerhards : > >>> The problem is, that xconsole is running at this point! > >>> To reiterate: > >>> I start rsyslogd > >>> then xconsole > >>> then try to log a message > >>> The result is the debug log and no message showing up in xconsole. > >> > >> that's what I am doing as well. I will try to record the session. > > > > Wait a sec. > > With 4.4.2 (or any earlier version) I tried this: > > start rsyslog > > start xconsole > > xconsole shows the last two log messages immediately > > > > With 4.6.0 the xconsole window is blank (which I guess made me > believe > > it blocked) > > > > With 4.6.0, I need to trigger a new log message (e.g. using logger > > foo), and then xconsole will show a new message. > > > > So, xconsole does indeed show messages, but the behaviour is > different > > to 4.4 and earlier > > Hm, it just happened again. > I rebooted my system, logged into my system, ran "sudo xconsole", > "logger foo", but nothing shows up in the xconsole window. > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > "logger foo" and this time it works. > > Weird. There is definitely something fishy here. OK, rather than trying to figure out what may be going on here, I'll see that I copy the old omfile code and create a new ompipe. So that will behave exactly like it did before. The new optimizations are not so important for pipes. Rainer From rgerhards at hq.adiscon.com Sun Feb 28 17:33:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 17:33:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com> > > Hm, it just happened again. > > I rebooted my system, logged into my system, ran "sudo xconsole", > > "logger foo", but nothing shows up in the xconsole window. > > > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > > "logger foo" and this time it works. > > > > Weird. There is definitely something fishy here. > > OK, rather than trying to figure out what may be going on here, I'll > see that > I copy the old omfile code and create a new ompipe. So that will behave > exactly like it did before. The new optimizations are not so important > for > pipes. Michael, I have done this now. I would appreciate if you could checkout the v4-stable git head and try with that one. It should now have the exact same behavior as in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked like before. Thinking about it, I guess it helps ease development in the future, because I now do not need to think about pipes when I work on omfile and vice versa. Rainer From tbergfeld at hq.adiscon.com Fri Feb 5 15:41:32 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 5 Feb 2010 15:41:32 +0100 Subject: [rsyslog] rsyslog 5.5.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103822@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.5.2, a member of the v5-devel branch. Rsyslog 5.5.2 primarily provides bug-fixes, but offers an enhanced functionality to escape 8-bit characters on reception. See Changelog for more details. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-193.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From ryan.b.lynch at gmail.com Fri Feb 5 23:07:26 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 5 Feb 2010 17:07:26 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Under 5.3.6, I am trying to use property-based filtering to separate local messages (coming in via imuxsock, presumably) from remote messages (coming in via UDP). Initially, I tried using the 'inputname' property to distinguish between the two: :inputname, isequal, "imudp" /remote_udp.log :inputname, isequal, "imuxsock" /local_socket.log With this config, nothing ever ends up in the 2nd file (/local_socket.log)--instead, because the 'inputname' property is always set to 'imudp', regardless of where the input originates. The property replacer docs mention that the value of 'inputname' isn't guaranteed, so I guess this isn't necessarily a bug. Also, I found a comment in the source code for 'imuxsock.c' that makes this behavior seem to be intentional. Why does only imuxsock behave like this? The other input modules I've used (imrelp, imklog) provide the expected 'inputname' values ('imrelp' and 'imklog', respectively), which is intuitive and consistent. Is there a reason why imuxsock needs to be different, or is this a bug? (In case anyone is wondering, I could work around this by comparing the 'hostname' and '$myhostname' properties, in addition to checking 'inputname'. But that would force me to use expression-based filters (property-based filtering doesn't support AND/OR logic), which seems to increase the CPU activity per message. I really can't afford the lost efficiency, here, if it can be helped.) Ryan B. Lynch ryan.b.lynch at gmail.com From ryan.b.lynch at gmail.com Sat Feb 6 06:13:17 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 00:13:17 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. Message-ID: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> I'm trying to use multiple rulesets and omruleset actions to send a single incoming log message to two different log files, at the same time. I think my config file will help explain it: # BEGIN rsyslog.conf $ModLoad imuxsock $ModLoad imklog $ModLoad immark $ModLoad omruleset $MarkMessagePeriod 1 $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% %timereported:::date-rfc3339% %hostname% %fromhost% %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% %msg%\n" # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', each of which writes output to its own, separate file ('RS_Alfa.log' and 'RS_Bravo.log', respectively) using an action with a *.* selector. $Ruleset RS_Bravo $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset $Ruleset RS_Alfa $RulesetCreateMainQueue on $ActionResumeInterval 1 *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile $Ruleset RSYSLOG_DefaultRuleset # SECOND: I create a pair of actions (in the default ruleset), both with *.* selectors and the action ':omruleset:', but preceded by different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and 'RS_Bravo', respectively $ActionOmrulesetRulesetName RS_Bravo $ActionResumeInterval 1 *.* :omruleset: $ActionOmrulesetRulesetName RS_Alfa $ActionResumeInterval 1 *.* :omruleset: # END rsyslog.conf Based on the documentation, I believe that incoming log messages should be processed by the default ruleset, which sends the message to both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I should see a copy of each log message written to each of the two output files, and each output file should have identical contents. But that isn't the result I'm getting, here. I get two copies of each message, but almost all of them are written to the file 'RS_Alfa'. Only the occasional lone random message is written to 'RS_Bravo'. (If I reverse the order of the :omrulset: actions, I see the oppositie pattern.) The exact proportion varies, but only about 1 in 20 messages makes it to the 1st (Bravo) output file. All different types of messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' file--again, it seems completely random to me. I never get more total messages logged than I should, though. If I run rsyslog long enough to let 10 incoming messages appear, I will see a total of 20 messages when I combine the contents of 'RS_Alfa' and 'RS_Bravo'--they just end up in the wrong files. I turned on debug logging during a short session and captured the results to a file, which I've attached, along with the rsyslog output files from that session. The host in question runs Centos Linux 5.4 as an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. If there is any more info I can provide, or any other debugging steps I should take, please let me know. Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Sat Feb 6 12:00:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:00:59 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> I will check, but the most probably explanation is that it is a bug that so far nobody noticed ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Friday, February 05, 2010 11:07 PM > To: rsyslog-users > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > as'imudp'? > > Under 5.3.6, I am trying to use property-based filtering to separate > local messages (coming in via imuxsock, presumably) from remote > messages (coming in via UDP). Initially, I tried using the 'inputname' > property to distinguish between the two: > > :inputname, isequal, "imudp" /remote_udp.log > :inputname, isequal, "imuxsock" /local_socket.log > > With this config, nothing ever ends up in the 2nd file > (/local_socket.log)--instead, because the 'inputname' property is > always set to 'imudp', regardless of where the input originates. > > The property replacer docs mention that the value of 'inputname' isn't > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > comment in the source code for 'imuxsock.c' that makes this behavior > seem to be intentional. > > Why does only imuxsock behave like this? The other input modules I've > used (imrelp, imklog) provide the expected 'inputname' values > ('imrelp' and 'imklog', respectively), which is intuitive and > consistent. Is there a reason why imuxsock needs to be different, or > is this a bug? > > (In case anyone is wondering, I could work around this by comparing > the 'hostname' and '$myhostname' properties, in addition to checking > 'inputname'. But that would force me to use expression-based filters > (property-based filtering doesn't support AND/OR logic), which seems > to increase the CPU activity per message. I really can't afford the > lost efficiency, here, if it can be helped.) > > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sat Feb 6 12:04:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 6 Feb 2010 12:04:18 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Indeed, it was a bug. Patch here: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 d5305b1f6cc0fe22 Thanks for reporting the issue! Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Saturday, February 06, 2010 12:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > propertyas'imudp'? > > I will check, but the most probably explanation is that it is a bug > that so > far nobody noticed ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > > Sent: Friday, February 05, 2010 11:07 PM > > To: rsyslog-users > > Subject: [rsyslog] Why does imuxsock report its 'inputname' property > > as'imudp'? > > > > Under 5.3.6, I am trying to use property-based filtering to separate > > local messages (coming in via imuxsock, presumably) from remote > > messages (coming in via UDP). Initially, I tried using the > 'inputname' > > property to distinguish between the two: > > > > :inputname, isequal, "imudp" /remote_udp.log > > :inputname, isequal, "imuxsock" /local_socket.log > > > > With this config, nothing ever ends up in the 2nd file > > (/local_socket.log)--instead, because the 'inputname' property is > > always set to 'imudp', regardless of where the input originates. > > > > The property replacer docs mention that the value of 'inputname' > isn't > > guaranteed, so I guess this isn't necessarily a bug. Also, I found a > > comment in the source code for 'imuxsock.c' that makes this behavior > > seem to be intentional. > > > > Why does only imuxsock behave like this? The other input modules I've > > used (imrelp, imklog) provide the expected 'inputname' values > > ('imrelp' and 'imklog', respectively), which is intuitive and > > consistent. Is there a reason why imuxsock needs to be different, or > > is this a bug? > > > > (In case anyone is wondering, I could work around this by comparing > > the 'hostname' and '$myhostname' properties, in addition to checking > > 'inputname'. But that would force me to use expression-based filters > > (property-based filtering doesn't support AND/OR logic), which seems > > to increase the CPU activity per message. I really can't afford the > > lost efficiency, here, if it can be helped.) > > > > Ryan B. Lynch > > ryan.b.lynch at gmail.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From ryan.b.lynch at gmail.com Sat Feb 6 19:27:52 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Sat, 6 Feb 2010 13:27:52 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as 'imudp'? Message-ID: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> No problem, and thank you for the patch. I'll try it out and let you know how it works. This is a huge help to me, BTW: I should be able to get my log server working properly, now, without having to worry about omruleset and multiple rulesets. (Hence my other problem report from last night.) Big sigh of relief. Not that I don't like omruleset. Like RELP, it feels really elegant, and it's easy to expressive complex log routing arrangements. I think it's one of your best ideas, to date, and I'm looking forward to trying out some crazy ideas with it. -Ryan On 2010-02-06, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 > > Thanks for reporting the issue! > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Saturday, February 06, 2010 12:01 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' >> propertyas'imudp'? >> >> I will check, but the most probably explanation is that it is a bug >> that so >> far nobody noticed ;) >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch >> > Sent: Friday, February 05, 2010 11:07 PM >> > To: rsyslog-users >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' property >> > as'imudp'? >> > >> > Under 5.3.6, I am trying to use property-based filtering to separate >> > local messages (coming in via imuxsock, presumably) from remote >> > messages (coming in via UDP). Initially, I tried using the >> 'inputname' >> > property to distinguish between the two: >> > >> > :inputname, isequal, "imudp" /remote_udp.log >> > :inputname, isequal, "imuxsock" /local_socket.log >> > >> > With this config, nothing ever ends up in the 2nd file >> > (/local_socket.log)--instead, because the 'inputname' property is >> > always set to 'imudp', regardless of where the input originates. >> > >> > The property replacer docs mention that the value of 'inputname' >> isn't >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found a >> > comment in the source code for 'imuxsock.c' that makes this behavior >> > seem to be intentional. >> > >> > Why does only imuxsock behave like this? The other input modules I've >> > used (imrelp, imklog) provide the expected 'inputname' values >> > ('imrelp' and 'imklog', respectively), which is intuitive and >> > consistent. Is there a reason why imuxsock needs to be different, or >> > is this a bug? >> > >> > (In case anyone is wondering, I could work around this by comparing >> > the 'hostname' and '$myhostname' properties, in addition to checking >> > 'inputname'. But that would force me to use expression-based filters >> > (property-based filtering doesn't support AND/OR logic), which seems >> > to increase the CPU activity per message. I really can't afford the >> > lost efficiency, here, if it can be helped.) >> > >> > Ryan B. Lynch >> > ryan.b.lynch at gmail.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Ryan B. Lynch ryan.b.lynch at gmail.com From rgerhards at hq.adiscon.com Mon Feb 8 09:06:58 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 09:06:58 +0100 Subject: [rsyslog] Why does imuxsock report its 'inputname' property as'imudp'? References: <115906d11002061027p24fd75d7s1a13ff3c65b83449@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103828@GRFEXC.intern.adiscon.com> Ryan, thanks for the feedback. I'll also try to look into the omruleset issue you reported. Looked too complex for the weekend ;) More later... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Saturday, February 06, 2010 7:28 PM > To: rsyslog-users > Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > property as'imudp'? > > No problem, and thank you for the patch. I'll try it out and let you > know how it works. > > This is a huge help to me, BTW: I should be able to get my log server > working properly, now, without having to worry about omruleset and > multiple rulesets. (Hence my other problem report from last night.) > Big sigh of relief. > > Not that I don't like omruleset. Like RELP, it feels really elegant, > and it's easy to expressive complex log routing arrangements. I think > it's one of your best ideas, to date, and I'm looking forward to > trying out some crazy ideas with it. > > -Ryan > > > On 2010-02-06, Rainer Gerhards wrote: > > Indeed, it was a bug. Patch here: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d0 > 54dac0 > > d5305b1f6cc0fe22 > > > > Thanks for reporting the issue! > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Saturday, February 06, 2010 12:01 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Why does imuxsock report its 'inputname' > >> propertyas'imudp'? > >> > >> I will check, but the most probably explanation is that it is a bug > >> that so > >> far nobody noticed ;) > >> > >> Rainer > >> > >> > -----Original Message----- > >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > >> > Sent: Friday, February 05, 2010 11:07 PM > >> > To: rsyslog-users > >> > Subject: [rsyslog] Why does imuxsock report its 'inputname' > property > >> > as'imudp'? > >> > > >> > Under 5.3.6, I am trying to use property-based filtering to > separate > >> > local messages (coming in via imuxsock, presumably) from remote > >> > messages (coming in via UDP). Initially, I tried using the > >> 'inputname' > >> > property to distinguish between the two: > >> > > >> > :inputname, isequal, "imudp" /remote_udp.log > >> > :inputname, isequal, "imuxsock" /local_socket.log > >> > > >> > With this config, nothing ever ends up in the 2nd file > >> > (/local_socket.log)--instead, because the 'inputname' property is > >> > always set to 'imudp', regardless of where the input originates. > >> > > >> > The property replacer docs mention that the value of 'inputname' > >> isn't > >> > guaranteed, so I guess this isn't necessarily a bug. Also, I found > a > >> > comment in the source code for 'imuxsock.c' that makes this > behavior > >> > seem to be intentional. > >> > > >> > Why does only imuxsock behave like this? The other input modules > I've > >> > used (imrelp, imklog) provide the expected 'inputname' values > >> > ('imrelp' and 'imklog', respectively), which is intuitive and > >> > consistent. Is there a reason why imuxsock needs to be different, > or > >> > is this a bug? > >> > > >> > (In case anyone is wondering, I could work around this by > comparing > >> > the 'hostname' and '$myhostname' properties, in addition to > checking > >> > 'inputname'. But that would force me to use expression-based > filters > >> > (property-based filtering doesn't support AND/OR logic), which > seems > >> > to increase the CPU activity per message. I really can't afford > the > >> > lost efficiency, here, if it can be helped.) > >> > > >> > Ryan B. Lynch > >> > ryan.b.lynch at gmail.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Ryan B. Lynch > ryan.b.lynch at gmail.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 10:03:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 10:03:14 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265619794.5427.490.camel@localhost> Hi Ryan, once again thanks for the bug report. I could confirm the problem and opend a bug tracker (it may make sense to subscribe to it): http://bugzilla.adiscon.com/show_bug.cgi?id=179 I am now digging into the details. Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From sergei at bslos.com Mon Feb 8 13:18:41 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 17:18:41 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 Message-ID: <201002081718.41670.sergei@bslos.com> Hi, I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit system): $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ --enable-omprog ... checking whether the compiler provides atomic builtins... no checking whether the compiler provides atomic builtins for 64 bit data types... no ... Configure script run without errors. $ make ... CC rsyslogd-omfile.o In file included from omfile.c:67: ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not available, using nul operations - rsyslogd will probably be racy!" omfile.c: In function 'getClockFileAccess': omfile.c:95: warning: implicit declaration of function 'ATOMIC_INC_AND_FETCH' CC rsyslogd-omdiscard.o CC rsyslogd-pmrfc5424.o CC rsyslogd-pmrfc3164.o CC rsyslogd-iminternal.o CC rsyslogd-pidfile.o pidfile.c: In function 'read_pid': pidfile.c:58: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result pidfile.c: In function 'write_pid': pidfile.c:116: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result CCLD rsyslogd rsyslogd-omfile.o: In function `getClockFileAccess': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools/omfile.c:95: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function `SetGlobalInputTermination': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/glbl.c:136: undefined reference to `ATOMIC_STORE_1_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function `msgDestruct': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/msg.c:802: undefined reference to `ATOMIC_INC_AND_FETCH' ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function `wtiSetState': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:109: undefined reference to `ATOMIC_STORE_1_TO_INT' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/wti.c:111: undefined reference to `ATOMIC_STORE_0_TO_INT' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function `DoDeleteBatchFromQStore': /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: undefined reference to `ATOMIC_SUB' /usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1291: undefined reference to `ATOMIC_SUB' ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/runtime/queue.c:1292: more undefined references to `ATOMIC_SUB' follow collect2: ld returned 1 exit status make[2]: *** [rsyslogd] Error 1 make[2]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' make: *** [all] Error 2 Any help is appreciated. -- Regards, Sergei Butakov From rgerhards at hq.adiscon.com Mon Feb 8 13:53:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 13:53:55 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> this looks like you have generated code for a machine which does not support atomic instructions. rsyslog needs them to work correctly. I suggest generating code for i686. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > Sent: Monday, February 08, 2010 1:19 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > Hi, > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > system): > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile \ > --enable-omprog > ... > checking whether the compiler provides atomic builtins... no > checking whether the compiler provides atomic builtins for 64 bit data > types... no > ... > > Configure script run without errors. > > > > $ make > ... > CC rsyslogd-omfile.o > In file included from omfile.c:67: > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > available, > using nul operations - rsyslogd will probably be racy!" > omfile.c: In function 'getClockFileAccess': > omfile.c:95: warning: implicit declaration of function > 'ATOMIC_INC_AND_FETCH' > CC rsyslogd-omdiscard.o > CC rsyslogd-pmrfc5424.o > CC rsyslogd-pmrfc3164.o > CC rsyslogd-iminternal.o > CC rsyslogd-pidfile.o > pidfile.c: In function 'read_pid': > pidfile.c:58: warning: ignoring return value of 'fscanf', declared with > attribute warn_unused_result > pidfile.c: In function 'write_pid': > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > with > attribute warn_unused_result > CCLD rsyslogd > rsyslogd-omfile.o: In function `getClockFileAccess': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/tools/omfile.c:95: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > `SetGlobalInputTermination': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/glbl.c:136: > undefined reference to `ATOMIC_STORE_1_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > `msgDestruct': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/msg.c:802: > undefined reference to `ATOMIC_INC_AND_FETCH' > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > `wtiSetState': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:109: > undefined reference to `ATOMIC_STORE_1_TO_INT' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/wti.c:111: > undefined reference to `ATOMIC_STORE_0_TO_INT' > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > `DoDeleteBatchFromQStore': > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > undefined reference to `ATOMIC_SUB' > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1291: > undefined reference to `ATOMIC_SUB' > ../runtime/.libs/librsyslog.a(librsyslog_la- > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > 5.3.7/runtime/queue.c:1292: > more undefined references to `ATOMIC_SUB' follow > collect2: ld returned 1 exit status > make[2]: *** [rsyslogd] Error 1 > make[2]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > make: *** [all] Error 2 > > > > Any help is appreciated. > > > -- > Regards, > Sergei Butakov > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 8 14:00:20 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 08 Feb 2010 14:00:20 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> Message-ID: <1265634020.5427.491.camel@localhost> Ryan, thanks again for the bug report. I managed to fix it. Details in the tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=179 Rainer On Sat, 2010-02-06 at 06:13 +0100, Ryan Lynch wrote: > I'm trying to use multiple rulesets and omruleset actions to send a > single incoming log message to two different log files, at the same > time. I think my config file will help explain it: > > # BEGIN rsyslog.conf > > $ModLoad imuxsock > $ModLoad imklog > $ModLoad immark > $ModLoad omruleset > > $MarkMessagePeriod 1 > > $template TP_ToFile,"%inputname% %timegenerated:::date-rfc3339% > %timereported:::date-rfc3339% %hostname% %fromhost% > %syslogfacility-text% %syslogseverity-text% %syslogtag% %programname% > %msg%\n" > > # FIRST: I set up two parallel rulesets, 'RS_Alfa' and 'RS_Bravo', > each of which writes output to its own, separate file ('RS_Alfa.log' > and 'RS_Bravo.log', respectively) using an action with a *.* selector. > > $Ruleset RS_Bravo > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Bravo.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > $Ruleset RS_Alfa > $RulesetCreateMainQueue on > $ActionResumeInterval 1 > *.* /var/log/rsyslog/.__LOG/RS_Alfa.log;TP_ToFile > $Ruleset RSYSLOG_DefaultRuleset > > # SECOND: I create a pair of actions (in the default ruleset), both > with *.* selectors and the action ':omruleset:', but preceded by > different '$ActionOmrulesetRulesetName' values ('RS_Alfa' and > 'RS_Bravo', respectively > > $ActionOmrulesetRulesetName RS_Bravo > $ActionResumeInterval 1 > *.* :omruleset: > > $ActionOmrulesetRulesetName RS_Alfa > $ActionResumeInterval 1 > *.* :omruleset: > > # END rsyslog.conf > > Based on the documentation, I believe that incoming log messages > should be processed by the default ruleset, which sends the message to > both 'RS_Alfa' and 'RS_Bravo'. If I understand this correctly, I > should see a copy of each log message written to each of the two > output files, and each output file should have identical contents. > > But that isn't the result I'm getting, here. I get two copies of each > message, but almost all of them are written to the file 'RS_Alfa'. > Only the occasional lone random message is written to 'RS_Bravo'. (If > I reverse the order of the :omrulset: actions, I see the oppositie > pattern.) The exact proportion varies, but only about 1 in 20 messages > makes it to the 1st (Bravo) output file. All different types of > messages (facility, priority, hostname, etc.) end up in the 'RS_Bravo' > file--again, it seems completely random to me. > > I never get more total messages logged than I should, though. If I run > rsyslog long enough to let 10 incoming messages appear, I will see a > total of 20 messages when I combine the contents of 'RS_Alfa' and > 'RS_Bravo'--they just end up in the wrong files. > > I turned on debug logging during a short session and captured the > results to a file, which I've attached, along with the rsyslog output > files from that session. The host in question runs Centos Linux 5.4 as > an x86_64 KVM virtual guest on an x86_64 Intel Xeon machine. > > If there is any more info I can provide, or any other debugging steps > I should take, please let me know. > > Ryan B. Lynch > ryan.b.lynch at gmail.com > > From rgerhards at hq.adiscon.com Mon Feb 8 14:10:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 8 Feb 2010 14:10:54 +0100 Subject: [rsyslog] can't compile rsyslog 5.3.7 References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> see also at the bottom of this page: http://www.rsyslog.com/doc-build_from_repo.html Raienr > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, February 08, 2010 1:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] can't compile rsyslog 5.3.7 > > this looks like you have generated code for a machine which does not > support > atomic instructions. rsyslog needs them to work correctly. I suggest > generating code for i686. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Sergei Butakov > > Sent: Monday, February 08, 2010 1:19 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] can't compile rsyslog 5.3.7 > > > > Hi, > > > > I can't compile rsyslog-5.3.7 by gcc-4.1.2 (on 32-bit > > system): > > > > > > $ ./configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/bin \ > > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \ > > --localstatedir=/var --datadir=/usr/share --infodir=/usr/share/info \ > > --mandir=/usr/share/man --enable-pgsql --enable-mail --enable-imfile > \ > > --enable-omprog > > ... > > checking whether the compiler provides atomic builtins... no > > checking whether the compiler provides atomic builtins for 64 bit > data > > types... no > > ... > > > > Configure script run without errors. > > > > > > > > $ make > > ... > > CC rsyslogd-omfile.o > > In file included from omfile.c:67: > > ../runtime/atomic.h:64:3: warning: #warning "atomic builtins not > > available, > > using nul operations - rsyslogd will probably be racy!" > > omfile.c: In function 'getClockFileAccess': > > omfile.c:95: warning: implicit declaration of function > > 'ATOMIC_INC_AND_FETCH' > > CC rsyslogd-omdiscard.o > > CC rsyslogd-pmrfc5424.o > > CC rsyslogd-pmrfc3164.o > > CC rsyslogd-iminternal.o > > CC rsyslogd-pidfile.o > > pidfile.c: In function 'read_pid': > > pidfile.c:58: warning: ignoring return value of 'fscanf', declared > with > > attribute warn_unused_result > > pidfile.c: In function 'write_pid': > > pidfile.c:116: warning: ignoring return value of 'fscanf', declared > > with > > attribute warn_unused_result > > CCLD rsyslogd > > rsyslogd-omfile.o: In function `getClockFileAccess': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/tools/omfile.c:95: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-glbl.o): In function > > `SetGlobalInputTermination': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/glbl.c:136: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-msg.o): In function > > `msgDestruct': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/msg.c:802: > > undefined reference to `ATOMIC_INC_AND_FETCH' > > ../runtime/.libs/librsyslog.a(librsyslog_la-wti.o): In function > > `wtiSetState': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:109: > > undefined reference to `ATOMIC_STORE_1_TO_INT' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/wti.c:111: > > undefined reference to `ATOMIC_STORE_0_TO_INT' > > ../runtime/.libs/librsyslog.a(librsyslog_la-queue.o): In function > > `DoDeleteBatchFromQStore': > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > undefined reference to `ATOMIC_SUB' > > /usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1291: > > undefined reference to `ATOMIC_SUB' > > ../runtime/.libs/librsyslog.a(librsyslog_la- > > queue.o):/usr/src/pakbuilder-work/server/rsyslog/rsyslog- > > 5.3.7/runtime/queue.c:1292: > > more undefined references to `ATOMIC_SUB' follow > > collect2: ld returned 1 exit status > > make[2]: *** [rsyslogd] Error 1 > > make[2]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7/tools' > > make[1]: *** [all-recursive] Error 1 > > make[1]: Leaving directory > > `/usr/src/pakbuilder-work/server/rsyslog/rsyslog-5.3.7' > > make: *** [all] Error 2 > > > > > > > > Any help is appreciated. > > > > > > -- > > Regards, > > Sergei Butakov > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From sergei at bslos.com Mon Feb 8 15:11:32 2010 From: sergei at bslos.com (Sergei Butakov) Date: Mon, 8 Feb 2010 19:11:32 +0500 Subject: [rsyslog] can't compile rsyslog 5.3.7 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> References: <201002081718.41670.sergei@bslos.com> <9B6E2A8877C38245BFB15CC491A11DA710382F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103831@GRFEXC.intern.adiscon.com> Message-ID: <201002081911.33210.sergei@bslos.com> On Monday 08 February 2010, Rainer Gerhards wrote: > see also at the bottom of this page: > > http://www.rsyslog.com/doc-build_from_repo.html Thank you! The adding CFLAGS="-march=i686" to the configure script solved my trouble. -- Regards, Sergei Butakov From ryan.b.lynch at gmail.com Mon Feb 8 23:10:35 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:10:35 -0500 Subject: [rsyslog] Why does imuxsock report its 'inputname' propertyas'imudp'? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> References: <115906d11002051407u531b9803ga245ad89f5f890d5@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA7103825@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103826@GRFEXC.intern.adiscon.com> Message-ID: <115906d11002081410v17596b5eg6a1fab390aad04d6@mail.gmail.com> On Sat, Feb 6, 2010 at 06:04, Rainer Gerhards wrote: > Indeed, it was a bug. Patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=85a0ed1ccefc9bf0d054dac0 > d5305b1f6cc0fe22 Works for me with 5.3.7, with different line alignment in the patch. That might be due to 'rpmbuild' and its built-in patch macros. In case it saves anyone else time, I've attached a copy of what works for me. -Ryan From ryan.b.lynch at gmail.com Mon Feb 8 23:16:25 2010 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Mon, 8 Feb 2010 17:16:25 -0500 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. In-Reply-To: <1265634020.5427.491.camel@localhost> References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> Message-ID: <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards wrote: > Ryan, > > thanks again for the bug report. I managed to fix it. Details in the > tracker: > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 Works fine for me with 5.3.7. Thanks! From rgerhards at hq.adiscon.com Tue Feb 9 07:43:48 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 9 Feb 2010 07:43:48 +0100 Subject: [rsyslog] Possible bug in 5.3.6: Multiple $Ruleset, :omruleset cause random output destination selection. References: <115906d11002052113u318a159ci6c0cb9c0e7091598@mail.gmail.com> <1265634020.5427.491.camel@localhost> <115906d11002081416sa83f3euc57be7784d906961@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710383F@GRFEXC.intern.adiscon.com> Thanks for the feedback. I think I can do the "official" merge today :) > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ryan Lynch > Sent: Monday, February 08, 2010 11:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] Possible bug in 5.3.6: Multiple > $Ruleset,:omruleset cause random output destination selection. > > On Mon, Feb 8, 2010 at 08:00, Rainer Gerhards > wrote: > > Ryan, > > > > thanks again for the bug report. I managed to fix it. Details in the > > tracker: > > > > http://bugzilla.adiscon.com/show_bug.cgi?id=179 > > Works fine for me with 5.3.7. Thanks! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikolaj at kucharski.name Wed Feb 10 13:02:30 2010 From: mikolaj at kucharski.name (Mikolaj Kucharski) Date: Wed, 10 Feb 2010 12:02:30 +0000 Subject: [rsyslog] last message repeated n times from remote hosts Message-ID: <20100210120230.GI3838@x40.openbsd.home.lan> Hi, I have few Linux based machines with sysklogd installed, and I have central syslog server based on CentOS 5 with rsyslog-2.0.6-1.el5. I have issue with missing hostname when sysklog sends "last message repeated N times" over the wire to rsyslog server. Let's have a look. I used logger(1) to repeatedly sent one message few times followed by one different message. Here is what I see in the log file on my central rsyslog server: Feb 10 11:39:46 10.101.43.124 root: remote test start Feb 10 11:39:54 last message repeated 14 times Feb 10 11:39:54 10.101.43.124 root: remote test end and here is tcpdump(8) log from the source (10.101.43.124) machine: 11:39:46.642297 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 28 0x0000: 4500 0038 0000 4000 4011 d1fb 0a65 2b7c E..8.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0024 68ef 3c31 333e .e(t.....$h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 7374 6172 740a t.start. 11:39:54.904820 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 35 0x0000: 4500 003f 0000 4000 4011 d1f4 0a65 2b7c E..?.. at .@....e+| 0x0010: 0a65 2874 0202 0202 002b 68f6 3c31 333e .e(t.....+h.<13> 0x0020: 6c61 7374 206d 6573 7361 6765 2072 6570 last.message.rep 0x0030: 6561 7465 6420 3134 2074 696d 6573 0a eated.14.times. 11:39:54.904826 IP 10.101.43.124.syslog > 10.101.40.116.syslog: SYSLOG user.notice, length: 26 0x0000: 4500 0036 0000 4000 4011 d1fd 0a65 2b7c E..6.. at .@....e+| 0x0010: 0a65 2874 0202 0202 0022 68ed 3c31 333e .e(t....."h.<13> 0x0020: 726f 6f74 3a20 7265 6d6f 7465 2074 6573 root:.remote.tes 0x0030: 7420 656e 640a t.end. I searched the list, and saw a comment which say the fault is on the sysklogd end as it never sends hostname in the repeated-n-times packet, but from above I cannot see that it ever sends the packet with hostname, so I think the issues is on rsyslog side, and not on the sysklogd. Could someone shed some light on my issue, as I would like to see all the time the source IP or hostname of incomming messages to rsyslog daemon. Is this missing source hostname/IP a bug of rsyslog? Is there any way to workaround that? Thanks. PS1. I cannot change client machines, I cannot reinstall them with different syslog implementation, the only machine where I have permission to do modifications is central rsyslog server. PS2. I know about DNS and RevDNS and yes, above server doesn't have revDNS setup. -- best regards q# From tbergfeld at hq.adiscon.com Wed Feb 10 15:31:17 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 10 Feb 2010 15:31:17 +0100 Subject: [rsyslog] rsyslog 4.5.8 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710385E@GRFEXC.intern.adiscon.com> Hi all, We have just release rsyslog 4.5.8, a member of the v4-beta branch. This is a bug-fixing release, fixing all those issues that have been reported the past weeks. Full details are available in the change log. This is scheduled to replace the current v4-stable, and conclude the v4 development efforts. So we would appreciate if you could give it a try and report any issues you see with it. ChangeLog: http://www.rsyslog.com/Article439.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-194.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Feb 16 10:13:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 16 Feb 2010 10:13:14 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Hi folks, almost by accident, I learned today that rsyslog seems to be the default syslogd for SuSe as well: http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 I am a bit surprised I never heared about that change, but I have to admit I am quite happy that rsyslog now seems to be the default syslogd on (almost) all important distros... I just thought I share the good news. Rainer From mbiebl at gmail.com Tue Feb 16 15:29:57 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 16 Feb 2010 15:29:57 +0100 Subject: [rsyslog] rsyslog default on SuSe 11.2... In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71038A4@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/16 Rainer Gerhards : > Hi folks, > > almost by accident, I learned today that rsyslog seems to be the default > syslogd for SuSe as well: > > http://www.novell.com/linux/releasenotes/x86_64/openSUSE/11.2/#10 > > I am a bit surprised I never heared about that change, but I have to admit I > am quite happy that rsyslog now seems to be the default syslogd on (almost) > all important distros... > > I just thought I share the good news. Congrats Rainer for your hard work and persistence! Let's welcome our openSUSE friends on board :-) Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Sun Feb 21 20:31:40 2010 From: reader at newsguy.com (Harry Putnam) Date: Sun, 21 Feb 2010 13:31:40 -0600 Subject: [rsyslog] [basic] No remote logs getting thru Message-ID: <87iq9qmllv.fsf@newsguy.com> Not sure if the localmachine is not sending or the remote not receiving. But it appears the logs expected to show up on the remote are not there. Both ends are gentoo linux, I'm not at all familiar with rsyslog but hoped I set it up right according to what I see on the example file and at: http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications Local and remote configs appear in full [inlined] at the end. Just a few lines (including commented ones) from local conf that I hoped were what I needed to send the logs: # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host The above last 2 commented lines would be less confusing of they used the same notation for remote host in both instances. remote host remote-host Appears to be talking about two different things *.* @@remote-host So is that really all it needs? ------- --------- ---=--- --------- -------- Now a few lines from the remote rsyslog.conf # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. That is all it appears to indicate is needed to receive logs. And this (the remote) instance of rsyslog is being started with -c3 -t514 flags. ------- --------- ---=--- --------- -------- Complete LOCAL rsyslog conf: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. $WorkDirectory /var/rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional # remote host is: # logsrv:512 # e.g. 192.168.0.1:514, port optional *.* @@192.168.0.26:514 # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 ------- --------- ---=--- --------- -------- Complete remote rsyslog.conf $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # [HP 112609_192532 Line above will cause rsyslog to use the previous # sysklogd date/time instead of the obnoxious but precision `new' format] #] # rsyslog v3: load input # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. $ModLoad immark.so # provides --MARK-- message capability $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) $ModLoad imklog.so # kernel logging (formerly provided by rklogd) # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none -/var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/mail.log # Log cron stuff cron.* -/var/log/cron.log # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit -/var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log *.debug;mail.none;news.none -/var/log/debug.log # Remote Logging (we use TCP for reliable delivery) # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /rsyslog/spool # where to place spool files #$ActionQueueFileName uniqName # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinety retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) $ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 From mrdemeanour at jackpot.uk.net Mon Feb 22 07:27:28 2010 From: mrdemeanour at jackpot.uk.net (Mr. Demeanour) Date: Mon, 22 Feb 2010 06:27:28 +0000 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87iq9qmllv.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <4B8223D0.8040503@jackpot.uk.net> Harry Putnam wrote: > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_--_starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. Hi, You seem to have all of the output ('action') configuration on the client machine - that's all the stuff starting with *.*, local1.* etc. As a consequence only the client machine is ever going to output anything. Please verify that the client machine is indeed logging to disk in /var/log. If you now set up a catch-all action line in the server's rsyslog.conf (and restart the service on both machines), I prophesy that output originating from the client will start appearing on the server. E.g.: *.* /var/log/catchall.log You can test using the logger command on the client, e.g. $ logger -p local1.info "Test log message from client" -- Jack. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none -/var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* -/var/log/mail.log > > > # Log cron stuff > cron.* -/var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit -/var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > *.debug;mail.none;news.none -/var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Feb 22 08:40:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 08:40:49 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Harry, I think this may contain useful information for you: http://cookbook.rsyslog.com/node13.html there is also a guide on receiving messages and putting them to different bins. Looking briefly at your config, I'd suggest that you use explicit port numbers (e.g. 10514) and make sure they are enabled in our firewall. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Sunday, February 21, 2010 8:32 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [basic] No remote logs getting thru > > Not sure if the localmachine is not sending or the remote not > receiving. But it appears the logs expected to show up on the remote > are not there. > > Both ends are gentoo linux, I'm not at all familiar with rsyslog but > hoped I set it up right according to what I see on the example file > and at: > http://wiki.rsyslog.com/index.php/Very_simple_config_-- > _starting_point_for_modifications > > Local and remote configs appear in full [inlined] at the end. > > Just a few lines (including commented ones) from local conf that I > hoped were what I needed to send the logs: > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > The above last 2 commented lines would be less confusing of they > used the same notation for remote host in both instances. > > remote host > remote-host > > Appears to be talking about two different things > > *.* @@remote-host > > So is that really all it needs? > ------- --------- ---=--- --------- -------- > > Now a few lines from the remote rsyslog.conf > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support > it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > That is all it appears to indicate is needed to receive logs. > > And this (the remote) instance of rsyslog is being started with > > -c3 -t514 flags. > > ------- --------- ---=--- --------- -------- > > Complete LOCAL rsyslog conf: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input modules > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > $WorkDirectory /var/rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > # remote host is: > # logsrv:512 # e.g. 192.168.0.1:514, port optional > *.* @@192.168.0.26:514 > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > ------- --------- ---=--- --------- -------- > > Complete remote rsyslog.conf > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > # [HP 112609_192532 Line above will cause rsyslog to use the previous > # sysklogd date/time instead of the obnoxious but precision `new' > format] > #] > # rsyslog v3: load input > # If you do not load inputs, nothing happens! > # You may need to set the module load path if modules are not found. > > $ModLoad immark.so # provides --MARK-- message capability > $ModLoad imuxsock.so # provides support for local system logging (e.g. > via logger command) > $ModLoad imklog.so # kernel logging (formerly provided by rklogd) > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > # Log anything (except mail) of level info or higher. > # Don't log private authentication messages! > *.info;mail.none;authpriv.none;cron.none - > /var/log/messages > > # The authpriv file has restricted access. > authpriv.* /var/log/secure > > # Log all the mail messages in one place. > mail.* - > /var/log/mail.log > > > # Log cron stuff > cron.* - > /var/log/cron.log > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit - > /var/log/spooler > > # Save boot messages also to boot.log > local7.* > /var/log/boot.log > > *.debug;mail.none;news.none - > /var/log/debug.log > > # Remote Logging (we use TCP for reliable delivery) > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to disk and sent when it is up again. > #$WorkDirectory /rsyslog/spool # where to place spool files > #$ActionQueueFileName uniqName # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinety retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host > > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > $ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Mon Feb 22 17:25:02 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 10:25:02 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> Message-ID: <87wry5kzkx.fsf@newsguy.com> "Rainer Gerhards" writes: > Harry, > > I think this may contain useful information for you: > > http://cookbook.rsyslog.com/node13.html From:http://cookbook.rsyslog.com/node14.html *.* @@192.0.2.1:10514 # if you need to forward to other systems as well, just # add additional config lines: *.* @@other-server.example.net:10514 I must be missing something... (I am a novice with this) but I don't see anything different in the suggested config from what I've already posted in the client config. Except it shows a second remote server being sent logs... which I'm not trying to do. Also what is needed in the server to make rsyslog listen to port 514? As I've mentioned, that (server) instance of rsyslog is being started with -t514. I've seen nothing so far either in the stock rsyslog.conf example configs or at: http://cookbook.rsyslog.com/node13.html to indicate some better or more precise way to get the server to listen on 514 On the server: `ps wwaux' shows how the server is running: ps wwaux|grep rsyslog (wrapped for mail) root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ /var/run/rsyslogd.pid -f /etc/rsyslog.conf There are no firewalls involved here... its a local network behind a firewall router so all rsyslog traffic is only on the local side. I'm not sure if it means anything but trying: telnet server_host 514 from the client machine does fail to connect. And the attempt does not show up in the catchall ouput on server either. From rgerhards at hq.adiscon.com Mon Feb 22 18:51:36 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 22 Feb 2010 18:51:36 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Monday, February 22, 2010 5:25 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] [basic] No remote logs getting thru > > "Rainer Gerhards" writes: > > > Harry, > > > > I think this may contain useful information for you: > > > > http://cookbook.rsyslog.com/node13.html > > From:http://cookbook.rsyslog.com/node14.html > > *.* @@192.0.2.1:10514 > # if you need to forward to other systems as well, just > # add additional config lines: > *.* @@other-server.example.net:10514 > > I must be missing something... (I am a novice with this) but I don't > see anything different in the suggested config from what I've already > posted in the client config. Except it shows a second remote server > being sent logs... which I'm not trying to do. > > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. > > I've seen nothing so far either in the stock rsyslog.conf example > configs or at: http://cookbook.rsyslog.com/node13.html to indicate > some better or more precise way to get the server to listen on 514 see http://cookbook.rsyslog.com/node24.html and http://www.rsyslog.com/doc-imtcp.html Rainer > > On the server: > > `ps wwaux' shows how the server is running: > > ps wwaux|grep rsyslog (wrapped for mail) > > root 21399 0.0 0.1 35192 1208 ? Sl Feb25 \ > 0:00 /usr/sbin/rsyslogd -c3 -t514 -i \ > /var/run/rsyslogd.pid -f /etc/rsyslog.conf > > There are no firewalls involved here... its a local network behind a > firewall router so all rsyslog traffic is only on the local side. > > I'm not sure if it means anything but trying: > telnet server_host 514 > > from the client machine does fail to connect. And the attempt does > not show up in the catchall ouput on server either. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Mon Feb 22 18:46:26 2010 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 22 Feb 2010 10:46:26 -0700 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87wry5kzkx.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> Message-ID: On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: > Also what is needed in the server to make rsyslog listen to port 514? > As I've mentioned, that (server) instance of rsyslog is being started > with -t514. You loaded the module but I don't see anything explicitly starting inputTCP (not sure if this is required or not). In my config I have: $ModLoad imtcp.so # load module $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 I also have: # UDP Syslog Server: $ModLoad imudp.so # provides UDP syslog reception $UDPServerRun 514 # start a UDP syslog server at standard port 514 Generally - there's not much harm running both afaik and it makes it easier when you have a device that doesn't do TCP (network gear). > I'm not sure if it means anything but trying: > ?telnet server_host 514 > > from the client machine does fail to connect. ?And the attempt does > not show up in the catchall ouput on server either. Yeah, that's. bad. Try the above, and if you still cannot connect remotely run this on the syslog server: sudo netstat -alnp | grep 514 You may get some other stuff but you should get a line that looks like this: tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 29233/rsyslogd That tells you rsyslogd is listening on port 514 - if you did all of the above you should also see: udp 0 0 0.0.0.0:514 0.0.0.0:* 29233/rsyslogd Which tells you it's listening on udp 514. Hope this helps, Aaron From reader at newsguy.com Mon Feb 22 21:41:42 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 14:41:42 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> Message-ID: <87k4u5j94p.fsf@newsguy.com> "Rainer Gerhards" writes: [...] >> I must be missing something... (I am a novice with this) but I don't >> see anything different in the suggested config from what I've already >> posted in the client config. Except it shows a second remote server >> being sent logs... which I'm not trying to do. [...] >> I've seen nothing so far either in the stock rsyslog.conf example >> configs or at: http://cookbook.rsyslog.com/node13.html to indicate >> some better or more precise way to get the server to listen on 514 > > see http://cookbook.rsyslog.com/node24.html > and http://www.rsyslog.com/doc-imtcp.html Thanks for your patience Rainer... ok I get it now. But see comment at the end. Oh, and what about that dashed notation in the example file like: *.info;mail.none;authpriv.none;cron.none -/var/log/messages What role does the dash play? [...] Aaron Nichols writes: > On Mon, Feb 22, 2010 at 9:25 AM, Harry Putnam wrote: >> Also what is needed in the server to make rsyslog listen to port 514? >> As I've mentioned, that (server) instance of rsyslog is being started >> with -t514. > > You loaded the module but I don't see anything explicitly starting > inputTCP (not sure if this is required or not). In my config I have: > > $ModLoad imtcp.so # load module > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 Yup, those were right on the money, and of course I do see the info at `netstat -alnp | grep 514' you talked about... its listening now, and with a restart at both ends... logs info is now comming across. Many thanks... you input proved to be right on the money. Oh, do you know if there are many things that can only log by udp? From reader at newsguy.com Tue Feb 23 01:33:05 2010 From: reader at newsguy.com (Harry Putnam) Date: Mon, 22 Feb 2010 18:33:05 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> Message-ID: <87fx4skczi.fsf@newsguy.com> Harry Putnam writes: > Thanks for your patience Rainer... ok I get it now. But see comment > at the end. After saying that about something at the end.... I forgot to include it. I was just wondering why the example config that comes with the rsyslog distro leaves out such important stuff as what it takes to get the server listening for log messages. It has a commented section that claims to be about: `Receiving Messages from Remote Hosts' # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 But it doesn't bother to include the things that make tcp work, like the all important lines posted by Aaron N. $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) $InputTCPServerRun 514 # start up TCP listener at port 514 Again thank you both for your input and for website too Rainer. From ktm at rice.edu Tue Feb 23 03:03:19 2010 From: ktm at rice.edu (Kenneth Marshall) Date: Mon, 22 Feb 2010 20:03:19 -0600 Subject: [rsyslog] [basic] No remote logs getting thru In-Reply-To: <87fx4skczi.fsf@newsguy.com> References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> Message-ID: <20100223020319.GA21475@it.is.rice.edu> On Mon, Feb 22, 2010 at 06:33:05PM -0600, Harry Putnam wrote: > Harry Putnam writes: > > > Thanks for your patience Rainer... ok I get it now. But see comment > > at the end. > > After saying that about something at the end.... I forgot to include > it. > > I was just wondering why the example config that comes with the > rsyslog distro leaves out such important stuff as what it takes to get > the server listening for log messages. > > It has a commented section that claims to be about: > `Receiving Messages from Remote Hosts' > > # ######### Receiving Messages from Remote Hosts ########## > # TCP Syslog Server: > # provides TCP syslog reception and GSS-API (if compiled to support it) > #$ModLoad imtcp.so # load module > # Note: as of now, you need to use the -t command line option to > # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) > # This will change in later v3 releases. > > # UDP Syslog Server: > #$ModLoad imudp.so # provides UDP syslog reception > #$UDPServerRun 514 # start a UDP syslog server at standard port 514 > > But it doesn't bother to include the things that make tcp work, like > the all important lines posted by Aaron N. > > $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) > $InputTCPServerRun 514 # start up TCP listener at port 514 > > Again thank you both for your input and for website too Rainer. > Maybe the sample configuration files are for an earlier release of rsyslog. Cheers, Ken From reader at newsguy.com Tue Feb 23 16:02:24 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:02:24 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <87r5ochu67.fsf@newsguy.com> Kenneth Marshall writes: [...] Harry wrote: >> I was just wondering why the example config that comes with the >> rsyslog distro leaves out such important stuff as what it takes to get >> the server listening for log messages. [...] Harry wrote: >> But it doesn't bother to include the things that make tcp work, like >> the all important lines posted by Aaron N. >> >> $InputTCPMaxSessions 1000 # Maximum TCP sessions (default 200) >> $InputTCPServerRun 514 # start up TCP listener at port 514 [...] Kenneth wrote: > Maybe the sample configuration files are for an earlier release of > rsyslog. You mean like maybe the version advanced and nobody bothered to update the example conf? Yeah that might be what happened but even so, it seems you'd want the example conf shipped with a version to match the version. What I'm looking at was installed with version 3.22.1 From reader at newsguy.com Tue Feb 23 16:11:18 2010 From: reader at newsguy.com (Harry Putnam) Date: Tue, 23 Feb 2010 09:11:18 -0600 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <87mxz0htrd.fsf@newsguy.com> Just getting started with rsyslog and finding the documentation pretty confusing. Probably not the fault of the documentation but more my lack of knowledge. What I'd like to do is have rsyslog create a directory for incoming foreign logs for each host that sends syslog info. >From there, its ok if it all goes in a single log, inside the directory for now, and I may work out some splitting scheme later. So far, with help from posters here I've managed to get the logs comming in from remote machines and logging to a file named after the host. ... I suspect I should be able to figure it out from the pages here (line wrapped for mail): http://wiki.rsyslog.com/index.php \ /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory $template DYNmessages,"/var/log/%HOSTNAME%/messages" But I'd want the directory named for the host and include all logs from that host, not just the `messages' log. Not quite getting how to edit that line to make it happen. From anichols at trumped.org Tue Feb 23 16:49:48 2010 From: anichols at trumped.org (Aaron Nichols) Date: Tue, 23 Feb 2010 08:49:48 -0700 Subject: [rsyslog] How to make rsyslog create dir per host In-Reply-To: <87mxz0htrd.fsf@newsguy.com> References: <87mxz0htrd.fsf@newsguy.com> Message-ID: On Tue, Feb 23, 2010 at 8:11 AM, Harry Putnam wrote: > What I'd like to do is have rsyslog create a directory for incoming > foreign logs for each host that sends syslog info. [...] > I suspect I should be able to figure it out from the pages here (line > wrapped for mail): > ?http://wiki.rsyslog.com/index.php \ > ? /Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > ?$template DYNmessages,"/var/log/%HOSTNAME%/messages" > > But I'd want the directory named for the host and include all logs from > that host, not just the `messages' log. You want to get familiar with templates and actions - there are two pieces to this. Rather than tell you exactly how to solve your problem you should read this: http://www.rsyslog.com/doc-rsyslog_conf_templates.html and this: http://www.rsyslog.com/doc-rsyslog_conf_actions.html And bookmark this: http://www.rsyslog.com/doc-property_replacer.html You are correct, the documentation isn't perfect. That said, the application works pretty good and solves a lot of problems that aren't otherwise easily solved, so people like to use it. I would prefer the developers put their few precious resources into making the application solid vs. creating explicit documentation. You can always come here for help. But for that reason I recommend you read the above thoroughly and also the other sections of the configuration documentation. You'll spend some time putting together a configuration that works for you but once you understand how the configs work I think you'll find it relatively easy to figure out how to do specific things. If you think you understand how the selectors, actions and templates work together and still have problems or questions, let us know. Aaron From rgerhards at hq.adiscon.com Tue Feb 23 18:20:51 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:20:51 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103930@GRFEXC.intern.adiscon.com> Hi Aaron, thanks for all your help - MUCH appreciated. One thing I'd like to mention: > You are correct, the documentation isn't perfect. That said, the > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree to your analysis, but the doc problem has even more facets. I have blogged about it (once again), so that I have the information right at hand and available for easy sharing. I'd really appreciate if you could take out a few minutes and read this: http://blog.gerhards.net/2010/02/why-is-writing-good-user-doc-such.html In short: it would be very useful if "the average user" would contribute somewhat to the doc (or wiki!), because other "average users" understand that probably much better than what I write... Rainer PS: as a motivation: the blog post also contains links to all the good documentation sources ;) From rgerhards at hq.adiscon.com Tue Feb 23 18:49:57 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 23 Feb 2010 18:49:57 +0100 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com><87wry5kzkx.fsf@newsguy.com><9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com><87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> > Maybe the sample configuration files are for an earlier release of > rsyslog. Harry, please do me a favor and complain with your distribution - they probably have simply overlooked this (can easily happen). I have just checked, the sample rsyslog.conf I provide includes commented-out remote reception commands (for a reason ;)): http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Also, I have added some text to the rsyslog cookbook for remote reception. I would appreciate if you could have a look and let me know if it is useful. If not, some text suggestions would be most welcome: http://cookbook.rsyslog.com/node16.html Rainer > > Cheers, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rory at ooma.com Wed Feb 24 00:37:07 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 15:37:07 -0800 Subject: [rsyslog] amazon s3 as logstore Message-ID: <4B8466A3.8000304@ooma.com> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? From rory at ooma.com Wed Feb 24 01:05:20 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:05:20 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8466A3.8000304@ooma.com> References: <4B8466A3.8000304@ooma.com> Message-ID: <4B846D40.3070505@ooma.com> On 2/23/10 3:37 PM, Rory Toma wrote: > Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > Or, generically, how would I take a group of rsyslog servers accepting logs and have them log to central storage, without using a single central logger, and preferably, not using SQL injection. From david at lang.hm Wed Feb 24 01:10:48 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:10:48 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > Or, generically, how would I take a group of rsyslog servers accepting > logs and have them log to central storage, without using a single > central logger, and preferably, not using SQL injection. so you want the logs send to a central place but not to a central logger. what's the difference? as a security person SQL injection is a exploit of badly written code, not something anyone would ever want to use, so I don't understand what you are asking here. David Lang From pgollucci at p6m7g8.com Wed Feb 24 01:11:16 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Wed, 24 Feb 2010 00:11:16 +0000 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B846D40.3070505@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B846EA4.1090800@p6m7g8.com> On 02/24/10 00:05, Rory Toma wrote: > On 2/23/10 3:37 PM, Rory Toma wrote: >> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? The easiest way AFAIK, is to have your own central rsyslog server with a db (mysql). Then use tarnsap (http://tarsnap.com) to copy either log files or db dumps to it. It uses AES256 bit encryption among other things so it can even story CC related data if needed. FWIW, tarnsap uses s3/ec2 for storage et al. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From rory at ooma.com Wed Feb 24 01:20:22 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:20:22 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> Message-ID: <4B8470C6.507@ooma.com> I would like several front end servers writing to a single file-system, but I would expect that, out-of-the-box I would get out of order files and would have locking issues. On 2/23/10 4:10 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > > >> On 2/23/10 3:37 PM, Rory Toma wrote: >> >>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> Or, generically, how would I take a group of rsyslog servers accepting >> logs and have them log to central storage, without using a single >> central logger, and preferably, not using SQL injection. >> > so you want the logs send to a central place but not to a central logger. > > what's the difference? > > as a security person SQL injection is a exploit of badly written code, not > something anyone would ever want to use, so I don't understand what you > are asking here. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Feb 24 01:34:05 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:34:05 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: <4B8470C6.507@ooma.com> References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, Rory Toma wrote: > I would like several front end servers writing to a single file-system, but I > would expect that, out-of-the-box I would get out of order files and would > have locking issues. Yes, if you use a standard filesystem you can only have one system writing to it at a time. If you want multiple systems to be able to write to a single filesystem you need to select a cluster filesystem or a network filesystem such a filesystem will allow multiple machines to write to different files on the shared filesystem at once, but they cannot write to the same file from different machines. all of the options have very different performance, reliability, and cost trade-offs. You will need to spend a lot of time thinking though your particular requirements and the capabilities of the different systems to pick one. I'm not sure exactly what you mean by 'out of order files', but if you are thinking that everything would end up in the same file and be in a different order than they were 'really' generated, then you are correct. The problem is that packets going over the network may not arrive in the same order that they were sent. Thus it is impossible for any system to guarantee the log entries reamin in the same order. the best that anything can do it to record very accurate timestamps and re-sort the logs after they have arrived. Very few people go down this road for logs. Almost everyone relays the logs to a single server and lets that server write to a filesystem (and if they need to eliminate this as a single point of failure, they make another system able to take over this filesystem if the first one goes down) David Lang > On 2/23/10 4:10 PM, david at lang.hm wrote: >> On Tue, 23 Feb 2010, Rory Toma wrote: >> >> >>> On 2/23/10 3:37 PM, Rory Toma wrote: >>> >>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> Or, generically, how would I take a group of rsyslog servers accepting >>> logs and have them log to central storage, without using a single >>> central logger, and preferably, not using SQL injection. >>> >> so you want the logs send to a central place but not to a central logger. >> >> what's the difference? >> >> as a security person SQL injection is a exploit of badly written code, not >> something anyone would ever want to use, so I don't understand what you >> are asking here. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > From rory at ooma.com Wed Feb 24 01:38:49 2010 From: rory at ooma.com (Rory Toma) Date: Tue, 23 Feb 2010 16:38:49 -0800 Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: <4B847519.8060702@ooma.com> On 2/23/10 4:34 PM, david at lang.hm wrote: > On Tue, 23 Feb 2010, Rory Toma wrote: > >> I would like several front end servers writing to a single >> file-system, but I would expect that, out-of-the-box I would get out >> of order files and would have locking issues. > > Yes, if you use a standard filesystem you can only have one system > writing to it at a time. > > If you want multiple systems to be able to write to a single > filesystem you need to select a cluster filesystem or a network > filesystem > > such a filesystem will allow multiple machines to write to different > files on the shared filesystem at once, but they cannot write to the > same file from different machines. > > all of the options have very different performance, reliability, and > cost trade-offs. You will need to spend a lot of time thinking though > your particular requirements and the capabilities of the different > systems to pick one. > > I'm not sure exactly what you mean by 'out of order files', but if you > are thinking that everything would end up in the same file and be in a > different order than they were 'really' generated, then you are > correct. The problem is that packets going over the network may not > arrive in the same order that they were sent. Thus it is impossible > for any system to guarantee the log entries reamin in the same order. > the best that anything can do it to record very accurate timestamps > and re-sort the logs after they have arrived. > > Very few people go down this road for logs. Almost everyone relays the > logs to a single server and lets that server write to a filesystem > (and if they need to eliminate this as a single point of failure, they > make another system able to take over this filesystem if the first one > goes down) > Yeah, I didn't think it was practical, but if someone had done something unorthodox, and it worked, I thought it might be nice to hear about it. From david at lang.hm Wed Feb 24 01:57:28 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 23 Feb 2010 16:57:28 -0800 (PST) Subject: [rsyslog] amazon s3 as logstore In-Reply-To: References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com> <4B8470C6.507@ooma.com> Message-ID: On Tue, 23 Feb 2010, david at lang.hm wrote: > I'm not sure exactly what you mean by 'out of order files', but if you are > thinking that everything would end up in the same file and be in a different > order than they were 'really' generated, then you are correct. The problem is > that packets going over the network may not arrive in the same order that > they were sent. Thus it is impossible for any system to guarantee the log > entries reamin in the same order. the best that anything can do it to record > very accurate timestamps and re-sort the logs after they have arrived. By the way, if keeping the logs in order is critical to you, you need to be aware that rsyslog may not process the logs in order under all conditions. For example: if you have multiple worker threads, they each grab some logs off the queue and attempt to deliver/write them. This can cause the logs to be delivered/written in a different order than rsyslog received them. Since there are so many other things that can re-order logs slightly, this is not viewed as a significant problem. David Lang > >> On 2/23/10 4:10 PM, david at lang.hm wrote: >>> On Tue, 23 Feb 2010, Rory Toma wrote: >>> >>> >>>> On 2/23/10 3:37 PM, Rory Toma wrote: >>>> >>>>> Has anyone used amazon s3 (or ec2) for collecting/storing logs yet? >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> Or, generically, how would I take a group of rsyslog servers accepting >>>> logs and have them log to central storage, without using a single >>>> central logger, and preferably, not using SQL injection. >>>> >>> so you want the logs send to a central place but not to a central logger. >>> >>> what's the difference? >>> >>> as a security person SQL injection is a exploit of badly written code, not >>> something anyone would ever want to use, so I don't understand what you >>> are asking here. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> > From rgerhards at hq.adiscon.com Wed Feb 24 07:28:15 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 07:28:15 +0100 Subject: [rsyslog] amazon s3 as logstore References: <4B8466A3.8000304@ooma.com> <4B846D40.3070505@ooma.com><4B8470C6.507@ooma.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103933@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, February 24, 2010 1:57 AM > To: Rory Toma > Cc: rsyslog-users > Subject: Re: [rsyslog] amazon s3 as logstore > > On Tue, 23 Feb 2010, david at lang.hm wrote: > > By the way, if keeping the logs in order is critical to you, you need > to > be aware that rsyslog may not process the logs in order under all > conditions. > > For example: if you have multiple worker threads, they each grab some > logs > off the queue and attempt to deliver/write them. This can cause the > logs > to be delivered/written in a different order than rsyslog received > them. > > Since there are so many other things that can re-order logs slightly, > this > is not viewed as a significant problem. > David is very right here. If you rely on the sequence as recorded in the log file, you have a big problem. That problem is not caused by rsyslog's threading, but rather by the fact that syslog is usually used inside a remote system. In such a system, there is no guaranteed sequence: UDP get's reordered, TCP messages sit for different timespans in interim routers, syslog relay chains may delay messages even for hours (if a relay is down). I thought I had blogged about this, but can't find the post myself right now ;) But in short: you need to be very careful about your sequence expectation. Timestamps are much better, but of course not perfect. RFC5424 provides the plumbing for a better solution, but none is standardized yet and I know of no such implementation. I have Lamport clocks on my mind as the "ultimate" solution (and maybe I get time to implement that in a distant future...). Rainer From tbergfeld at hq.adiscon.com Wed Feb 24 14:57:16 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 24 Feb 2010 14:57:16 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld Hi all, We have just release rsyslog 4.6.0. Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new features of 4.5.x to the stable releases. This includes bug fixes as well as many new features. Out of the many, I would like to highlight the ability to have output files automatically zipped, multiple TCP listeners and overall performance improvement. For all details, please check the 4.5.x change log entries. This release has undergone rigorous testing in some very demanding environments. We held release until all tests went out well. We are happy to have reached this stage now and sincerely think that the version is ready for prime time. Thanks to everyone who contributed test time and bug reports! We recommend that all v4-stable users have a look at 4.6.0 and consider updating to it. Please note that 4.6.0 also does have some patches which 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available via support contracts). ChangeLog: http://www.rsyslog.com/Article443.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-195.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From reader at newsguy.com Wed Feb 24 21:03:57 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:03:57 -0600 Subject: [rsyslog] [basic] No remote logs getting thru References: <87iq9qmllv.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103903@GRFEXC.intern.adiscon.com> <87wry5kzkx.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA710391E@GRFEXC.intern.adiscon.com> <87k4u5j94p.fsf@newsguy.com> <87fx4skczi.fsf@newsguy.com> <20100223020319.GA21475@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA7103931@GRFEXC.intern.adiscon.com> Message-ID: <871vgaieoi.fsf@newsguy.com> "Rainer Gerhards" writes: >> Maybe the sample configuration files are for an earlier release of >> rsyslog. > > Harry, > > please do me a favor and complain with your distribution - they probably have > simply overlooked this (can easily happen). I have just checked, the sample > rsyslog.conf I provide includes commented-out remote reception commands (for > a reason ;)): > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=rsyslog.conf;h=47fc4402a174367 > e332a3815b3eb0b627db60b3c;hb=HEAD#l52 Thanks for showing that... and for a moment there I was afraid I was the dope that had somehow deleted the important missing lines. So just to make sure I checked my distros' (gentoo) rsyslog.conf shipped for installation by the distro maintainer and sure enough they have: # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module # Note: as of now, you need to use the -t command line option to # enable TCP reception (e.g. -t514 to run a server at port 514/tcp) # This will change in later v3 releases. # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 When the original shipped with rsyslog pkg has the lines below as you've shown. # ######### Receiving Messages from Remote Hosts ########## # TCP Syslog Server: # provides TCP syslog reception and GSS-API (if compiled to support it) #$ModLoad imtcp.so # load module #$InputTCPServerRun 514 # start up TCP listener at port 514 # UDP Syslog Server: #$ModLoad imudp.so # provides UDP syslog reception #$UDPServerRun 514 # start a UDP syslog server at standard port 514 Apparently my distro has removed that one all important line. `#$InputTCPServerRun 514 # start up TCP listener at port 514' And yes, I will make sure to contact the package maintainer about it. Thanks for the heads up and I'm sure the maintainer will want to get that back in there too. > Also, I have added some text to the rsyslog cookbook for remote reception. I > would appreciate if you could have a look and let me know if it is useful. If > not, some text suggestions would be most welcome: > > http://cookbook.rsyslog.com/node16.html > Thanks... that couldn't be more clear... brief and directly on point. Thanks. From reader at newsguy.com Wed Feb 24 21:11:14 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 14:11:14 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <87mxz0htrd.fsf@newsguy.com> Message-ID: <87wry2gzrx.fsf@newsguy.com> Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. From rgerhards at hq.adiscon.com Wed Feb 24 22:09:01 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 24 Feb 2010 22:09:01 +0100 Subject: [rsyslog] How to make rsyslog create dir per host Message-ID: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> At least for me, it would be usefuly if you could describe your needs in relatively precise words. It may be obvious for you, but that is what is hard to grasp for me... ----- Urspr?ngliche Nachricht ----- Von: "Harry Putnam" An: "rsyslog at lists.adiscon.com" Gesendet: 24.02.10 21:12 Betreff: Re: [rsyslog] How to make rsyslog create dir per host Aaron Nichols writes: > You are correct, the documentation isn't perfect. That said, the I didn't say anything about the documentation not being perfect. Maybe you miss-read my intent. "Probably not the fault of the documentation but more my lack of knowledge." Note it says `not the fault of the documentation'. In other words I'm laying any problems understanding it on my own pea brain... > application works pretty good and solves a lot of problems that aren't > otherwise easily solved, so people like to use it. I would prefer the > developers put their few precious resources into making the > application solid vs. creating explicit documentation. You can always > come here for help. I agree fully with above... and quite happy to have this program to work with. Thanks for all the good info, I'm still working thru the urls and experimenting. You gave a good legup. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 01:45:35 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:45:35 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/24 Tom Bergfeld : > Hi all, > We have just released rsyslog 4.6.0. > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings all new > features of 4.5.x to the stable releases. This includes bug fixes as well as > many new features. Out of the many, I would like to highlight the ability to > have output files automatically zipped, multiple TCP listeners and overall > performance improvement. For all details, please check the 4.5.x change log > entries. > > This release has undergone rigorous testing in some very demanding > environments. We held release until all tests went out well. We are happy to > have reached this stage now and sincerely think that the version is ready for > prime time. Thanks to everyone who contributed test time and bug reports! > > We recommend that all v4-stable users have a look at 4.6.0 and consider > updating to it. Please note that 4.6.0 also does have some patches which > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still available > via support contracts). Seems, the xconsole/pipe related fixes did not make it into this release. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:48:53 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:48:53 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Michael Biebl : > > Seems, the xconsole/pipe related fixes did not make it into this release. Running rsyslogd in debug mode and opening xconsole (as root) I get 8841.463601465:b6e86b70: Message from UNIX socket: #3 8841.463676615:b6e86b70: logmsg: flags 4, from 'pluto', msg Feb 25 01:47:21 sudo: michael : TTY=pts/4 ; PWD=/home/michael/debian/rsyslog ; USER=root ; COMMAND=/usr/bin/xconsole 8841.463705390:b6e86b70: Message has legacy syslog format. 8841.463735003:b6e86b70: main Q: entry added, size now 1 entries 8841.463762660:b6e86b70: wtpAdviseMaxWorkers signals busy 8841.463795905:b6e86b70: main Q: EnqueueMsg advised worker start 8841.463825239:b6e86b70: --------imuxsock calling select, active file descriptors (max 3): 3 8841.463884465:b7687b70: main Q: entry deleted, state 0, size now 0 entries 8841.463925812:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'NetworkManager': FALSE 8841.463959895:b7687b70: Filter: check for property 'programname' (value 'sudo') contains 'wpa_supplicant': FALSE 8841.463997330:b7687b70: Called action, logging to builtin-file 8841.464033369:b7687b70: file to log to: /var/log/auth.log 8841.464057674:b7687b70: doWrite, pData->pStrm 0x8a32360, lenBuf 124 8841.464081699:b7687b70: strm 0x8a32360: file 9 flush, buflen 124 8841.464150983:b7687b70: strm 0x8a32360: file 9 write wrote 124 bytes 8841.464196520:b7687b70: Called action, logging to builtin-file 8841.464224177:b7687b70: file to log to: /dev/xconsole 8841.464247086:b7687b70: doWrite, pData->pStrm 0x8a3c248, lenBuf 124 8841.464269714:b7687b70: strm 0x8a3c248: file -1 flush, buflen 494 8841.464298210:b7687b70: Note: stream '/dev/xconsole' is a named pipe, open with O_NONBLOCK 8841.464340674:b7687b70: file '/dev/xconsole' opened as #-1 with mode 416 8841.464380065:b7687b70: strm 0x8a3c248: open error 6, file '/dev/xconsole': No such device or address 8841.464413869:b7687b70: main Q:Reg/w0: worker IDLE, waiting for work. The relevant rsyslog.conf configuration is daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Feb 25 01:50:45 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 01:50:45 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: As additional information: If i run rsyslogd first, then xconsole, I get the failure Running xconsole first, then rsyslogd, it works fine. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From reader at newsguy.com Thu Feb 25 01:58:52 2010 From: reader at newsguy.com (Harry Putnam) Date: Wed, 24 Feb 2010 18:58:52 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> Message-ID: <87fx4qgmgj.fsf@newsguy.com> "Rainer Gerhards" writes: > At least for me, it would be usefuly if you could describe your > needs in relatively precise words. It may be obvious for you, but > that is what is hard to grasp for me... Not sure what you are talking about here Rainer. I was just replying to someone who appeared to think I had made some disparaging remark about the documentation. I simply wanted to clear that up... Now if you mean for me to explain more precisely what I wanted from rsyslog, I'll do that. However, let me say first that with Aarons' pointers to URLS that I am now working my way thru and also having just watched one of your videos and the basics... I have to tell you that that caliber of the documentation is really quite high. There is in depth coverage and then very helpful outside resources such as the video and other postings of yours. I wish I had the command of a foreign language as well as you have of english. I'm a native english speaker... but am sadly nearly illiterate in my own language. The documentation really does out class the usual documentation one encounters in unix/linux tools. ------- --------- ---=--- --------- -------- What I sought to convey in my question about creating directories: Its really low level so sorry I didn't get it across very well. Once I got the clients writing across the network to the rsyslog server, and the server listening and logging the data, I thought for now it would be good to have rsyslog on the server create directories for each remote client, and write everything from each client to /var/log/%hostname%/everthing.log (as a beginning point. While I figure out how to set things up so that each incoming log from remote would be split up further under the top level directories. Kind of like the example shown at (wrapped for mail): http://wiki.rsyslog.com/index.php/ \ Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory Just one level deeper. So one level of dynamic directory is created for each client host. Later I'll probably want to spit that a bit more Once I have a more permanent plan figured out I would at least have the full output from each client host on hand. I needed to get the basics up some whet urgently in that I had one host failing silently or at least not leaving a trace I could find in its logs. It would just freeze up, where the mouse and keyboard became unresponsive and could not connect to the problem host remotely either. On mechanical reboot, it would look like the system logger froze up too but someone told me I might catch some log data that would be lost on a hard reboot if I had that host logging remotely...thru rsyslog. I doubted that would be true since it appears as though networking is froze. But I have gotten rsyslog logging remotely now and waiting for a freezeup to see if I can get any more clues as to what is causing the problem. From buraphalinuxserver at gmail.com Thu Feb 25 08:34:29 2010 From: buraphalinuxserver at gmail.com (BuraphaLinux Server) Date: Thu, 25 Feb 2010 14:34:29 +0700 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x Message-ID: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Will there be an rsyslog 4.4.3 release? I see references to changes from it in the website, but cannot find the 4.4.3 release itself, and the 4.6.0 release seems to say that there will not be any 4.x releases any more. The web page says 4.6.0 is a stable release and a beta at the same time, which is very confusing for me. http://www.rsyslog.com/Article443.phtml The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which means it is a beta, but then the text on the page also says "This is a new stable v4 version." and "Do not expect any more new developments in v4.". I'm a bit confused. Does this mean if we want bugfixes we have to run a 5.x version? From rgerhards at hq.adiscon.com Thu Feb 25 09:09:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:09:24 +0100 Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x References: <5d75f4611002242334g451013d8q7eccfd57024e27d4@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of BuraphaLinux Server > Sent: Thursday, February 25, 2010 8:34 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 4.4.3, 4.6.0, and 5.x > > Will there be an rsyslog 4.4.3 release? I see references to changes > from it in the website, but cannot find the 4.4.3 release itself, No. I initially planned to have a 4.4.3, but Monday, when I tried to prepare it, I found some issues in the automated test suite. Whatever they were, they were fixed in 4.6.0. So I skipped that release, saving me the time to find what already is fixed (the original plan was to do a last 4.4.3 release followed by 4.6.0 two days later, so I only skipped the 4.4.3 step). > and > the 4.6.0 release seems to say that there will not be any 4.x releases > any more. The web page says 4.6.0 is a stable release and a beta at > the same time, which is very confusing for me. > > http://www.rsyslog.com/Article443.phtml Thanks for pointing that out. That's a simple typo, now corrected. > > The top says "Version 4.6.0 [v4-beta] (rgerhards), 2010-02-24" which > means it is a beta, but then the text on the page also says "This is a > new stable v4 version." and "Do not expect any more new developments > in v4.". I'm a bit confused. Does this mean if we want bugfixes we > have to run a 5.x version? Bugfixes are not "new developments", at least in my point of view. But if you need some new feature, you need to run v5 if we implement it. That's the way it was always handled in rsyslog, with the exception of v4 and v5, which had two different development branches at the same time. There were good reasons for that, but it also caused a lot of work and some issues. I am very happy to be back to a single development version again. I won't split off a new second development branch again until there is *very* good reason. I hope this clarifies. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:25:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:25:18 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Hi all, on Debian, a discussion came up on the inconsistency of some config statements. The right cure would be to correct this, but that requires some time. I would appreciate your feedback on the issue. The details can be found here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 Thanks, Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:26:47 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:26:47 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710395E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:46 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/24 Tom Bergfeld : > > Hi all, > > We have just released rsyslog 4.6.0. > > > > Rsyslog 4.6.0 is the next revision of the v4-stable branch. It brings > all new > > features of 4.5.x to the stable releases. This includes bug fixes as > well as > > many new features. Out of the many, I would like to highlight the > ability to > > have output files automatically zipped, multiple TCP listeners and > overall > > performance improvement. For all details, please check the 4.5.x > change log > > entries. > > > > This release has undergone rigorous testing in some very demanding > > environments. We held release until all tests went out well. We are > happy to > > have reached this stage now and sincerely think that the version is > ready for > > prime time. Thanks to everyone who contributed test time and bug > reports! > > > > We recommend that all v4-stable users have a look at 4.6.0 and > consider > > updating to it. Please note that 4.6.0 also does have some patches > which > > 4.4.2 does not have. Support for 4.4.2 is concluded now (but still > available > > via support contracts). > > Seems, the xconsole/pipe related fixes did not make it into this > release. mhhh... The delta between 4.5.8 and 4.6.0 is very slim. So I wonder what might have broken that fix... Will try to check. Rainer From rgerhards at hq.adiscon.com Thu Feb 25 09:44:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:44:10 +0100 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Harry Putnam > Sent: Thursday, February 25, 2010 1:59 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] How to make rsyslog create dir per host > > "Rainer Gerhards" writes: > > > At least for me, it would be usefuly if you could describe your > > needs in relatively precise words. It may be obvious for you, but > > that is what is hard to grasp for me... > sorry, that was too brief. I was on a PDA and it was late ;) My intent is to see if I can write up a cookbook entry for what you need, but I have no clear understanding what's your intent. Now let me read through the rest of your mail and see where I come to... ;) > Not sure what you are talking about here Rainer. I was just replying > to someone who appeared to think I had made some disparaging remark > about the documentation. > > I simply wanted to clear that up... > > > Now if you mean for me to explain more precisely what I wanted from > rsyslog, I'll do that. > > However, let me say first that with Aarons' pointers to URLS that I am > now working my way thru and also having just watched one of your > videos and the basics... I have to tell you that that caliber of the > documentation is really quite high. > > There is in depth coverage and then very helpful outside resources such > as the video and other postings of yours. > > I wish I had the command of a foreign language as well as you have of > english. I'm a native english speaker... but am sadly nearly > illiterate in my own language. > > The documentation really does out class the usual documentation one > encounters in unix/linux tools. Thanks for the kind words, but you had a valid point in your initial remarks: the rsyslog doc may be good for someone who intends to understand the whole system and be an expert at that. However, most often people look for a solution to a specific problem. They don't want to understand how everything works together, just solve that immediate need quickly. I think this is a valid need, I have it myself with other softwares often enough ;) The rsyslog doc completely fails to address this use case. I hope the cookbook will somewhat improve that situation. I am not sure if you saw my longer blogpost, it has the details ;) > > ------- --------- ---=--- --------- -------- > What I sought to convey in my question about creating directories: > > Its really low level so sorry I didn't get it across very well. > > Once I got the clients writing across the network to the rsyslog > server, and the server listening and logging the data, I thought for > now it would be good to have rsyslog on the server create directories > for each remote client, and write everything from each client to > /var/log/%hostname%/everthing.log (as a beginning point. > > While I figure out how to set things up so that each incoming log > from remote would be split up further under the top level directories. > > Kind of like the example shown at (wrapped for mail): > > http://wiki.rsyslog.com/index.php/ \ > Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory > > Just one level deeper. So one level of dynamic directory is created > for each client host. Later I'll probably want to spit that a bit more ... and here is where some information is missing ;) *How* do you want to split these logs? I am not sure about this, and so I have no clear idea of how a configuration could look like. As there are some limits on the way filters can be combined, this is a very important point. As a side-note: I do not know your log volume, but if it is not overwhelmingly large to prevent storing logs to a database, I'd probably do that and analyze and search it with free GPLed Adiscon LogAnalyzer: http://loganalyzer.adiscon.com These days, we are preparing a new v3 release which also supports reporting (and you can schedule that via cron, too). (Adiscon LogAnalyzer also works with plain text files, but this is only meant for low-volume, few-system consolidated logs - otherwise the performance will be very bad). HTH Rainer > > Once I have a more permanent plan figured out I would at least have > the full output from each client host on hand. > > I needed to get the basics up some whet urgently in that I had one host > failing silently or at least not leaving a trace I could find in its > logs. It would just freeze up, where the mouse and keyboard became > unresponsive and could not connect to the problem host remotely > either. > > On mechanical reboot, it would look like the system logger froze up > too but someone told me I might catch some log data that would be lost > on a hard reboot if I had that host logging remotely...thru rsyslog. > > I doubted that would be true since it appears as though networking is > froze. But I have gotten rsyslog logging remotely now and waiting for > a freezeup to see if I can get any more clues as to what is causing > the problem. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Feb 25 09:51:31 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 09:51:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, February 25, 2010 1:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. ah! I think I now understand the issue. This smells like a new one. Did this really work with 4.5.8? Rainer > > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Thu Feb 25 13:22:02 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 13:22:02 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Thursday, February 25, 2010 1:51 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. > > ah! I think I now understand the issue. This smells like a new one. Did this > really work with 4.5.8? I haven't tested 4.5.8. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 13:56:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 13:56:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> > -----Original Message----- > 2010/2/25 Rainer Gerhards : > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl > >> Sent: Thursday, February 25, 2010 1:51 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > >> > >> As additional information: > >> > >> If i run > >> rsyslogd first, then xconsole, I get the failure > >> Running xconsole first, then rsyslogd, it works fine. > > > > ah! I think I now understand the issue. This smells like a new one. > Did this > > really work with 4.5.8? > > I haven't tested 4.5.8. ah, ok, so which was the latest version you tested against? Rainer From mbiebl at gmail.com Thu Feb 25 14:01:19 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 14:01:19 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> -----Original Message----- >> 2010/2/25 Rainer Gerhards : >> >> -----Original Message----- >> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> >> Sent: Thursday, February 25, 2010 1:51 AM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released >> >> >> >> As additional information: >> >> >> >> If i run >> >> rsyslogd first, then xconsole, I get the failure >> >> Running xconsole first, then rsyslogd, it works fine. >> > >> > ah! I think I now understand the issue. This smells like a new one. >> Did this >> > really work with 4.5.8? >> >> I haven't tested 4.5.8. > > ah, ok, so which was the latest version you tested against? > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Feb 25 14:06:11 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 14:06:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103961@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103965@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103966@GRFEXC.intern.adiscon.com> > >> >> If i run > >> >> rsyslogd first, then xconsole, I get the failure > >> >> Running xconsole first, then rsyslogd, it works fine. > >> > > >> > ah! I think I now understand the issue. This smells like a new > one. > >> Did this > >> > really work with 4.5.8? > >> > >> I haven't tested 4.5.8. > > > > ah, ok, so which was the latest version you tested against? > > > > I did run 4.4.2 successfully. I haven't tested any 4.5.x snapshots. OK, misunderstanding on my side. I thought you had verified that the "xconsole" patch worked. I will try to dig again into the code, but it's some while since I crafted the patch, so it may take a while. Maybe I manage to do that today, if not it'll be monday - I have meetings tomorrow :( Rainer From rgerhards at hq.adiscon.com Thu Feb 25 15:27:44 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 25 Feb 2010 15:27:44 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> > As additional information: > > If i run > rsyslogd first, then xconsole, I get the failure > Running xconsole first, then rsyslogd, it works fine. > I just verified the case. I get the failure if xconsole is not active, what is perfectly OK IMHO (because otherwise it would block on the pipe). As soon as I start xconsole, everything works fine and messages are displayed in xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it everything works again. So I do not see an issue. I tested on Debian lenny with all current updates applied. Rainer > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From reader at newsguy.com Thu Feb 25 20:55:19 2010 From: reader at newsguy.com (Harry Putnam) Date: Thu, 25 Feb 2010 13:55:19 -0600 Subject: [rsyslog] How to make rsyslog create dir per host References: <001101cab595$ae2ca5a3$100013ac@intern.adiscon.com> <87fx4qgmgj.fsf@newsguy.com> <9B6E2A8877C38245BFB15CC491A11DA7103960@GRFEXC.intern.adiscon.com> Message-ID: <87y6ihf5ug.fsf@newsguy.com> "Rainer Gerhards" writes: >> Just one level deeper. So one level of dynamic directory is created >> for each client host. Later I'll probably want to spit that a bit more > > ... and here is where some information is missing ;) *How* do you want to > split these logs? I am not sure about this, and so I have no clear idea of > how a configuration could look like. As there are some limits on the way > filters can be combined, this is a very important point. He he... yes it is.. but its missing because I don't have a solid plan for what I want yet. I really was only asking about the simple part. I now know how to do that, and when you see what I was after (for now) you will be disappointed because its probably to obvious and simple to need to be in your cook book. It just takes me a while to see how to get started. On the client machines: *.info;mail.none @@192.168.0.26:514 (192.168.0.26 is `logsrv' mentioned below) On the server: This template and action is all I was after (for now). (aside: DDF = Dynamic directory and filename) $template DDF,"/var/log/%hostname%/%hostname%.log" if \ $source != 'logsrv' \ then -?DDF I hope that is at least close to a correct formulation. I just used the actual host names because I wan't sure which property provided it .. just the simple alphabetic hostname with no domain. On thing I didn't see explained in what I've read so far is what role the question mark in -?DDF plays. I used it because I saw it used that way in the examples and so far the formulation seems to be working...(I actually haven't inspected the logs closely yet.... but a tail -f of the resulting log files would seem to indicate its working. At least each log appears to be only from a specific client, and to include everything incoming from that client.) From mbiebl at gmail.com Thu Feb 25 21:44:31 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 25 Feb 2010 21:44:31 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/25 Rainer Gerhards : >> As additional information: >> >> If i run >> rsyslogd first, then xconsole, I get the failure >> Running xconsole first, then rsyslogd, it works fine. >> > > I just verified the case. I get the failure if xconsole is not active, what > is perfectly OK IMHO (because otherwise it would block on the pipe). As soon > as I start xconsole, everything works fine and messages are displayed in > xconsole. When I stop xconsole I get "broken pipe" errors and if I restart it > everything works again. > > So I do not see an issue. I tested on Debian lenny with all current updates > applied. hm, it's perfectly reproducible here. starting rsyslogd, then xconsole, will cause xconsole to hang and no message is written to the xconsole window. 4.4.2 did not behave that way. See also the debug log I sent previously Is there anything else I can do to help you debug this? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Fri Feb 26 01:09:16 2010 From: lanas at securenet.net (lanas) Date: Thu, 25 Feb 2010 19:09:16 -0500 Subject: [rsyslog] Sending time in remote syslog Message-ID: <20100225190916.0a83b31d@mistral.stie> Hello ! Is it possible with rsyslog to somehow send the time of the local event as part of the log message when sending to a remote server ? Here's a use case. I have tried with syslogd. I enable syslogd on a unit to send log messages to a remote host running rsyslog (it so happens that it runs rsyslog because it's a Fedora host). Then I change the time on the unit by an offset of two hours. When the unit (syslogd) sends the log message, it is received by rsyslogd although the time stamp in the log msg shows the time on the host, not the time when the event happened on the unit itself. If I replace syslogd by rsyslog on the unit, will it be able to somehow send the local time of the unit as part of the log message ? Thanks, Al From david at lang.hm Fri Feb 26 08:46:21 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 25 Feb 2010 23:46:21 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100225190916.0a83b31d@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> Message-ID: On Thu, 25 Feb 2010, lanas wrote: > Hello ! > > Is it possible with rsyslog to somehow send the time of the local > event as part of the log message when sending to a remote server ? > > Here's a use case. I have tried with syslogd. I enable syslogd on a > unit to send log messages to a remote host running rsyslog (it so > happens that it runs rsyslog because it's a Fedora host). Then I > change the time on the unit by an offset of two hours. When the unit > (syslogd) sends the log message, it is received by rsyslogd although > the time stamp in the log msg shows the time on the host, not the time > when the event happened on the unit itself. > > If I replace syslogd by rsyslog on the unit, will it be able to > somehow send the local time of the unit as part of the log message ? each event has two timestamps, one is the timestamp in the message that was sent, the other is the time the message was received. I believe that the default template uses the recieved timestamp, but you can change it to timereportedto get the timestamp put on the message by the sender. David Lang From lanas at securenet.net Fri Feb 26 11:46:05 2010 From: lanas at securenet.net (lanas) Date: Fri, 26 Feb 2010 05:46:05 -0500 Subject: [rsyslog] Sending time in remote syslog In-Reply-To: References: <20100225190916.0a83b31d@mistral.stie> Message-ID: <20100226054605.3469f03a@mistral.stie> On Thu, 25 Feb 2010 23:46:21 -0800 (PST), david at lang.hm wrote : > On Thu, 25 Feb 2010, lanas wrote: > > Is it possible with rsyslog to somehow send the time of the local > > event as part of the log message when sending to a remote server ? > > each event has two timestamps, one is the timestamp in the message > that was sent, the other is the time the message was received. I > believe that the default template uses the recieved timestamp, but > you can change it to timereportedto get the timestamp put on the > message by the sender. Thanks, I will try it. Basically, this means that (if there's a syslog message standard ...) that syslog messages contains both timestamps, be them generated by syslogd or rsyslog, isn't it ? And it's at the receiving side that the proper display option should be chosen (in this case, does rsyslog have an advantage over others by providing more display options ?). Regards, Al From rgerhards at hq.adiscon.com Fri Feb 26 12:32:10 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:32:10 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> > > I just verified the case. I get the failure if xconsole is not > active, what > > is perfectly OK IMHO (because otherwise it would block on the pipe). > As soon > > as I start xconsole, everything works fine and messages are displayed > in > > xconsole. When I stop xconsole I get "broken pipe" errors and if I > restart it > > everything works again. > > > > So I do not see an issue. I tested on Debian lenny with all current > updates > > applied. > > hm, it's perfectly reproducible here. also on Debian lenny? Anything else I could do to set up my environment so that the problem occurs? > starting rsyslogd, then xconsole, will cause xconsole to hang and no > message is written to the xconsole window. > 4.4.2 did not behave that way. > > See also the debug log I sent previously The log tells me that it has a problem to write to the pipe, and thus ignores it. This also happens at my system, but only if xconsole is not running. So the debug log does no help very much this time. > Is there anything else I can do to help you debug this? When I get the platform information from you, I will see if I can record a session in my lab. Maybe you see that I do things differently from you. One thing I notice is that xconsole seems to be unable to access /dev/xconsole if I start it as non-root, thus resulting in no listener on the pipe and thus the error appears in rsyslog. However, this is not rsyslog-related, but I guess these permissions are set by xconsole. After all, there should be no change to 4.4.2 (and I don't see any difference in that regard to 4.4.2 on my system). Rainer > > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Fri Feb 26 12:55:11 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 12:55:11 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> > I just verified the case. I get the failure if xconsole is not >> active, what >> > is perfectly OK IMHO (because otherwise it would block on the pipe). >> As soon >> > as I start xconsole, everything works fine and messages are displayed >> in >> > xconsole. When I stop xconsole I get "broken pipe" errors and if I >> restart it >> > everything works again. >> > >> > So I do not see an issue. I tested on Debian lenny with all current >> updates >> > applied. >> >> hm, it's perfectly reproducible here. > > also on Debian lenny? Anything else I could do to set up my environment so > that the problem occurs? I'm can reproduce the problem, both on Debian lenny and Debian unstable (i386) >> starting rsyslogd, then xconsole, will cause xconsole to hang and no >> message is written to the xconsole window. >> 4.4.2 did not behave that way. >> >> See also the debug log I sent previously > > The log tells me that it has a problem to write to the pipe, and thus ignores > it. This also happens at my system, but only if xconsole is not running. So > the debug log does no help very much this time. The problem is, that xconsole is running at this point! To reiterate: I start rsyslogd then xconsole then try to log a message The result is the debug log and no message showing up in xconsole. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 12:56:49 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 12:56:49 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> > The problem is, that xconsole is running at this point! > To reiterate: > I start rsyslogd > then xconsole > then try to log a message > The result is the debug log and no message showing up in xconsole. that's what I am doing as well. I will try to record the session. Rainer From rgerhards at hq.adiscon.com Fri Feb 26 13:20:17 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 13:20:17 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103973@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, February 26, 2010 12:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > > The problem is, that xconsole is running at this point! > > To reiterate: > > I start rsyslogd > > then xconsole > > then try to log a message > > The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Please find a video of my session at http://www.rsyslog.com/download/rsyslog-xconsole.avi So I don't see the problem. Or did you mean something different? Rainer From mbiebl at gmail.com Fri Feb 26 13:43:23 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 26 Feb 2010 13:43:23 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Rainer Gerhards : >> The problem is, that xconsole is running at this point! >> To reiterate: >> I start rsyslogd >> then xconsole >> then try to log a message >> The result is the debug log and no message showing up in xconsole. > > that's what I am doing as well. I will try to record the session. Wait a sec. With 4.4.2 (or any earlier version) I tried this: start rsyslog start xconsole xconsole shows the last two log messages immediately With 4.6.0 the xconsole window is blank (which I guess made me believe it blocked) With 4.6.0, I need to trigger a new log message (e.g. using logger foo), and then xconsole will show a new message. So, xconsole does indeed show messages, but the behaviour is different to 4.4 and earlier Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Feb 26 15:19:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 15:19:55 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103975@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, February 26, 2010 1:43 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Rainer Gerhards : > >> The problem is, that xconsole is running at this point! > >> To reiterate: > >> I start rsyslogd > >> then xconsole > >> then try to log a message > >> The result is the debug log and no message showing up in xconsole. > > > > that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Mhhh... I don't think it is worth preserving "backward compatibility" here. I think these two (just two!) messages were an artifact of the way files were written to pipes that nobody cares about. The old file writer was *very* suboptimal. If I would need to retain that old "functionality", I would need to remove almost all optimizations. So it would make more sense than to craft a new output module specifically for pipes. Not sure if this is granted by that "use case". I'd appreciate comments from the community at large. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Feb 26 17:11:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 26 Feb 2010 17:11:29 +0100 Subject: [rsyslog] Log Normalization effort Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Hi all, I have blogged about my quest for log normalization. I think there is some good information on the upcoming GPLed Adiscon LogAnalyzer and future directions for rsyslog in the blog post. So I thought I share the link: http://blog.gerhards.net/2010/02/syslog-normalization.html Please note that part of the effort requires community involvement. I would be very interested to learn if you think we could win enough support to make this a useful effort. I am asking for your feedback, because it will help me streamline my priorities for future rsyslog work. Thanks, Rainer From david at lang.hm Fri Feb 26 19:34:04 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 10:34:04 -0800 (PST) Subject: [rsyslog] Sending time in remote syslog In-Reply-To: <20100226054605.3469f03a@mistral.stie> References: <20100225190916.0a83b31d@mistral.stie> <20100226054605.3469f03a@mistral.stie> Message-ID: On Fri, 26 Feb 2010, lanas wrote: > On Thu, 25 Feb 2010 23:46:21 -0800 (PST), > david at lang.hm wrote : > >> On Thu, 25 Feb 2010, lanas wrote: >>> Is it possible with rsyslog to somehow send the time of the local >>> event as part of the log message when sending to a remote server ? >> >> each event has two timestamps, one is the timestamp in the message >> that was sent, the other is the time the message was received. I >> believe that the default template uses the recieved timestamp, but >> you can change it to timereportedto get the timestamp put on the >> message by the sender. > > Thanks, I will try it. Basically, this means that (if there's a syslog > message standard ...) that syslog messages contains both timestamps, be > them generated by syslogd or rsyslog, isn't it ? And it's at the > receiving side that the proper display option should be chosen (in this > case, does rsyslog have an advantage over others by providing more > display options ?). the messages sent over the wire only have one timestamp in them, but when rsyslog receives a message it records that time as well. This gives rsyslog the option of using either timestamp when it then writes (or sends) the log. David Lang From david at lang.hm Fri Feb 26 22:32:56 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 26 Feb 2010 13:32:56 -0800 (PST) Subject: [rsyslog] Log Normalization effort In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 26 Feb 2010, Rainer Gerhards wrote: > Hi all, > > I have blogged about my quest for log normalization. I think there is some > good information on the upcoming GPLed Adiscon LogAnalyzer and future > directions for rsyslog in the blog post. So I thought I share the link: > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > Please note that part of the effort requires community involvement. I would > be very interested to learn if you think we could win enough support to make > this a useful effort. I am asking for your feedback, because it will help me > streamline my priorities for future rsyslog work. a few comments (but remember that I am usually dealing with high data rates, so my concerns are biased in that direction) log analysis is usually done in batches as opposed to in real-time. some of this is due to the difficulty in doing it in real time, but a lot of it is the processing overhead (you don't want to take so long to process an individual request that you miss the next one to arrive) at low volumes the idea of name-value pairs in the logs makes a lot of sense, but there is significantly more overhead in parsing a log with name-value pairs in arbitrary orders than there is in using a tree parsing approach to analyze known log formats in a fixed order. The message size can also increase significantly. As a result, at high traffic volumes this starts to be a bad (or at least questionable) idea. I would love to see rsyslog gain the ability to efficiently do tree-based parsing instead of regex parsing. regex parsing is easy to understand and tinker with, but very expensive to implement. it may be that having something that 'compiles' a list of regex parsers into a tree parser is the right answer for usability. I would save several hours of processing a day if I could easily (and efficiently) make rsyslog write different logs to different files (at high data rates and with a few hundred conditions based variations in the syslog tag) While there are some common events across different types of logs (logins for example) they almost always contain slightly different data in them. I also have no faith at all that anyone is going to make much effort to clean up their logs to make them nicely parseable, and if they do I see even less chance that they will end up using the same terms for the same thing. As such I see more value in trying to get samples of logs and what they mean than in trying to define a normalized version to shoehorn the logs into. It is worth doing this for some events (logins, failed logins for example), but I think it's a mistake to think that this will end up covering all, or even the majority of log messages. There's also a problem in that the ideal format for the output depends on what you are doing with the output. If I could wave a magic wand and get the result I would look for something like this the parser starts at the beginning of the message (at the priority) and can branch on priority/faclilty, timestamp, host, syslogtag, message and indicate if the message should be parsed into name-value pairs, or split based on a character (or character sequence like the perl split command allows) into individually addressable elements (defaulting to whitespace separated elements), then the format (and if needed dynafile path/file components) could be constructed from these variables. At any point in the parsing it should be possible to jump to another parser tree (so that you could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end up using the same parser for the message without having to redefine the rules for each one) With this capability, people could start writing parser 'branches' to understand a specific log type and output a 'standard' format (as such a format can start to be defined). This can be done in rsyslog today, but it is fairly difficult to define, and as I understand it, inefficient enough that it's not practical to do in real-time under heavy load. If this is fast enough, then the next step would be to add the ability to have the format/action be 'increment a counter for log type X' and a signal to rsyslog could generate a report on these counters. Although at some point it becomes better to feed the message into another opensource tool (SEC, Simple Event Correlator for example) instead of trying to do everything in rsyslog. parsing the file to know what to do with it, and be able to re-format log messages is very defiantly something that can fit into the rsyslog model of receiving, formatting, and delivering logs. Alerting on specific log entries, counting the number of times one thing shows up in logs, and this sort of thing start pushing beyond the core of rsyslog, and it may be better to feed other tools instead. David Lang From lanas at securenet.net Sat Feb 27 12:38:07 2010 From: lanas at securenet.net (lanas) Date: Sat, 27 Feb 2010 06:38:07 -0500 Subject: [rsyslog] rsyslog migration and syslogd.conf Message-ID: <20100227063807.5918fafc@mistral.stie> Hello, First, thanks to David for the reply on time stamps. I have a question about perhaps being able to replace syslogd by rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I reckon with limited possibilities) with a standard syslog config file ? That would enable a basic set of functionality as a first step toward using rsyslog's features. Al From david at lang.hm Sun Feb 28 01:02:16 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:02:16 -0800 (PST) Subject: [rsyslog] rsyslog migration and syslogd.conf In-Reply-To: <20100227063807.5918fafc@mistral.stie> References: <20100227063807.5918fafc@mistral.stie> Message-ID: On Sat, 27 Feb 2010, lanas wrote: > Hello, > > First, thanks to David for the reply on time stamps. > > I have a question about perhaps being able to replace syslogd by > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > reckon with limited possibilities) with a standard syslog config > file ? That would enable a basic set of functionality as a first step > toward using rsyslog's features. no, rsyslog will do nothing if you just feed it a syslog.conf file. rsyslog needs a few additional instructions at the top of the file to load the modules, but below that you can paste in the syslog.conf file. another thing that you can do is to make a generic rsyslog.conf file that includes the syslog.conf after the rsyslog specific headers. My rsyslog.conf (on my ubuntu 9.10 laptop) has $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog $RepeatedMsgReduction on $RepeatedMsgContainsOriginalMsg on David Lang From david at lang.hm Sun Feb 28 01:45:14 2010 From: david at lang.hm (david at lang.hm) Date: Sat, 27 Feb 2010 16:45:14 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 25 Feb 2010, Rainer Gerhards wrote: > Hi all, > > on Debian, a discussion came up on the inconsistency of some config > statements. The right cure would be to correct this, but that requires some > time. I would appreciate your feedback on the issue. The details can be found > here: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 First off, the config language really has two parts. The first part is the configuration of rsyslog, the second part is the instructions to rsyslog telling it what to do with the log messages it has received.. I think you are wildly optimistic if you think that "a year from now nobody will be using the old config language". I expect that 3-5 years from now users will still be using 3.x versions (because they are what was released in 'enterprise' releases) even for new versions there will need to be a transition path (including the ability to easily revert to an old version) I fully understand the desire to scrap the existing evelved config language and implement a new, designed version instead (without needing to support both), but I don't see that as being realistic. A new designed config language should be able to have all the existing configs map to it, and I expect that you will have to maintain a shim/transition layer for many years that will read the old configs and map it to the new config. for the first part of the configs (the configuration options for rsyslog itself, having nothing to do with processing the log messages) As such, I think making the infrastructure to log 'this config option has been replaced by that config option' could actually end up cleaning things up and making it easier to modify the config language going forward. If the logging of changed is done one option at a time it will be a mess, but if it's done as something like a mapping layer it could actually be better. I'm thinking of a table along the lines of config_option, type, destination where type could be 'boolean', 'string', 'filesystem path', 'number', or 'depriciated', 'ignore' if the type is depriciated then the destination would be another config_option (which you would then look up) if the type is 'ignore' then it would just be skipped over (possibly with a warning being logged that the config line is being ignored) this would also clean up some of the current cases where a boolean option doesn't honor on/off and true/false. For the second half of the config language (telling rsyslog what to do with the logs it has received) also has several variations in place and is showing issues. However I think that the right answer here is to end up implementing something like the parsing trees and then compile the other config language options into that tree to be consistant (and fast) under the covers, no matter which way the instructions are written (except when you have to use regex matches) David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:11:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:11:55 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Hi David, thanks for the thoughtful comments. But I seem to have incorrectly phrased my question, leaving out too many of those things that I obviously took for granted and thus not mentioned :(... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:45 AM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Thu, 25 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > on Debian, a discussion came up on the inconsistency of some config > > statements. The right cure would be to correct this, but that > requires some > > time. I would appreciate your feedback on the issue. The details can > be found > > here: > > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=571202 > > First off, the config language really has two parts. > > The first part is the configuration of rsyslog, the second part is the > instructions to rsyslog telling it what to do with the log messages it > has > received.. > > I think you are wildly optimistic if you think that "a year from now > nobody will be using the old config language". I expect that 3-5 years > from now users will still be using 3.x versions (because they are what > was > released in 'enterprise' releases) > That was probably the most important thing I did not mention: I do NOT think that removing inconsistencies in the config language is a bug fix. And if it is not a bug fix, it is a feature request. But if it is a feature request, it will go into the recent devel. And as such it will be a v5-exclusive thing. As such, what is with v3 and v4 doesn't really matter, because it would not be changed (and, yes, I totally agree that v3/3 will be around for a long time). One may argue if this feature is so essential that it should be rolled back into the old versions. However, I became very skeptic of doing so. Almost always in the past years, when I put some non-essential change into the stable releases directly, big problems were the result. I could probably open up new devel/beta branches for v3/v4 as well. But that would a) cause a lot of extra work b) people would most probably not try it out so we would end up with the same issue when it is then named stable c) the experience with two v4/v5 devel versions was a bad one as well. It is too easy to mix up where the regression is And now to the time slot: I guess, when a clean config language is there, most people would tend to use the new format. This is why I assumed that after a year the old language (in v5!) will most probably not be used much longer. To be precise, I fear that I take out time to implement this in v5, then have the new format (let's hope) 6 month later available then wonder if that would really make a difference over not doing the first step. If I had lots of time, it would probably still make sense. But unfortunately, I currently have far less time than I wish I had. Thankfully, the newer versions get more and more adopted, but that also means more bug fixes to do. So I sadly see that my available time is currently eaten up by those bug fixes and clarifying things inside the doc so that people can use the features. That is almost painful to me, because I have so many cool things on my mind, but do not find time to implement a single one of them :( > even for new versions there will need to be a transition path > (including > the ability to easily revert to an old version) > > I fully understand the desire to scrap the existing evelved config > language and implement a new, designed version instead (without needing > to > support both), but I don't see that as being realistic. > > A new designed config language should be able to have all the existing > configs map to it, and I expect that you will have to maintain a > shim/transition layer for many years that will read the old configs and > map it to the new config. The new config language doesn't mean that the old will no longer work. It is a primary design goal to support both in parallel. I always tried hard to make rsyslog as backwards-compatible as possible and it turns out that this is a major advantage, at least at the low end. I do not intend to drop that (one need also consider existing know-how: people should be able to get rsyslog immediately up and running, for simple cases, without the need to learn anything new). > > for the first part of the configs (the configuration options for > rsyslog > itself, having nothing to do with processing the log messages) > > As such, I think making the infrastructure to log 'this config option > has > been replaced by that config option' could actually end up cleaning > things > up and making it easier to modify the config language going forward. > > If the logging of changed is done one option at a time it will be a > mess, > but if it's done as something like a mapping layer it could actually be > better. > > I'm thinking of a table along the lines of > > config_option, type, destination > > where type could be 'boolean', 'string', 'filesystem path', 'number', > or > 'depriciated', 'ignore' This *is* how the system works today! But I don't have a deprecated flag and aliases, so I would need to rewrite that whole system. BTW: this system is the reason why I avoided writing a new system until absolutely necessary: it is too easy to add a new directive as all you need to do is add a new "table entry". > > if the type is depriciated then the destination would be another > config_option (which you would then look up) > > if the type is 'ignore' then it would just be skipped over (possibly > with > a warning being logged that the config line is being ignored) > > this would also clean up some of the current cases where a boolean > option > doesn't honor on/off and true/false. True/false is NOT a valid boolean option. See http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > For the second half of the config language (telling rsyslog what to do > with the logs it has received) also has several variations in place and > is > showing issues. > > However I think that the right answer here is to end up implementing > something like the parsing trees and then compile the other config > language options into that tree to be consistant (and fast) under the > covers, no matter which way the instructions are written (except when > you > have to use regex matches) I don't fully agree here with you. For example, the traditional PRI based filter is lightning fast. I don't see any way it could be nearly as fast with any unified approach. Right now, we have a "unified" filter structure, but it contains three filter cases, each one adding potential power at the price of decreased speed. I think we need to keep different types of filters in order to have some lightning-fast ones. But more of this could be done under the hood. Rainer > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 11:15:21 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:15:21 +0100 Subject: [rsyslog] rsyslog migration and syslogd.conf References: <20100227063807.5918fafc@mistral.stie> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 1:02 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog migration and syslogd.conf > > On Sat, 27 Feb 2010, lanas wrote: > > > Hello, > > > > First, thanks to David for the reply on time stamps. > > > > I have a question about perhaps being able to replace syslogd by > > rsyslog in a so-called 'drop-in' way. Eg. will rsyslog work (I > > reckon with limited possibilities) with a standard syslog config > > file ? That would enable a basic set of functionality as a first > step > > toward using rsyslog's features. > > no, rsyslog will do nothing if you just feed it a syslog.conf file. Actually, rsyslog is designed as a drop-in replacement. So it should work if you just feed it a regular rsyslog.conf. This is controlled via the -c option, which sets the compatibility level. If you call rsyslogd like you called sysklogd (without the -c option), it will fall back to sysklogd compatibility and generate the necessary config commands before parsing the the config file. Then, -r starts up a UDP listener as it did in sysklogd. ... At least this is how it is meant to work, I have to admit I have not tested it lately. For various reasons, I could not add this to the automated testbench, so I might not have noticed a regression. But if it does not work, this is a bug and not how things are designed to work. Rainer > rsyslog needs a few additional instructions at the top of the file to > load > the modules, but below that you can paste in the syslog.conf file. > > another thing that you can do is to make a generic rsyslog.conf file > that > includes the syslog.conf after the rsyslog specific headers. > > My rsyslog.conf (on my ubuntu 9.10 laptop) has > > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $PrivDropToUser syslog > $PrivDropToGroup syslog > $RepeatedMsgReduction on > $RepeatedMsgContainsOriginalMsg on > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Sun Feb 28 11:28:18 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 02:28:18 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA710395D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710397C@GRFEXC.intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang From rgerhards at hq.adiscon.com Sun Feb 28 11:30:24 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 11:30:24 +0100 Subject: [rsyslog] Log Normalization effort References: <9B6E2A8877C38245BFB15CC491A11DA710397A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397E@GRFEXC.intern.adiscon.com> David, thanks again for the good feedback. I have a couple of points: First of all: I agree that a "one size fits all approach" does not exist. Thus the idea to craft a library, which than can be used at various places: e.g. inside message parsers, output modules or even processing runs outside of rsyslog. Secondly, an efficient normalizing parser tree must not be much slower than the regular parser. I think that the parser overhead will be very acceptable for average messages. Another story, however, is the normalized data that has been gathered. In short words, that is extra data, so copy overhead is much higher everywhere. Also, accessing the properties take some time. I guess that's the primary problem inside a real-time solution, even if very efficient lookup methods are used. On the normalized properties: I think it is really worth the effort to try to define an as-broad as possible set of normalized properties. But that does not imply all needs to be done at once. It can be an evolutionary approach. First of all, one would look for pretty obvious things like traffic flow data (one may even reuse existing data models like from ipfix!) and user login/logoff activity. The key point IMHO is that we would just need to gather what people so far have used and try to get folks to re-use these fields that already exist. If we have success with this approach, I think, we will have huge benefit in the reporting and analysis area where programs could then work on standard property sets (of normalized syslog data) rather than on the raw data itself. So you do not need to write an analyzer for each of the myriad vendor/device/version formats but rather only once for the normalized data and then create a one-line (!) parse template for each of the vendor/device/version formats (of course, for each message there, so it would be multiple lines, but very easy and intuitively to do). However, all of that does not really work without community involvement. None (expect large commercial entities) can create the necessary mass of parse templates alone. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, February 26, 2010 10:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Log Normalization effort > > On Fri, 26 Feb 2010, Rainer Gerhards wrote: > > > Hi all, > > > > I have blogged about my quest for log normalization. I think there is > some > > good information on the upcoming GPLed Adiscon LogAnalyzer and future > > directions for rsyslog in the blog post. So I thought I share the > link: > > > > http://blog.gerhards.net/2010/02/syslog-normalization.html > > > > Please note that part of the effort requires community involvement. I > would > > be very interested to learn if you think we could win enough support > to make > > this a useful effort. I am asking for your feedback, because it will > help me > > streamline my priorities for future rsyslog work. > > a few comments (but remember that I am usually dealing with high data > rates, so my concerns are biased in that direction) > > log analysis is usually done in batches as opposed to in real-time. > some of this is due to the difficulty in doing it in real time, but a > lot > of it is the processing overhead (you don't want to take so long to > process an individual request that you miss the next one to arrive) > > at low volumes the idea of name-value pairs in the logs makes a lot of > sense, but there is significantly more overhead in parsing a log with > name-value pairs in arbitrary orders than there is in using a tree > parsing > approach to analyze known log formats in a fixed order. The message > size > can also increase significantly. As a result, at high traffic volumes > this > starts to be a bad (or at least questionable) idea. > > I would love to see rsyslog gain the ability to efficiently do tree- > based > parsing instead of regex parsing. regex parsing is easy to understand > and > tinker with, but very expensive to implement. it may be that having > something that 'compiles' a list of regex parsers into a tree parser is > the right answer for usability. I would save several hours of > processing > a day if I could easily (and efficiently) make rsyslog write different > logs to different files (at high data rates and with a few hundred > conditions based variations in the syslog tag) > > > While there are some common events across different types of logs > (logins > for example) they almost always contain slightly different data in > them. I > also have no faith at all that anyone is going to make much effort to > clean up their logs to make them nicely parseable, and if they do I see > even less chance that they will end up using the same terms for the > same > thing. As such I see more value in trying to get samples of logs and > what > they mean than in trying to define a normalized version to shoehorn the > logs into. It is worth doing this for some events (logins, failed > logins > for example), but I think it's a mistake to think that this will end up > covering all, or even the majority of log messages. > > There's also a problem in that the ideal format for the output depends > on > what you are doing with the output. > > > If I could wave a magic wand and get the result I would look for > something > like this > > the parser starts at the beginning of the message (at the priority) and > can branch on priority/faclilty, timestamp, host, syslogtag, message > and > indicate if the message should be parsed into name-value pairs, or > split > based on a character (or character sequence like the perl split command > allows) into individually addressable elements (defaulting to > whitespace > separated elements), then the format (and if needed dynafile path/file > components) could be constructed from these variables. At any point in > the > parsing it should be possible to jump to another parser tree (so that > you > could say that sm-mta, sendmail, Sendmail, etc as syslog tags all end > up > using the same parser for the message without having to redefine the > rules > for each one) > > With this capability, people could start writing parser 'branches' to > understand a specific log type and output a 'standard' format (as such > a > format can start to be defined). > > This can be done in rsyslog today, but it is fairly difficult to > define, > and as I understand it, inefficient enough that it's not practical to > do > in real-time under heavy load. > > > If this is fast enough, then the next step would be to add the ability > to > have the format/action be 'increment a counter for log type X' and a > signal to rsyslog could generate a report on these counters. Although > at > some point it becomes better to feed the message into another > opensource > tool (SEC, Simple Event Correlator for example) instead of trying to do > everything in rsyslog. > > parsing the file to know what to do with it, and be able to re-format > log > messages is very defiantly something that can fit into the rsyslog > model > of receiving, formatting, and delivering logs. Alerting on specific log > entries, counting the number of times one thing shows up in logs, and > this sort of thing start pushing > beyond the core of rsyslog, and it may be better to feed other tools > instead. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 12:25:14 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 12:25:14 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements Message-ID: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 28.02.10 11:28 Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements On Sun, 28 Feb 2010, Rainer Gerhards wrote: >> >> if the type is depriciated then the destination would be another >> config_option (which you would then look up) >> >> if the type is 'ignore' then it would just be skipped over (possibly >> with >> a warning being logged that the config line is being ignored) >> >> this would also clean up some of the current cases where a boolean >> option >> doesn't honor on/off and true/false. > > True/false is NOT a valid boolean option. See > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 > 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 I was not remembering correctly then, maby it was yes/no vs on/off. I know I ran into something where what was documented didn't work and I changed it to another version of a boolean answer and it fixed the problem. >> For the second half of the config language (telling rsyslog what to do >> with the logs it has received) also has several variations in place and >> is >> showing issues. >> >> However I think that the right answer here is to end up implementing >> something like the parsing trees and then compile the other config >> language options into that tree to be consistant (and fast) under the >> covers, no matter which way the instructions are written (except when >> you >> have to use regex matches) > > I don't fully agree here with you. For example, the traditional PRI based > filter is lightning fast. I don't see any way it could be nearly as fast with > any unified approach. Right now, we have a "unified" filter structure, but it > contains three filter cases, each one adding potential power at the price of > decreased speed. I think we need to keep different types of filters in order > to have some lightning-fast ones. But more of this could be done under the > hood. My guess/expectation is that the tree-based parsing would be about the same speed as the traditional PRI based filter for choices made based on PRI, slowing down for other types of conditions only in that more of the message may need to be scanned (and if there is no selection at a level, skipping that level should be lightning fast). As a result, a set of filteres based soley on programname would be as fast as the current PRI filters. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Sun Feb 28 14:01:58 2010 From: david at lang.hm (david at lang.hm) Date: Sun, 28 Feb 2010 05:01:58 -0800 (PST) Subject: [rsyslog] Feedback requested: inconsistency in config statements In-Reply-To: <001201cab868$d148da7c$100013ac@intern.adiscon.com> References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: On Sun, 28 Feb 2010, Rainer Gerhards wrote: > Quickly from the mobile: pri based filters require *excatly* one single word (!) lookup currently. You can't beat that. > > I am not totally sure how you think about parse trees. In my view, parsin and filteing are two different things./stages. Maybe we have different concepts on our minds? I am probably using the wrong terminology here. say you have a set of rules that say if programname startswith abc log to /abc if programname startswith acc log to /acc if programname startswith acc log to @1.1.1.1 if programname startswith bcd log to /bcd and assuming that these were the only possible programnames for simplicity in the explination. the filtering logic would go somthing like this the rules would compile into a tree -a-b -> /abc -c -> /acc, at 1.1.1.1 -b -> /bcd receive programname abc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "b", There are no other decisions to make, invoke the action /abc receive progranmane bcd I have no facility/PRI filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's a "b", There are no other decisions to make, invoke the action /bcd receive programname acc I have no facility/Pri filter rules I have no time filter rules I have no system filter rules I have programname filter rules look at the first character of the programname it's "a", I have more than one rule that fits that. look at the second character of the programname it's a "c", There are no other decisions to make, invoke the action /acc and the action @1.1.1.1 similarly the facility/pri filtering rules would be either compiled into a tree, or (since it's 256 entries total) into a lookup table with pointers to the list of actions to take for that entry the idea is to spend the time at startup to create the tree that represents the ruleset in order to speed up the processing of each individual message. the real tree would be a bit more complicated than I describe here as it needs to have 'anything else' entries between the known entries (which means that it would not be able to shortcut quite as much as I describe), and have provision for 'do a more complicated thing (like regex) here and if it matches continue'. but except for regex matches, this would make processing the rules pretty much independant of how many rules there were or how complex the ruleset is. there doe snot need to be one single programname filter tree, if you had a rule that said if pri = info and programname startswith def then def the pri tree/table would have an entry for info that would point to a filter tree for programname that just had the check for def in it am I makeing any more sense now? David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 28.02.10 11:28 > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > >>> >>> if the type is depriciated then the destination would be another >>> config_option (which you would then look up) >>> >>> if the type is 'ignore' then it would just be skipped over (possibly >>> with >>> a warning being logged that the config line is being ignored) >>> >>> this would also clean up some of the current cases where a boolean >>> option >>> doesn't honor on/off and true/false. >> >> True/false is NOT a valid boolean option. See >> >> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87 >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > I was not remembering correctly then, maby it was yes/no vs on/off. I know > I ran into something where what was documented didn't work and I changed > it to another version of a boolean answer and it fixed the problem. > >>> For the second half of the config language (telling rsyslog what to do >>> with the logs it has received) also has several variations in place and >>> is >>> showing issues. >>> >>> However I think that the right answer here is to end up implementing >>> something like the parsing trees and then compile the other config >>> language options into that tree to be consistant (and fast) under the >>> covers, no matter which way the instructions are written (except when >>> you >>> have to use regex matches) >> >> I don't fully agree here with you. For example, the traditional PRI based >> filter is lightning fast. I don't see any way it could be nearly as fast with >> any unified approach. Right now, we have a "unified" filter structure, but it >> contains three filter cases, each one adding potential power at the price of >> decreased speed. I think we need to keep different types of filters in order >> to have some lightning-fast ones. But more of this could be done under the >> hood. > > My guess/expectation is that the tree-based parsing would be about the > same speed as the traditional PRI based filter for choices made based on > PRI, slowing down for other types of conditions only in that more of the > message may need to be scanned (and if there is no selection at a level, > skipping that level should be lightning fast). As a result, a set of > filteres based soley on programname would be as fast as the current PRI > filters. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Sun Feb 28 14:15:46 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Sun, 28 Feb 2010 14:15:46 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: 2010/2/26 Michael Biebl : > 2010/2/26 Rainer Gerhards : >>> The problem is, that xconsole is running at this point! >>> To reiterate: >>> I start rsyslogd >>> then xconsole >>> then try to log a message >>> The result is the debug log and no message showing up in xconsole. >> >> that's what I am doing as well. I will try to record the session. > > Wait a sec. > With 4.4.2 (or any earlier version) I tried this: > start rsyslog > start xconsole > xconsole shows the last two log messages immediately > > With 4.6.0 the xconsole window is blank (which I guess made me believe > it blocked) > > With 4.6.0, I need to trigger a new log message (e.g. using logger > foo), and then xconsole will show a new message. > > So, xconsole does indeed show messages, but the behaviour is different > to 4.4 and earlier Hm, it just happened again. I rebooted my system, logged into my system, ran "sudo xconsole", "logger foo", but nothing shows up in the xconsole window. I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, "logger foo" and this time it works. Weird. There is definitely something fishy here. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Sun Feb 28 14:56:30 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 14:56:30 +0100 Subject: [rsyslog] Feedback requested: inconsistency in config statements References: <001201cab868$d148da7c$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710397F@GRFEXC.intern.adiscon.com> Hi David, yes, that makes much more sense to me. Let's call that filter expression tree for the time being (the parse tree used for normalization has some similar ideas, but a totally different scope and semantics). While I now see where you are heading, I fail to see how this could sufficiently easy be implemented for a real (and complex) configuration. Let's look at this example conf: mail.* /Var/log/mail.log if programname startswith abc log to /abc if programname startswith 'acc' and MSG contains 'error' log to /acc # we don't want debug messages from here on *.debug ~ if programname startswith 'acc' or (programname contains 'bde' and (facility > 1 and facility < 5)) log to @1.1.1.1 if programname startswith bcd or MSG startswith '123' or MSG contains 'error2' log to /bcd *.* /var/log/catchall I simply cannot envision how you would store this as a tree that does not look like what we have today. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Sunday, February 28, 2010 2:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > Quickly from the mobile: pri based filters require *excatly* one > single word (!) lookup currently. You can't beat that. > > > > I am not totally sure how you think about parse trees. In my view, > parsin and filteing are two different things./stages. Maybe we have > different concepts on our minds? > > I am probably using the wrong terminology here. > > say you have a set of rules that say > > if programname startswith abc log to /abc > if programname startswith acc log to /acc > if programname startswith acc log to @1.1.1.1 > if programname startswith bcd log to /bcd > > and assuming that these were the only possible programnames for > simplicity > in the explination. > > the filtering logic would go somthing like this > > the rules would compile into a tree > > -a-b -> /abc > -c -> /acc, at 1.1.1.1 > -b -> /bcd > > receive programname abc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "b", There are no other decisions to make, > invoke the action /abc > > receive progranmane bcd > I have no facility/PRI filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's a "b", There are no other decisions to make, > invoke the action /bcd > > receive programname acc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "c", There are no other decisions to make, > invoke the action /acc and the action @1.1.1.1 > > similarly the facility/pri filtering rules would be either compiled > into a > tree, or (since it's 256 entries total) into a lookup table with > pointers > to the list of actions to take for that entry > > the idea is to spend the time at startup to create the tree that > represents the ruleset in order to speed up the processing of each > individual message. > > the real tree would be a bit more complicated than I describe here as > it > needs to have 'anything else' entries between the known entries (which > means that it would not be able to shortcut quite as much as I > describe), > and have provision for 'do a more complicated thing (like regex) here > and > if it matches continue'. but except for regex matches, this would make > processing the rules pretty much independant of how many rules there > were > or how complex the ruleset is. > > there doe snot need to be one single programname filter tree, if you > had a > rule that said > if pri = info and programname startswith def then def > > the pri tree/table would have an entry for info that would point to a > filter tree for programname that just had the check for def in it > > am I makeing any more sense now? > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 28.02.10 11:28 > > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > >>> > >>> if the type is depriciated then the destination would be another > >>> config_option (which you would then look up) > >>> > >>> if the type is 'ignore' then it would just be skipped over > (possibly > >>> with > >>> a warning being logged that the config line is being ignored) > >>> > >>> this would also clean up some of the current cases where a boolean > >>> option > >>> doesn't honor on/off and true/false. > >> > >> True/false is NOT a valid boolean option. See > >> > >> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18 > 4c0d87 > >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > > I was not remembering correctly then, maby it was yes/no vs on/off. I > know > > I ran into something where what was documented didn't work and I > changed > > it to another version of a boolean answer and it fixed the problem. > > > >>> For the second half of the config language (telling rsyslog what to > do > >>> with the logs it has received) also has several variations in place > and > >>> is > >>> showing issues. > >>> > >>> However I think that the right answer here is to end up > implementing > >>> something like the parsing trees and then compile the other config > >>> language options into that tree to be consistant (and fast) under > the > >>> covers, no matter which way the instructions are written (except > when > >>> you > >>> have to use regex matches) > >> > >> I don't fully agree here with you. For example, the traditional PRI > based > >> filter is lightning fast. I don't see any way it could be nearly as > fast with > >> any unified approach. Right now, we have a "unified" filter > structure, but it > >> contains three filter cases, each one adding potential power at the > price of > >> decreased speed. I think we need to keep different types of filters > in order > >> to have some lightning-fast ones. But more of this could be done > under the > >> hood. > > > > My guess/expectation is that the tree-based parsing would be about > the > > same speed as the traditional PRI based filter for choices made based > on > > PRI, slowing down for other types of conditions only in that more of > the > > message may need to be scanned (and if there is no selection at a > level, > > skipping that level should be lightning fast). As a result, a set of > > filteres based soley on programname would be as fast as the current > PRI > > filters. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Sun Feb 28 15:01:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 15:01:54 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Sunday, February 28, 2010 2:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 4.6.0 (v4-stable) released > > 2010/2/26 Michael Biebl : > > 2010/2/26 Rainer Gerhards : > >>> The problem is, that xconsole is running at this point! > >>> To reiterate: > >>> I start rsyslogd > >>> then xconsole > >>> then try to log a message > >>> The result is the debug log and no message showing up in xconsole. > >> > >> that's what I am doing as well. I will try to record the session. > > > > Wait a sec. > > With 4.4.2 (or any earlier version) I tried this: > > start rsyslog > > start xconsole > > xconsole shows the last two log messages immediately > > > > With 4.6.0 the xconsole window is blank (which I guess made me > believe > > it blocked) > > > > With 4.6.0, I need to trigger a new log message (e.g. using logger > > foo), and then xconsole will show a new message. > > > > So, xconsole does indeed show messages, but the behaviour is > different > > to 4.4 and earlier > > Hm, it just happened again. > I rebooted my system, logged into my system, ran "sudo xconsole", > "logger foo", but nothing shows up in the xconsole window. > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > "logger foo" and this time it works. > > Weird. There is definitely something fishy here. OK, rather than trying to figure out what may be going on here, I'll see that I copy the old omfile code and create a new ompipe. So that will behave exactly like it did before. The new optimizations are not so important for pipes. Rainer From rgerhards at hq.adiscon.com Sun Feb 28 17:33:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 28 Feb 2010 17:33:59 +0100 Subject: [rsyslog] rsyslog 4.6.0 (v4-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA710394B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103968@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA710396F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7103972@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA7103980@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103981@GRFEXC.intern.adiscon.com> > > Hm, it just happened again. > > I rebooted my system, logged into my system, ran "sudo xconsole", > > "logger foo", but nothing shows up in the xconsole window. > > > > I then killed xconsole, restarted rsyslog, ran "sudo xconsole" again, > > "logger foo" and this time it works. > > > > Weird. There is definitely something fishy here. > > OK, rather than trying to figure out what may be going on here, I'll > see that > I copy the old omfile code and create a new ompipe. So that will behave > exactly like it did before. The new optimizations are not so important > for > pipes. Michael, I have done this now. I would appreciate if you could checkout the v4-stable git head and try with that one. It should now have the exact same behavior as in 4.4.2 (the pipe code is now the same!). I tested on Debian and it worked like before. Thinking about it, I guess it helps ease development in the future, because I now do not need to think about pipes when I work on omfile and vice versa. Rainer