From leitstelle at bnet.li Mon Mar 3 15:07:23 2008 From: leitstelle at bnet.li (leitstelle at bnet.li) Date: Mon, 03 Mar 2008 15:07:23 +0100 Subject: [rsyslog] Selector and MySQL Message-ID: <20080303150723.5h0cdcxbeygwcwoo@webmail.bnet.li> Hi all, i tried to write data into a DB with a selector in rsyslog.conf: ... !dhcpd $template test, "...",sql :msg, contains, "dinfo" >127.0.0.1,db,user,pw;test #eof one of the messages, which rsyslog is like: 2008-03-03 14:14.20 dhcpd : dinfo:, myipadress and mac So from my point of view the selector should write this message to the DB. BUT it doesn`t. If i replace the selector e.g. with *.info it works (but then all message are written to the DB) Thanks for your support Andre From rgerhards at hq.adiscon.com Mon Mar 3 15:17:21 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 3 Mar 2008 15:17:21 +0100 Subject: [rsyslog] Selector and MySQL In-Reply-To: <20080303150723.5h0cdcxbeygwcwoo@webmail.bnet.li> References: <20080303150723.5h0cdcxbeygwcwoo@webmail.bnet.li> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308AC0@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of leitstelle at bnet.li > Sent: Monday, March 03, 2008 3:07 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Selector and MySQL > > Hi all, > > i tried to write data into a DB with a selector > > in rsyslog.conf: > ... > !dhcpd > $template test, "...",sql > :msg, contains, "dinfo" >127.0.0.1,db,user,pw;test > #eof > > one of the messages, which rsyslog is like: > 2008-03-03 14:14.20 dhcpd : dinfo:, myipadress and mac Umm... That message format looks strange. I guess the parser gets confused by it. Can you please run rsyslogd in interactively in debug mode (-d -n) and post the resulting log while such a message is received. Thanks, Rainer > > So from my point of view the selector should write this message to the > DB. > BUT it doesn`t. > > If i replace the selector e.g. with *.info it works (but then all > message are written to the DB) > > Thanks for your support > Andre > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Mon Mar 3 16:27:30 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 3 Mar 2008 16:27:30 +0100 Subject: [rsyslog] Solaris Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308AC4@grfint2.intern.adiscon.com> Hi all, is there anybody on this list with (open) Solaris? I installed a system and tried to compile rsyslog on it, but I can't get the environment ready :( I somebody is in the know, advise would deeply be appreciated (if not, that's a good indication I still do not need to care about Solaris, which also solves my issue ;)). Thanks, Rainer From Mark.Capelle at pcmc.com Tue Mar 4 15:43:19 2008 From: Mark.Capelle at pcmc.com (Capelle, Mark (PCMC-GB)) Date: Tue, 4 Mar 2008 08:43:19 -0600 Subject: [rsyslog] Recording IP address of sender in logs Message-ID: <12ED24B2537575458993F162D4CB3F6C43EA1D@PcmcMxsUsr01.BWCINC.ORG> I have been using syslogd 1.4.1 for quite some time to gather syslogs from multiple hosts which I then have a script which separates out the logs based on the source host. I just rebuilt my syslog server and took the opportunity to use rsyslogd 2.0.2 instead of the syslogd, but I noticed that my syslogs no longer record the address of the source host which will break my scripts. Is there any way to change this behavior? Examples - Syslogd: Mar 4 08:05:47 10.10.1.1 Mar 04 2008 08:05:47: %ASA-4-106023: Deny tcp src inside:x.x.x.x/3713 dst outside:x.x.x.x/1021 by access-group "inside_access_out" [0x0, 0x0] ^^^^^^^^^ Rsyslogd: Mar 4 08:41:17 Mar 04 2008 08:41:25: %ASA-4-106023: Deny tcp src inside:x.x.x.x/2125 dst outside:x.x.x.x/9018 by access-group "inside_access_out" [0x0, 0x0] Regards, Mark From rgerhards at hq.adiscon.com Tue Mar 4 16:58:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 4 Mar 2008 16:58:23 +0100 Subject: [rsyslog] Recording IP address of sender in logs In-Reply-To: <12ED24B2537575458993F162D4CB3F6C43EA1D@PcmcMxsUsr01.BWCINC.ORG> References: <12ED24B2537575458993F162D4CB3F6C43EA1D@PcmcMxsUsr01.BWCINC.ORG> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308AD4@grfint2.intern.adiscon.com> Mark, it looks like message format received is somewhat invalid. Rsyslog tries to parse according to rfc 3164 but also applies legacy legacy parsing if it finds a way to do so. If you could provide me a wireshark capture of the message, I could see if I can modify the parser. HOWEVER, that shouldn't limit to do you what you want. Nothing is fixed in rsyslog and you can modify the format via a template. It is just the default template that gives you the result below. Please see the rsyslog doc on how to define other template. If you have a hard time, let me know and I'll figure it out for you, but I am currently very busy with new work, so I can't promise on the timeline. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Capelle, Mark (PCMC-GB) > Sent: Tuesday, March 04, 2008 3:43 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Recording IP address of sender in logs > > I have been using syslogd 1.4.1 for quite some time to gather syslogs > from multiple hosts which I then have a script which separates out the > logs based on the source host. I just rebuilt my syslog server and > took > the opportunity to use rsyslogd 2.0.2 instead of the syslogd, but I > noticed that my syslogs no longer record the address of the source host > which will break my scripts. Is there any way to change this behavior? > > > > Examples - > > > > Syslogd: > > Mar 4 08:05:47 10.10.1.1 Mar 04 2008 08:05:47: > %ASA-4-106023: Deny tcp src inside:x.x.x.x/3713 dst > outside:x.x.x.x/1021 > by access-group "inside_access_out" [0x0, 0x0] > > ^^^^^^^^^ > > > > Rsyslogd: > > Mar 4 08:41:17 Mar 04 2008 08:41:25: %ASA-4-106023: Deny > tcp src inside:x.x.x.x/2125 dst outside:x.x.x.x/9018 by access-group > "inside_access_out" [0x0, 0x0] > > > > > > Regards, > > Mark > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From linuxweb at gmail.com Thu Mar 6 01:49:46 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 05 Mar 2008 19:49:46 -0500 Subject: [rsyslog] local/remote logging Message-ID: <47CF3FAA.6050302@gmail.com> I want to have logs written locally (the way they usually are) as well as sent to a central logger. The two situations that I want local logs for are: 1) in case the central logger is not available for any reason, I don't lose messages 2) if I need to troubleshoot something, I can just look at the local logs Obviously, I will rotate the local logs and have older ones discarded. Whereas the central logger will archive them for longer periods. Can I accomplish this with rsyslog? AIUI, I can, and I have to move to at least v3.11.x -- is that correct? Assuming yes, I already got the remote logging working. What do I do to get the simultaneous local logging to work too? I can't find any wiki or faq on this. If I get it working, I'd be happy to post it on the site. johnn From mbiebl at gmail.com Thu Mar 6 02:24:12 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 6 Mar 2008 02:24:12 +0100 Subject: [rsyslog] Solaris In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308AC4@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308AC4@grfint2.intern.adiscon.com> Message-ID: 2008/3/3, Rainer Gerhards : > Hi all, > > is there anybody on this list with (open) Solaris? I installed a system > and tried to compile rsyslog on it, but I can't get the environment > ready :( > > I somebody is in the know, advise would deeply be appreciated (if not, > that's a good indication I still do not need to care about Solaris, > which also solves my issue ;)). http://blogs.sun.com/AngeloJ/entry/virtualbox_and_opensolaris_indiana ? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Mar 5 19:16:18 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 05 Mar 2008 19:16:18 +0100 Subject: [rsyslog] local/remote logging In-Reply-To: <47CF3FAA.6050302@gmail.com> References: <47CF3FAA.6050302@gmail.com> Message-ID: <1204740978.567.17.camel@localhost.localdomain> Hi Johnn, On Wed, 2008-03-05 at 19:49 -0500, Johnny Tan wrote: > I want to have logs written locally (the way they usually > are) as well as sent to a central logger. > > The two situations that I want local logs for are: > > 1) in case the central logger is not available for any > reason, I don't lose messages You don't need to lose message when the remote host is unavailable. Use queued mode. This is currently documented for the database, but works for any other action as well: http://www.rsyslog.com/doc-rsyslog_high_database_rate.html > > 2) if I need to troubleshoot something, I can just look at > the local logs > > Obviously, I will rotate the local logs and have older ones > discarded. Whereas the central logger will archive them for > longer periods. > > > Can I accomplish this with rsyslog? AIUI, I can, and I have > to move to at least v3.11.x -- is that correct? Indeed. But don't use anything earlier than 3.12.0 - if you do, you'll run into a couple of bugs. > Assuming yes, I already got the remote logging working. What > do I do to get the simultaneous local logging to work too? Rsyslog carries out whatever you configure. So just add a rule for local logging, e.g. *.* -/var/log/local-logfile That's it ;) > I can't find any wiki or faq on this. If I get it working, > I'd be happy to post it on the site. That would be much appreciated! HTH Rainer From rgerhards at hq.adiscon.com Thu Mar 6 12:24:16 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 6 Mar 2008 12:24:16 +0100 Subject: [rsyslog] rsyslog on HP UX Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B02@grfint2.intern.adiscon.com> Hi all, thanks to HP's testdrive, I got rsyslog to compile and basically run on HP UX. My testing abilities on the testdrive are limited (non-root access), but I was successfully able to receive a UDP syslog message and store it to a file. So the engine and the plugins seem to run quite OK. If anybody has HP UX and is interested in trying rsyslog out, please let me know. Obviously, you'll receive great support if you try ;) The HP UX "support" will be part of 3.12.1, which I hope to release soon (today or tomorrow). During the compile, there is a set of warnings that I do not fully understand. If somebody can shed some light on them, I'd deeply appreciate that: net.c: In function `AddAllowedSender': net.c:203: warning: cast increases required alignment of target type net.c:212: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:280: warning: cast increases required alignment of target type net.c:290: warning: cast increases required alignment of target type net.c:294: warning: cast increases required alignment of target type net.c:295: warning: cast increases required alignment of target type net.c:296: warning: cast increases required alignment of target type net.c: In function `MaskCmp': net.c:491: warning: cast increases required alignment of target type net.c:492: warning: cast increases required alignment of target type net.c:502: warning: cast increases required alignment of target type net.c:503: warning: cast increases required alignment of target type net.c:512: warning: cast increases required alignment of target type net.c:513: warning: cast increases required alignment of target type net.c:513: warning: cast increases required alignment of target type net.c:516: warning: cast increases required alignment of target type net.c:517: warning: cast increases required alignment of target type net.c: In function `debugListenInfo': net.c:744: warning: cast increases required alignment of target type net.c:749: warning: cast increases required alignment of target type No suffix list. They seem to be related to casting IPv4/IPv6 structures. Feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Thu Mar 6 16:39:10 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 6 Mar 2008 16:39:10 +0100 Subject: [rsyslog] rsyslog 3.12.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B0E@grfint2.intern.adiscon.com> Hi all, Rsyslog 3.12.1 is available immediately. There has been a log going on under the hood, code has been cleaned up and become more modular (you'll see more of that in the upcoming releases) and a library plugin-interface has been added (which so far is not really visible to the end user). We also worked a bit on portability and rsyslogd now compiles under HP UX. We are looking for testers, so if you have HP UX, you are invited to try it out. Feature-wise, a new syslog mib as been added, runtime debugging support enhanced and the ability to discard timestamps received via unix sockets has now been added (and is the default). This is necessary to keep time information consistent even if local programs log with wrong time information (what they do in practice). The gssapi code has greatly been changed and uses the new dynamic plugin library system. Consequently, some trouble in this regard cannot be ruled out ;). Finally, there is an important bugfix: failed actions were not properly retried, causing message loss in a destination-failure scenario. This is now corrected. Download: http://www.rsyslog.com/Downloads-req-getit-lid-85.phtml Change Log: http://www.rsyslog.com/Article187.phtml As always, feedback is appreciated. Rainer Gerhards From stephen.carville at gmail.com Thu Mar 6 18:54:06 2008 From: stephen.carville at gmail.com (Stephen Carville) Date: Thu, 6 Mar 2008 09:54:06 -0800 Subject: [rsyslog] Log watch software Message-ID: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> I have a cenltralized repository usng rsyslogm and syslog to mirror /var/log/messages, /var/log/secure ,and information messages from cfengine. In the near future I hope to get auditd reporting to a central server. My immedate taks is to add some log analysis software on teh central server. I've started modifiying LogWatch to work with MySQL -- thats pretty straightforward -- but I'm curious what other solutions there may be out there. FOSS is preferred but a I'm not against a reasonably priced commercial product. So far everything Google has returned are commercial products for Windows sytems. -- Stephen Carville From rgerhards at hq.adiscon.com Thu Mar 6 18:55:51 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 6 Mar 2008 18:55:51 +0100 Subject: [rsyslog] Log watch software In-Reply-To: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> References: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B15@grfint2.intern.adiscon.com> I am not so involved with logwatch. Let me ask feature-wise: what capabilities do you need to do the job? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Stephen Carville > Sent: Thursday, March 06, 2008 6:54 PM > To: rsyslog-users > Subject: [rsyslog] Log watch software > > I have a cenltralized repository usng rsyslogm and syslog to mirror > /var/log/messages, /var/log/secure ,and information messages from > cfengine. In the near future I hope to get auditd reporting to a > central server. My immedate taks is to add some log analysis software > on teh central server. I've started modifiying LogWatch to work with > MySQL -- thats pretty straightforward -- but I'm curious what other > solutions there may be out there. FOSS is preferred but a I'm not > against a reasonably priced commercial product. So far everything > Google has returned are commercial products for Windows sytems. > > -- > Stephen Carville > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From Gerrard.Geldenhuis at datacash.com Thu Mar 6 18:57:35 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Thu, 6 Mar 2008 17:57:35 -0000 Subject: [rsyslog] Log watch software In-Reply-To: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> References: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> Message-ID: Hi Stephen, Did you mean to ask a question... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Stephen Carville > Sent: 06 March 2008 17:54 > To: rsyslog-users > Subject: [rsyslog] Log watch software > > I have a cenltralized repository usng rsyslogm and syslog to mirror > /var/log/messages, /var/log/secure ,and information messages from > cfengine. In the near future I hope to get auditd reporting to a > central server. My immedate taks is to add some log analysis software > on teh central server. I've started modifiying LogWatch to work with > MySQL -- thats pretty straightforward -- but I'm curious what other > solutions there may be out there. FOSS is preferred but a I'm not > against a reasonably priced commercial product. So far everything > Google has returned are commercial products for Windows sytems. > > -- > Stephen Carville > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From stephen.carville at gmail.com Thu Mar 6 20:44:11 2008 From: stephen.carville at gmail.com (Stephen Carville) Date: Thu, 6 Mar 2008 11:44:11 -0800 Subject: [rsyslog] Log watch software In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B15@grfint2.intern.adiscon.com> References: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B15@grfint2.intern.adiscon.com> Message-ID: <2428c0380803061144p784f3b93x44382fc4534b040a@mail.gmail.com> On Thu, Mar 6, 2008 at 9:55 AM, Rainer Gerhards wrote: > I am not so involved with logwatch. Let me ask feature-wise: what > capabilities do you need to do the job? About 99% of what's in messages or secure is trivia. JoeBob logged in, ran a sudo command and logged off. An authenticated mount request was received from ip.add.re.ss. That sort of thing. What I'm looking for is a parser that can pick out the (hopefully) rare messages that indicates a problem like a disk drive is reporting errors. I can modify big brother and logwatch to do this but I am curious if anyone has a favorite package I haven't heard of yet. > Rainer > > > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Stephen Carville > > Sent: Thursday, March 06, 2008 6:54 PM > > To: rsyslog-users > > Subject: [rsyslog] Log watch software > > > > I have a cenltralized repository usng rsyslogm and syslog to mirror > > /var/log/messages, /var/log/secure ,and information messages from > > cfengine. In the near future I hope to get auditd reporting to a > > central server. My immedate taks is to add some log analysis software > > on teh central server. I've started modifiying LogWatch to work with > > MySQL -- thats pretty straightforward -- but I'm curious what other > > solutions there may be out there. FOSS is preferred but a I'm not > > against a reasonably priced commercial product. So far everything > > Google has returned are commercial products for Windows sytems. > > > > -- > > Stephen Carville > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > -- Stephen Carville From rgerhards at hq.adiscon.com Thu Mar 6 21:27:13 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 6 Mar 2008 21:27:13 +0100 Subject: [rsyslog] Log watch software In-Reply-To: <2428c0380803061144p784f3b93x44382fc4534b040a@mail.gmail.com> References: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B15@grfint2.intern.adiscon.com> <2428c0380803061144p784f3b93x44382fc4534b040a@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B16@grfint2.intern.adiscon.com> Hi, I am seeing where you come from. That's the million-dollar question ;) I suggest you also post to the loganalysis list, that's probably a better place than over here: http://www.loganalysis.org/mailman/listinfo/loganalysis Let me hijack this thread to share an idea. Rsyslog has a lot of infrastructure in place. Once I am finished with the essentials (which will of course be in a few month...), I'd like to put that infrastructure to better use than just drive the simple outputs we currently have. One thing I have on my mind is an output plugin which stores (hashes) of all message within a timeframe (e.g. last 7 days). Then, when a new message comes in, it compares it to all previous messages and emits a special message itself if the message occured less than "n" times in the past. I think this goes into the direction of what you are looking for. But would it generally be considered to be a useful idea? Even though we are months away from an implementation, feedack would be very valuable to me as it helps me shape my mid- to long-term direction. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Stephen Carville > Sent: Thursday, March 06, 2008 8:44 PM > To: rsyslog-users > Subject: Re: [rsyslog] Log watch software > > On Thu, Mar 6, 2008 at 9:55 AM, Rainer Gerhards > wrote: > > I am not so involved with logwatch. Let me ask feature-wise: what > > capabilities do you need to do the job? > > About 99% of what's in messages or secure is trivia. JoeBob logged > in, ran a sudo command and logged off. An authenticated mount request > was received from ip.add.re.ss. That sort of thing. What I'm looking > for is a parser that can pick out the (hopefully) rare messages that > indicates a problem like a disk drive is reporting errors. > > I can modify big brother and logwatch to do this but I am curious if > anyone has a favorite package I haven't heard of yet. > > > Rainer > > > > > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Stephen Carville > > > Sent: Thursday, March 06, 2008 6:54 PM > > > To: rsyslog-users > > > Subject: [rsyslog] Log watch software > > > > > > I have a cenltralized repository usng rsyslogm and > syslog to mirror > > > /var/log/messages, /var/log/secure ,and information messages from > > > cfengine. In the near future I hope to get auditd reporting to a > > > central server. My immedate taks is to add some log > analysis software > > > on teh central server. I've started modifiying LogWatch > to work with > > > MySQL -- thats pretty straightforward -- but I'm curious > what other > > > solutions there may be out there. FOSS is preferred but a I'm not > > > against a reasonably priced commercial product. So far > everything > > > Google has returned are commercial products for Windows sytems. > > > > > > -- > > > Stephen Carville > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > > -- > Stephen Carville > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From linuxweb at gmail.com Thu Mar 6 21:47:47 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Thu, 06 Mar 2008 15:47:47 -0500 Subject: [rsyslog] local/remote logging In-Reply-To: <1204740978.567.17.camel@localhost.localdomain> References: <47CF3FAA.6050302@gmail.com> <1204740978.567.17.camel@localhost.localdomain> Message-ID: <47D05873.60300@gmail.com> Rainer Gerhards wrote: > Hi Johnn, > > On Wed, 2008-03-05 at 19:49 -0500, Johnny Tan wrote: >> I want to have logs written locally (the way they usually >> are) as well as sent to a central logger. >> >> The two situations that I want local logs for are: >> >> 1) in case the central logger is not available for any >> reason, I don't lose messages > > You don't need to lose message when the remote host is unavailable. Use > queued mode. This is currently documented for the database, but works > for any other action as well: Hmm... and this is doable even with 2.x, right? So I could have it queued in memory buffer long enough to bring it back up or bring up another one with same ip to accept the messages? >> Assuming yes, I already got the remote logging working. What >> do I do to get the simultaneous local logging to work too? > > Rsyslog carries out whatever you configure. So just add a rule for local > logging, e.g. > > *.* -/var/log/local-logfile Awesome! Thanks. I'll post on the wiki once I get it working. johnn From rgerhards at hq.adiscon.com Thu Mar 6 21:53:59 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 6 Mar 2008 21:53:59 +0100 Subject: [rsyslog] local/remote logging In-Reply-To: <47D05873.60300@gmail.com> References: <47CF3FAA.6050302@gmail.com><1204740978.567.17.camel@localhost.localdomain> <47D05873.60300@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B18@grfint2.intern.adiscon.com> > >> 1) in case the central logger is not available for any > >> reason, I don't lose messages > > > > You don't need to lose message when the remote host is > unavailable. Use > > queued mode. This is currently documented for the database, > but works > > for any other action as well: > > Hmm... and this is doable even with 2.x, right? Sorry, Dude, no such queues in v2 :-( > So I could have it queued in memory buffer long enough to > bring it back up or bring up another one with same ip to > accept the messages? It's *not* a main memory buffer. If you configure it to use disk *and* you have enough disk space, it can queue for ages. It's pretty advanced, the full doc on queues is here: http://www.rsyslog.com/doc-queues.html The disk, of course, is only touched if needed. If the destination server is only offline for a brief period, messages may be buffered in memory only. But if the volume is too large or you stop rsyslogd for a while, it is written disk. So if you have enough disk space and your destination dies on Friday and you are too lazy to come in over the weekend, rsyslogd will send all queued elements after the restart on Monday. The same, of course, works if you are on vacation ;) Rainer From Gerrard.Geldenhuis at datacash.com Fri Mar 7 10:34:17 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Fri, 7 Mar 2008 09:34:17 -0000 Subject: [rsyslog] Log watch software In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B16@grfint2.intern.adiscon.com> References: <2428c0380803060954h14664406u2cdd28892fa13075@mail.gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B15@grfint2.intern.adiscon.com><2428c0380803061144p784f3b93x44382fc4534b040a@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B16@grfint2.intern.adiscon.com> Message-ID: > Let me hijack this thread to share an idea. Rsyslog has a lot of > infrastructure in place. Once I am finished with the essentials (which > will of course be in a few month...), I'd like to put that > infrastructure to better use than just drive the simple outputs we > currently have. One thing I have on my mind is an output plugin which > stores (hashes) of all message within a timeframe (e.g. last 7 days). > Then, when a new message comes in, it compares it to all previous > messages and emits a special message itself if the message occured less > than "n" times in the past. I think this goes into the direction of what > you are looking for. > > But would it generally be considered to be a useful idea? Even though we > are months away from an implementation, feedack would be very valuable > to me as it helps me shape my mid- to long-term direction. > > Rainer Just thinking out loud... it would be very cool if one could build in some AI in such a plugin. You could then spend time "training" the plugin and buying "trained" AI's to regonize certain patterns in the logs etc. Regards From linuxweb at gmail.com Sat Mar 8 00:22:11 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Fri, 07 Mar 2008 18:22:11 -0500 Subject: [rsyslog] wiki with various application logging examples Message-ID: <47D1CE23.2010005@gmail.com> Just wondering if there is a wiki with examples of how to log to syslog/rsyslog for various apps. For my needs, I was specifically looking for apache, mysql, tomcat, puppet, etc. For apache, I already figured out error_log can just point to syslog, but access_log, you do: CustomLog "| /usr/bin/logger -p local0.info" combined Do I do something similar for any application where I can specify the logfile destination? johnn From rgerhards at hq.adiscon.com Sat Mar 8 13:54:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 8 Mar 2008 13:54:19 +0100 Subject: [rsyslog] wiki with various application logging examples Message-ID: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> I think nobody has added descriptions so far, but maybe that changes now ;) On the file logs - I suggest you look into imfule, that sound much cleaner and reliable to me than the logger trick. rainer ----- Urspr?ngliche Nachricht ----- Von: "Johnny Tan" An: "rsyslog-users" Gesendet: 08.03.08 00:22 Betreff: [rsyslog] wiki with various application logging examples Just wondering if there is a wiki with examples of how to log to syslog/rsyslog for various apps. For my needs, I was specifically looking for apache, mysql, tomcat, puppet, etc. For apache, I already figured out error_log can just point to syslog, but access_log, you do: CustomLog "| /usr/bin/logger -p local0.info" combined Do I do something similar for any application where I can specify the logfile destination? johnn _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Sat Mar 8 17:04:18 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 8 Mar 2008 17:04:18 +0100 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> Just to facilitate things: this is what I am thinking about: http://www.rsyslog.com/doc-imfile.html It can monitor all application log files natively. Did I understand your need right? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Rainer Gerhards > Sent: Saturday, March 08, 2008 1:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] wiki with various application logging examples > > I think nobody has added descriptions so far, but maybe that > changes now ;) > > On the file logs - I suggest you look into imfule, that sound > much cleaner and reliable to me than the logger trick. > > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Johnny Tan" > An: "rsyslog-users" > Gesendet: 08.03.08 00:22 > Betreff: [rsyslog] wiki with various application logging examples > > Just wondering if there is a wiki with examples of how to > log to syslog/rsyslog for various apps. > > For my needs, I was specifically looking for apache, mysql, > tomcat, puppet, etc. > > For apache, I already figured out error_log can just point > to syslog, but access_log, you do: > CustomLog "| /usr/bin/logger -p local0.info" combined > > Do I do something similar for any application where I can > specify the logfile destination? > > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From linuxweb at gmail.com Sun Mar 9 01:14:47 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Sat, 08 Mar 2008 19:14:47 -0500 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> Message-ID: <47D32BF7.5030108@gmail.com> Rainer Gerhards wrote: > Just to facilitate things: this is what I am thinking > about: > > http://www.rsyslog.com/doc-imfile.html > > It can monitor all application log files natively. Did I > understand your need right? Yes, that's exactly what I was looking for. Thanks, really really awesome! One quick error -- in your Sample, you use: $InputFilePollingInterval 10 It should be $InputFilePollInterval (no -ing on Poll) Thanks again! johnn From janfrode at tanso.net Sun Mar 9 12:56:15 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 9 Mar 2008 12:56:15 +0100 Subject: [rsyslog] wiki with various application logging examples References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> Message-ID: On 2008-03-08, Rainer Gerhards wrote: > Just to facilitate things: this is what I am thinking about: > > http://www.rsyslog.com/doc-imfile.html > > It can monitor all application log files natively. Did I understand your need right? > Seems like "imfile" could be trivially implemented trough: while sleep $InputFilePollInterval ; do logtail $InputFileName $InputFileStateFile | \ logger -t $InputFileTag -p $InputFileFacility.$InputFileSeverity done http://debian.stro.at/bzr-test/logtail/ Or for realtime monitoring: tail --follow=name --retry $InputFileName | \ logger -t $InputFileTag -p $InputFileFacility.$InputFileSeverity IMHO rsyslogd might be trying to do too much internally. When will it start reading mail ? That might not sound too far fetched.. :-) "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." -- jwz -jf From mbiebl at gmail.com Sun Mar 9 15:02:04 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Sun, 9 Mar 2008 15:02:04 +0100 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> Message-ID: 2008/3/9, Jan-Frode Myklebust : > IMHO rsyslogd might be trying to do too much internally. When will it start > reading mail ? That might not sound too far fetched.. :-) > > "Every program attempts to expand until it can read mail. Those programs > which cannot so expand are replaced by ones which can." -- jwz To some extent, I share your concerns. I very much believe in the Unix philosophy of small tools, which do one job, and do that right and can be chained/combined easily. rsyslog should keeps it's focus and stay what it's primary focus is, a syslog daemon. Taking your email example: Having the functionality to send out email notifications on certain events is certainly useful. But we should not build the email sending functionality into rsyslog itself, but simply hand out (pipe) the data to external tools (mail/sendmail). The point is, what we can expect to be available on a given system or not ("logtail" probably isn't, "mail" very likely is). Also, as it's implemented as a loadable module, it doesn't clutter the main rsyslog code and so keeps the code base maintainable. But again ,I think your concern is valid and we should evaluate carefully what to implement directly into rsyslog and where it makes more sense to utilise external tools. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Sun Mar 9 18:38:00 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 9 Mar 2008 18:38:00 +0100 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B39@grfint2.intern.adiscon.com> Hi all, and especially JF and Michael, Good discussion, please keep the thoughts flowing :) Let me start by tell you I am guilty of a much broader view of what a syslogd can do than most people probably have. To me, a syslogd processes *system events* and that's much more than just those pieces sent via syslog. I know that I should probably stopped the rsyslog project at some point in time and fork an eventlogd from it. But, why would this have made any sense? To me, the core point is maintainability of code and the freedom for people to use only (and exactly) what they need (and want!) to use. After all, there are also lots of folks how say that database support has nothing to do with a syslogd and thus should not be part of if (and that was the birthplace of rsyslog... ;)). So for me the natural thing was to make rsyslog pluggable AND keep only in the actual core what is ultimately needed. I got very serious about this last summer. There are a couple of posts from last August in my blog and I'd like to highlight this one: http://rgerhards.blogspot.com/2007/08/on-importance-of-plug-ins-for-rsys log.html We have now reached a point where we have a quite well implemented plugin interface and rsyslogd has become much more modular. Still, there are things that people need to run even if they never need them (like the syslog-protocol message parser and a couple of "standard" outputs that come with sysklogd). I am continuously working to remove these dependencies. For example, on Friday I moved regex support out to a separate module. So you no longer need to load regexpes if you do not need them. Now to the plugins. What is a plugin? Good question. A plugin is *a project in itself*. Just for convenience, it is part of the main tarball (we had a couple of real headaches when we tried to separate the initial plugins and came to the conclusion that it is by far more convenient to deliver them as part of the main tarball). Plugins need not be written by the rsyslog team. Maybe somebody has already written an email reception plugin and just not told us - may very well be (though I doubt ;)). Now let me take on the imfile example. The key point - at least IMHO - is that there is no single line in rsyslog core's code that has been added in support of imfile. And, more importantly, if imfile would go away, not a single line of code could be removed. So the imfile plugin (project) does neither add complexity nor code nor other overhead to rsyslog core. So can it be evil? Imfile was added in response of a user request. When I heared it, I thought it would be useful to have such a plugin. Did I know about the logger approach? Yes - and I have read some many folks complain over this or that aspect of a logger-approach that I knew I didn't want it. Maybe these were all false claims, maybe people simply did not understand to do it right. I don't know and I honestly admit that I don't care. The presence of many people feeling bugged about that approach made me not like it (if you look for actual samples, I suggest going through the loganalysis mailing list archive). Did I know about logtail? No. Had I written imfile if I did? Not sure: from the small shell script you provided, it looks like there is a problem if a) script is in sleep period b) data is appended to text file c) text file is rotated d) new lines are written to text file e) script awake for new polling loop On a quick look, it looks like the data written in b) will never make it to the syslogd. Imfile handles that. But even if logtail (or the script) handles it - as I said, I did not know about it. So I took roughly a day of my time to create an initial imfile. If logtail handles the situation described above, would be available on the platforms I was asked for and I knew about it, I'd probably refrained from implementing imfile. But, again, no harm happened. If you don't like imfile, simply don't use it. It adds nothing to the rsyslog core. You can still use the logtail approach. IMHO this is what we want: freedom of choice. And rsyslog provides this freedom. On the mail output case (though I need to be a bit brief as dinner is approaching ;)): I actually intend to add an email output plugin. It's on the list for long a time. I personally see some benefit in the ability to talk to a tightly coupled, in-memory, very performant accessible way to send mail message (just like with the SNMP traps). On the other hand, you can see that I didn't implement it for a long time - simply because I did not consider it important enough given what else needs to be done. But I now need to speak up, because I have actually begun to seriously look into implementing it before this discussion begun ;) Before I do, I will make sure I understand that other options and if there is some benefit in doing so. But if I implement it for one reason or the other, nothing bad happens to rsyslog: if you don't like it, you don't need to use it. It doesn't add any overhead to the core. And now please feedback. Am I overlooking something real evil? What is the bad thing in offering exotic functionality that only few will ever need? Thanks again for the discussion, I am *really* interested in getting more feedback on this topic. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Sunday, March 09, 2008 3:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] wiki with various application logging examples > > 2008/3/9, Jan-Frode Myklebust : > > IMHO rsyslogd might be trying to do too much internally. When will > it start > > reading mail ? That might not sound too far fetched.. :-) > > > > "Every program attempts to expand until it can read mail. > Those programs > > which cannot so expand are replaced by ones which can." -- > jwz > > To some extent, I share your concerns. > I very much believe in the Unix philosophy of small tools, which do > one job, and do that right and can be chained/combined easily. > > rsyslog should keeps it's focus and stay what it's primary focus is, a > syslog daemon. > Taking your email example: > Having the functionality to send out email notifications on certain > events is certainly useful. But we should not build the email sending > functionality into rsyslog itself, but simply hand out (pipe) the data > to external tools (mail/sendmail). > > The point is, what we can expect to be available on a given system or > not ("logtail" probably isn't, "mail" very likely is). Also, as it's > implemented as a loadable module, it doesn't clutter the main rsyslog > code and so keeps the code base maintainable. > > But again ,I think your concern is valid and we should evaluate > carefully what to implement directly into rsyslog and where it makes > more sense to utilise external tools. > > Cheers, > Michael > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From janfrode at tanso.net Sun Mar 9 23:05:10 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 9 Mar 2008 23:05:10 +0100 Subject: [rsyslog] wiki with various application logging examples References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B39@grfint2.intern.adiscon.com> Message-ID: On 2008-03-09, Rainer Gerhards wrote: > > Now let me take on the imfile example. The key point - at least IMHO - > is that there is no single line in rsyslog core's code that has been > added in support of imfile. And, more importantly, if imfile would go > away, not a single line of code could be removed. So the imfile plugin > (project) does neither add complexity nor code nor other overhead to > rsyslog core. So can it be evil? It's evil that you're forcing me to upgrade to rsyslog v3.x to take advantage of it ;-) That's what triggered my previous post.. I want to be tracking non-syslog logfiles, and if imfile/plugins where more in the unix philosophy of small tools that chain easily.. I might have been able to pick it from v3.x and use it on stable systems. > from the small shell script you provided, it looks like there is a > problem if > > a) script is in sleep period > b) data is appended to text file > c) text file is rotated > d) new lines are written to text file > e) script awake for new polling loop > > On a quick look, it looks like the data written in b) will never make it > to the syslogd. Imfile handles that. Yes, you're right. Cool that imfile handles it. > On the mail output case (though I need to be a bit brief as dinner is > approaching ;)): I actually intend to add an email output plugin. The quote was about being able to *read* mail: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can." and I was thinking it not too far fetched since "splunk" can do it (it can download email messages via IMAP, index them and create alerts on suspicious content). It might seem like you want to take rsyslog in that direction, i.e. your complete eventlogd&alertSystem fork of rsyslogd that can read any input (syslog, other-logfile, email, snmptraps) and analyze and alert on the data. That's not what *I* want from a syslog server. I just want it to reliably collect and store the logs in an organized manner. Then I'll use other tools to read and analyze them. -jf From rgerhards at hq.adiscon.com Mon Mar 10 08:52:46 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 10 Mar 2008 08:52:46 +0100 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B39@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B3B@grfint2.intern.adiscon.com> Hi JF, thanks for the note. Of course, it triggers a couple of responses (see below). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jan-Frode Myklebust > Sent: Sunday, March 09, 2008 11:05 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] wiki with various application logging examples > > On 2008-03-09, Rainer Gerhards wrote: > > > > Now let me take on the imfile example. The key point - at least IMHO > - > > is that there is no single line in rsyslog core's code that has been > > added in support of imfile. And, more importantly, if imfile would go > > away, not a single line of code could be removed. So the imfile > plugin > > (project) does neither add complexity nor code nor other overhead to > > rsyslog core. So can it be evil? > > It's evil that you're forcing me to upgrade to rsyslog v3.x to take > advantage of it ;-) That's what triggered my previous post.. I want > to be tracking non-syslog logfiles, and if imfile/plugins where more > in the unix philosophy of small tools that chain easily.. I might have > been able to pick it from v3.x and use it on stable systems. I've see the smily but let me do a couple of comments. First of all, the "philosophy of small tools that chain easily" implies that they chain via a pipe. This is great for a lot of applications, but it has its drawbacks. A plain pipe is a simplex, relatively loosely coupled IPC method. So if one part of the pipe dies, other parts will learn about it eventually, but not at the same instant and, most importantly, they do not exactly know what was processed and what not. It's pretty much the same thing as plain TCP transport, which pretends reliability but still has a few windows of exposure where message loss may occur (see http://www.monitorware.com/en/workinprogress/selp.txt section 2.4 and, yes, rsyslog is victim to this as long as we don't have a full RFC 3195 implementation). So relying on plain pipes is not exactly my premier communications method if I would like to implement a reliable syslogd (and "reliable" is the "r" in rsyslog). One can circumvent the problem by defining an app-level protocol on top of the pipe, working with app-level acks. HOWEVER, with that you would run into the pretty same situation, namely that you could not use a new version together with an old engine. One can circumvent that, too, but only as far as the old engine has implemented such methodology. And v1/2 do not have this (due to time constraints, rsyslog is still only about half finished...). There is some other reason that makes me avoid pipes. If I do an app-level ack, I need to do at least 4 system calls to pass a single message: 1. input write to pipe 2. engine read msg from pipe 3. engine post processing stat (ACK) to pipe 4. input read ACK from pipe If you do a little bit of math, you'll see with how many user/kernel space transitions you end up, plus how many cycles are needed to run the necessary housekeeping code inside the kernel and libraries. I don't sum them up now, but I am pretty sure that I can completely process the message in less time than is need just for the IPC in that case. That probably is fine if you look at a low-end workstation syslogd, but you don't want to have this overhead if you aim for an enterprise solutions capable of handling massive data input. Also, if everything goes to the system log socket, there is little you can multithread. But we need to be able to multithread as much as possible. If I write a new engine (as I currently do), I would like to see it well working for at least the next 10 years. If I think about hardware trends during that period, it is clear that a single core will become not much faster than it is today. But the number of cores will greatly increase. In order to utilize that, an application must be able to run on as many threads as possible - all with reasonable overhead, of course. So my conclusion is that rsyslog must be able to run massively multithreaded for the high-end use case. This also prohibits using primary interfaces which cannot easily be multithreaded. Finally there is the issue of flow control. Rsyslog *does* flow control, and will do more advanced flow control in the future. Especially with world-dominating UDP syslog it is vitally important to do flow control, because UDP cannot be flow controlled. Sound strange? Well... If we can not flow control UDP syslog, we need to apply intelligent and adoptable flow control that flow controls sources who can (tcp syslog, rfc 3195 and of course file data!) so that buffer space is kept for those precious UDP message which are lost if we can not buffer them at the right instant. If you think this through, you'll see that this requires different level and methods of flow control, depending on the source [so far, I see three levels: can not (UDP), can somewhat (TCP, local sockets), can easily (log files and other sources that generate data themselves)]. Having a native interface greatly reduces code complexity and thus program reliability when it comes at implementing these features. Also, it would require specialized plugins in any case, you couldn't do it with a simple "pipe me in approach" (well, ... some things yes, but at a complexity cost). Also, while I too believe in the Unix approach of small tools, I also think it is important that the average user is able to configure it. Rsyslog aims not only at being enterprise-class but at the some time aims at being easy to use for the novice. Novices don't understand complex scripting to get the job done. I think relying too much on complex glue doesn't help getting the job done. To come back to your orginal post, rsyslogd *should* of course support the Unix way of piping. It looks like there is a small plugin missing to read natively from a pipe. However, I never got a request to implement it. I guess most people use the logger trick to accomplish that task. I know syslog-ng can natively read from pipes, but have not yet considered this important enough given the lack of requests from the community. If you like such a plugin, it's probably a good idea to speak up now ;) > > > > from the small shell script you provided, it looks like there is a > > problem if > > > > a) script is in sleep period > > b) data is appended to text file > > c) text file is rotated > > d) new lines are written to text file > > e) script awake for new polling loop > > > > On a quick look, it looks like the data written in b) will never make > it > > to the syslogd. Imfile handles that. > > Yes, you're right. Cool that imfile handles it. > > > > On the mail output case (though I need to be a bit brief as dinner is > > approaching ;)): I actually intend to add an email output plugin. > > The quote was about being able to *read* mail: I was replying here to Michael Biebel. It looks I was a bit too much in hurry to point this out. Sorry... > > "Every program attempts to expand until it can read mail. > Those programs which cannot so expand are replaced by ones > which can." > > and I was thinking it not too far fetched since "splunk" can do it (it > can > download email messages via IMAP, index them and create alerts on > suspicious > content). In short: receiving email is very low on my agenda. Keep in mind that I already have architecture and at least partly written such a beast on Windows: http://www.monitorware.com/en/Product/product_comparision.php The email question never was in much demand. But if demand comes up, it for sure is not a big thing to add it... (and, of course, you already can do it today with a bit of scripting, the right mailbox rules and logger -- but that isn't appealing to most folks and is one reason I tend to write plugins ;)). > It might seem like you want to take rsyslog in that > direction, i.e. > your complete eventlogd&alertSystem fork of rsyslogd that can read any > input > (syslog, other-logfile, email, snmptraps) and analyze and alert on the > data. > > That's not what *I* want from a syslog server. I just want it to > reliably > collect and store the logs in an organized manner. Then I'll use other > tools to read and analyze them. ... and this is of course perfectly fine with me, too. However, if you look at the core engine needs, you'll see that the "do it all" eventprocessor and the "plain simple syslogd" have exactly the same needs - at least if you would like to extend the syslogd to be enterprise class. Take the queue engine in v3. Its complex. Actually horribly complex. I didn't like to include that complex beast, but it was the only clean solution to the need of being massively concurrent AND being able to queue data while a destination is down. Any other alternative IMHO would have been dirty and hackish - and in the long term much less maintainable. So I just did the right thing (hopefully), even though it was a really big effort and even though it probably will need a few more firedrills before it is really ready for prime time in all scenarios. Another alternative would have been to use one of the big enterprise class message queueing projects. But that would have created a dependency for such a system on each desktop - ouch... I hope I made the right compromise. Currently, the full queue engine is part of the core. If that turns out to be a problem, I can outsource that to a plugin, but that isn't currently very appealing to me. After all, it's "just" some memory overhead - if you don't need the advanced features, no code is executed to do that. The queue uses an internal driver model and simply configuration means simple code. Thus, the v3 queue engine is as reliable as v2, except if you use all those bells and whistles where I am sure currently a few bugs wait to be detected (even though the situation has much improved recently and will improve with each new feedback I receive). Another good example is the config file: of course, there is no need to have a scriptable configuration for a simple syslogd. But while thinking about the (necessary) expression support and a lot of user requests for a better to read config file format, I came to the conclusion that creating a scriptable format is actually the right route to take: http://rgerhards.blogspot.com/2008/02/introducing-rainerscript-and-some. html Anything else (IMHO) would again be less clean, less maintainable and, in this case, would even take longer to implement. So one might think it is evil to include a virtual machine inside a syslogd, but to me it is actually the least effort to implement things. Of course, you can rightly argue that all of this is over the board if you just want to have a plain local logger that takes messages from the local log socket, maybe UDP syslog and store it to local files. You are probably right. But in this case, you can still continue to use sysklogd. After all, rsyslog was spawned from it to create and advanced syslogd. So it comes at no surprise that I am adding features which may not be required for the simple use cases. ;) Let me conclude with two core points: - rsyslog core is as slim as possible, plugins are separate projects that extend the core; This means nobody is forced to run more code than actually required for his job - both a simple, but fairly enhanced syslogd as well as a full network Event processor share the same root engine needs Thus, rsyslog implements this core engine and I occasionally add a plugin here and there to take advantage of the core. Right now, rsyslog core is far from being finished, as are the plugins. At this time, I am working on getting the core right and doing the most requested plugins. When I am done with that, I'll look at the *real* advanced plugins for all kinds of things that users have interest in. I don't see any need to fork of a separate core project for that. In fact, I think it would be counter-productive as I would need to maintain two code bases and the newly forked project would always be able to do what rsyslogd does. So what would be the motivation to maintain another, feature-less project...? Anyhow, I may be totally wrong. Feedback on this topic is still highly appreciated (be it brief or elaborate ;)). I am probably abusing this thread to also tell you a bit about design decisions I have not yet communicated (sorry for that, so much to do, so few time...;)). Rainer > > > > -jf > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Mon Mar 10 08:54:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 10 Mar 2008 08:54:42 +0100 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: <47D32BF7.5030108@gmail.com> References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> <47D32BF7.5030108@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B3C@grfint2.intern.adiscon.com> Hi Johnn, I'd really appreciate if you could any actual samples (even brief) to the wiki at http://wiki.rsyslog.com They would definitely be of great benefit for other users, too. As you have probably seen in my other mails, I am pretty busy with developing right now so I hope that users contribute use cases. Thanks, Raiener > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Sunday, March 09, 2008 1:15 AM > To: rsyslog-users > Subject: Re: [rsyslog] wiki with various application logging examples > > Rainer Gerhards wrote: > > Just to facilitate things: this is what I am thinking > > about: > > > > http://www.rsyslog.com/doc-imfile.html > > > > It can monitor all application log files natively. Did I > > understand your need right? > > Yes, that's exactly what I was looking for. Thanks, really > really awesome! > > One quick error -- in your Sample, you use: > $InputFilePollingInterval 10 > > It should be $InputFilePollInterval (no -ing on Poll) > > Thanks again! > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From linuxweb at gmail.com Mon Mar 10 17:01:22 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Mon, 10 Mar 2008 12:01:22 -0400 Subject: [rsyslog] wiki with various application logging examples In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B3C@grfint2.intern.adiscon.com> References: <005801c8811b$88a23ab6$060013ac@intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B38@grfint2.intern.adiscon.com> <47D32BF7.5030108@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B3C@grfint2.intern.adiscon.com> Message-ID: <47D55B52.7010905@gmail.com> Rainer Gerhards wrote: > Hi Johnn, > > I'd really appreciate if you could any actual samples (even brief) to > the wiki at > > http://wiki.rsyslog.com > > They would definitely be of great benefit for other users, too. As you > have probably seen in my other mails, I am pretty busy with developing > right now so I hope that users contribute use cases. Yes, absolutely! I'm a rsyslog convert :). Since we're going live this week, I've got a ton of things on my plate. But I fully intend to put up my samples (however basic they are) on the wiki soon after launch! Thanks for a great program! johnn From maurizio.rottin at gmail.com Tue Mar 11 13:01:57 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Tue, 11 Mar 2008 13:01:57 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log Message-ID: Hi all, i set up an apache webserver wich logs with directives: ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com " "combined" then syslog forwards them with local5.err @10.10.10.89 local6.info @10.10.10.89 and it works good! now on 10.10.10.89 i set up an rsyslog server with directives $template ApacheRemoteErr,%msg%"/var/log/rsyslog/Apache/%syslogtag%/Err.log" local5.err -?ApacheRemoteErr $template ApacheRemoteCustom,%msg%"/var/log/rsyslog/Apache/%syslogtag%/Custom.log" local6.info -?ApacheRemoteCustom the result is that i get these files in /var/log/rsyslog/Apache/ ./www.mysite.com:/Custom.log with customLog ./error_www.mysite.com:/Err.log with ErroLog ./message/Err.log with lines like Mar 11 12:32:52 last message repeated 9 times Mar 11 12:32:52 last message repeated 5 times (Note that this is very annoying!!!) Now this a simple way i have found to divide logs per VHost. I don't know if there is a better way, but the real question is different: In the Custom or Error logs i have lines like: Mar 11 12:32:53 php03 www.mysite.com: 10.10.10.2 - - [11/Mar/2008:12:35:07 +0100] "GET /pagina.php?id=61 HTTP/1.1" 200 93 "http://www.mysite.com/home.php" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.13) Gecko/20060607" i don't want the "Mar 11 12:32:53 php03 www.mysite.com:" part do i have to use a double template setup? one for the file and one for the content? the template in the file seems to be this: $template TraditionalFormat,%timegenerated% %HOSTNAME% %syslogtag%%msg%\n" but i need just $template TheOneINeed,%msg%\n" i want only the %msg%\n in my file Thanks in advance -- mr From linuxweb at gmail.com Tue Mar 11 19:08:06 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Tue, 11 Mar 2008 14:08:06 -0400 Subject: [rsyslog] rsyslog v3 and selinux Message-ID: <47D6CA86.6060501@gmail.com> I took Fedora 8's rsyslog v2.0.2 SRPM and rebuilt it for v3.12.1 I'm using the same init script. When I used that init script (i.e., "/etc/init.d/rsyslog start") on v2.0.2, the only SELinux problem I had was the domain transition, as documented here: http://tanso.net/rsyslog/ However, when I rebuilt the RPM for v3.12.1, and used the same init script to run it, I get many SELinux errors. Eventually, I worked out all the things I had to allow for rsyslog v3.12.1 to run properly. It's enclosed below as a semanage module. Just curious if there was some change to rsyslog between versions 2 and 3 which would make rsyslog, even when running properly in domain syslogd_exec_t, to cause so many SELinux denials, including not being able to do TCP bind?? Thanks for any clues, johnn p.s. Here's the module that finally worked. Will document on wiki when all is done. ========== module rsyslog 1.0; require { class dir search; class file { getattr read write }; class filesystem remount; class tcp_socket { create accept read setopt bind name_bind node_bind listen }; type boot_t; type auditd_log_t; type var_log_t; type syslogd_t; type syslogd_port_t; type port_t; type mount_t; type system_map_t; type inaddr_any_node_t; type unspec_node_t; role system_r; }; allow syslogd_t boot_t:dir search; allow syslogd_t auditd_log_t:dir search; allow syslogd_t auditd_log_t:file { getattr read }; allow syslogd_t self:tcp_socket { create accept read setopt bind listen }; allow syslogd_t syslogd_port_t:tcp_socket name_bind; allow syslogd_t port_t:tcp_socket name_bind; allow syslogd_t system_map_t:file { read getattr }; allow syslogd_t inaddr_any_node_t:tcp_socket node_bind; allow syslogd_t unspec_node_t:tcp_socket node_bind; ========== From rgerhards at hq.adiscon.com Tue Mar 11 20:05:13 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 11 Mar 2008 20:05:13 +0100 Subject: [rsyslog] rsyslog v3 and selinux Message-ID: <005c01c883aa$d9cecd2e$060013ac@intern.adiscon.com> I have absolutely no idea about selinux ... But: may it be that the fact that now plugins - separate libraries - are loaded by rsyslogd to do the job? rainer ----- Urspr?ngliche Nachricht ----- Von: "Johnny Tan" An: "rsyslog-users" Gesendet: 11.03.08 19:08 Betreff: [rsyslog] rsyslog v3 and selinux I took Fedora 8's rsyslog v2.0.2 SRPM and rebuilt it for v3.12.1 I'm using the same init script. When I used that init script (i.e., "/etc/init.d/rsyslog start") on v2.0.2, the only SELinux problem I had was the domain transition, as documented here: http://tanso.net/rsyslog/ However, when I rebuilt the RPM for v3.12.1, and used the same init script to run it, I get many SELinux errors. Eventually, I worked out all the things I had to allow for rsyslog v3.12.1 to run properly. It's enclosed below as a semanage module. Just curious if there was some change to rsyslog between versions 2 and 3 which would make rsyslog, even when running properly in domain syslogd_exec_t, to cause so many SELinux denials, including not being able to do TCP bind?? Thanks for any clues, johnn p.s. Here's the module that finally worked. Will document on wiki when all is done. ========== module rsyslog 1.0; require { class dir search; class file { getattr read write }; class filesystem remount; class tcp_socket { create accept read setopt bind name_bind node_bind listen }; type boot_t; type auditd_log_t; type var_log_t; type syslogd_t; type syslogd_port_t; type port_t; type mount_t; type system_map_t; type inaddr_any_node_t; type unspec_node_t; role system_r; }; allow syslogd_t boot_t:dir search; allow syslogd_t auditd_log_t:dir search; allow syslogd_t auditd_log_t:file { getattr read }; allow syslogd_t self:tcp_socket { create accept read setopt bind listen }; allow syslogd_t syslogd_port_t:tcp_socket name_bind; allow syslogd_t port_t:tcp_socket name_bind; allow syslogd_t system_map_t:file { read getattr }; allow syslogd_t inaddr_any_node_t:tcp_socket node_bind; allow syslogd_t unspec_node_t:tcp_socket node_bind; ========== _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Mar 11 21:45:00 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 11 Mar 2008 21:45:00 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Maurizio Rottin > Sent: Tuesday, March 11, 2008 1:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog with apache and per vhost log > > Hi all, > i set up an apache webserver wich logs with directives: > ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" > CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com > " "combined" > > then syslog forwards them with > local5.err @10.10.10.89 > local6.info @10.10.10.89 > and it works good! > > now on 10.10.10.89 i set up an rsyslog server with directives > $template > ApacheRemoteErr,%msg%"/var/log/rsyslog/Apache/%syslogtag%/Err.log" > local5.err -?ApacheRemoteErr > $template > ApacheRemoteCustom,%msg%"/var/log/rsyslog/Apache/%syslogtag%/C > ustom.log" > local6.info -?ApacheRemoteCustom > > the result is that i get these files in /var/log/rsyslog/Apache/ > ./www.mysite.com:/Custom.log > with customLog > > ./error_www.mysite.com:/Err.log > with ErroLog > > ./message/Err.log > with lines like > Mar 11 12:32:52 last message repeated 9 times > Mar 11 12:32:52 last message repeated 5 times > (Note that this is very annoying!!!) Add -e to rsyslogd startup and they go away ;) (this is part of sysklogd compatibility, -e stand for "log *e*very message") > > > Now this a simple way i have found to divide logs per VHost. > I don't know if there is a better way, but the real question > is different: > In the Custom or Error logs i have lines like: > Mar 11 12:32:53 php03 www.mysite.com: 10.10.10.2 - - > [11/Mar/2008:12:35:07 +0100] "GET /pagina.php?id=61 HTTP/1.1" 200 93 > "http://www.mysite.com/home.php" "Mozilla/5.0 (X11; U; Linux i686; > en-US; rv:1.7.13) Gecko/20060607" > > i don't want the "Mar 11 12:32:53 php03 www.mysite.com:" part > do i have to use a double template setup? one for the file and one for > the content? > This is caused by the default template, which is what most peiople expect in their syslog files. But, as you write, the soluton is quite easy, just use another template ;) > the template in the file seems to be this: > $template TraditionalFormat,%timegenerated% %HOSTNAME% > %syslogtag%%msg%\n" > but i need just > $template TheOneINeed,%msg%\n" > i want only the %msg%\n in my file > So, use local6.info -?ApacheRemoteCustom;TheOneINeed HTH, feel free to ask any follow-up questions. :) Rainer > > Thanks in advance > > -- > mr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From linuxweb at gmail.com Wed Mar 12 00:42:22 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Tue, 11 Mar 2008 19:42:22 -0400 Subject: [rsyslog] segmentation fault Message-ID: <47D718DE.90603@gmail.com> Ahh, so close yet so far. I've setup a central rsyslog server and am setting up my first client to point to it. I am going over stunnel. I set it up pretty much like it is in the secure rsyslog article. There are some existing logs on the client box. I load imfile.so module to convert those logs to syslog and send them over TCP/stunnel to my central logger. Everytime I try to start it, it dies within seconds. When I turn on debug mode, I notice the client gets a segmentation fault. I have pasted the tail end of the debug output here: http://pastebin.com/m9a83946 Is it just the sheer number of messages that is causing it to segfault, or is something else going on? johnn From linuxweb at gmail.com Wed Mar 12 03:09:00 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Tue, 11 Mar 2008 22:09:00 -0400 Subject: [rsyslog] segmentation fault In-Reply-To: <47D718DE.90603@gmail.com> References: <47D718DE.90603@gmail.com> Message-ID: <47D73B3C.1000509@gmail.com> After MUCH trial & error, I've found the problem. If there is a blank line in one of the log files being read by imfile, then rsyslog dies. I filed a bug: http://bugzilla.adiscon.com/show_bug.cgi?id=49 johnn From rgerhards at hq.adiscon.com Wed Mar 12 08:00:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 08:00:40 +0100 Subject: [rsyslog] segmentation fault In-Reply-To: <47D73B3C.1000509@gmail.com> References: <47D718DE.90603@gmail.com> <47D73B3C.1000509@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B64@grfint2.intern.adiscon.com> Thanks for finding out. Looks like we got a bug-fixing day today... Will update the bug tracker as things progress. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Wednesday, March 12, 2008 3:09 AM > To: rsyslog-users > Subject: Re: [rsyslog] segmentation fault > > After MUCH trial & error, I've found the problem. If there > is a blank line in one of the log files being read by > imfile, then rsyslog dies. > > I filed a bug: > http://bugzilla.adiscon.com/show_bug.cgi?id=49 > > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From maurizio.rottin at gmail.com Wed Mar 12 10:47:31 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Wed, 12 Mar 2008 10:47:31 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> Message-ID: 2008/3/11, Rainer Gerhards : > > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > Maurizio Rottin > > Sent: Tuesday, March 11, 2008 1:02 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog with apache and per vhost log > > > > Hi all, > > i set up an apache webserver wich logs with directives: > > ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" > > CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com > > " "combined" > > > > then syslog forwards them with > > local5.err @10.10.10.89 > > local6.info @10.10.10.89 > > and it works good! > > > > now on 10.10.10.89 i set up an rsyslog server with directives > > $template > > ApacheRemoteErr,%msg%"/var/log/rsyslog/Apache/%syslogtag%/Err.log" > > local5.err -?ApacheRemoteErr > > $template > > ApacheRemoteCustom,%msg%"/var/log/rsyslog/Apache/%syslogtag%/C > > ustom.log" > > local6.info -?ApacheRemoteCustom > > > > the result is that i get these files in /var/log/rsyslog/Apache/ > > ./www.mysite.com:/Custom.log > > with customLog > > > > ./error_www.mysite.com:/Err.log > > with ErroLog > > > > ./message/Err.log > > with lines like > > Mar 11 12:32:52 last message repeated 9 times > > Mar 11 12:32:52 last message repeated 5 times > > (Note that this is very annoying!!!) > > > Add -e to rsyslogd startup and they go away ;) (this is part of sysklogd > compatibility, -e stand for "log *e*very message") And i red about the -e options a lot of times...thank! > > > > > > > Now this a simple way i have found to divide logs per VHost. > > I don't know if there is a better way, but the real question > > is different: > > In the Custom or Error logs i have lines like: > > Mar 11 12:32:53 php03 www.mysite.com: 10.10.10.2 - - > > [11/Mar/2008:12:35:07 +0100] "GET /pagina.php?id=61 HTTP/1.1" 200 93 > > "http://www.mysite.com/home.php" "Mozilla/5.0 (X11; U; Linux i686; > > en-US; rv:1.7.13) Gecko/20060607" > > > > i don't want the "Mar 11 12:32:53 php03 www.mysite.com:" part > > do i have to use a double template setup? one for the file and one for > > the content? > > > > This is caused by the default template, which is what most peiople > expect in their syslog files. But, as you write, the soluton is quite > easy, just use another template ;) > > > > the template in the file seems to be this: > > $template TraditionalFormat,%timegenerated% %HOSTNAME% > > %syslogtag%%msg%\n" > > but i need just > > $template TheOneINeed,%msg%\n" > > i want only the %msg%\n in my file > > > > > So, use > > local6.info -?ApacheRemoteCustom;TheOneINeed works great man! Thanks! > HTH, feel free to ask any follow-up questions. :) > just one question, do you know any way to pass the website name in "/usr/bin/logger -p local5.err -t error_www.mysite.com", even using an other program...i know only logger... -- mr From rgerhards at hq.adiscon.com Wed Mar 12 10:50:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 10:50:06 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> [happy snip] > just one question, do you know any way to pass the website name in > "/usr/bin/logger -p local5.err -t error_www.mysite.com", even using an > other program...i know only logger... Well, you could use the imfile plugin to read the log files directly: http://www.rsyslog.com/doc-imfile.html Then, I'd set the site name as tag and so you can process it from the tag value. HTH Rainer From maurizio.rottin at gmail.com Wed Mar 12 11:24:08 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Wed, 12 Mar 2008 11:24:08 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> Message-ID: 2008/3/12, Rainer Gerhards : > [happy snip] > > > just one question, do you know any way to pass the website name in > > "/usr/bin/logger -p local5.err -t error_www.mysite.com", even using an > > other program...i know only logger... > > > Well, you could use the imfile plugin to read the log files directly: > > http://www.rsyslog.com/doc-imfile.html > > Then, I'd set the site name as tag and so you can process it from the > tag value. > > HTH > > Rainer > mmmmm...i'll check that better but at a glance it seems a heavy process with big logs. -- mr From rgerhards at hq.adiscon.com Wed Mar 12 11:25:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 11:25:53 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com> > mmmmm...i'll check that better but at a glance it seems a heavy > process with big logs. Sorry, I don't really understand what you mean ;) But the problem is that I probably do not really know how what you do with logger really works. It is part of the apache config, is this right? Maybe you could post a full config sample? Rainer From janfrode at tanso.net Wed Mar 12 12:39:04 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 12 Mar 2008 12:39:04 +0100 Subject: [rsyslog] rsyslog v3 and selinux References: <47D6CA86.6060501@gmail.com> Message-ID: On 2008-03-11, Johnny Tan wrote: > > However, when I rebuilt the RPM for v3.12.1, and used the > same init script to run it, I get many SELinux errors. > Eventually, I worked out all the things I had to allow for > rsyslog v3.12.1 to run properly. It's enclosed below as a > semanage module. These seems very strange: > allow syslogd_t boot_t:dir search; > allow syslogd_t system_map_t:file { read getattr }; Why does it want to read /boot/System.map* ? > allow syslogd_t auditd_log_t:dir search; > allow syslogd_t auditd_log_t:file { getattr read }; And why read /var/log/audit/* ? -jf From rgerhards at hq.adiscon.com Wed Mar 12 13:48:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 13:48:42 +0100 Subject: [rsyslog] rsyslog v3 and selinux In-Reply-To: References: <47D6CA86.6060501@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B6D@grfint2.intern.adiscon.com> That's part of the klog(d) code I inherited from sysklogd. I've never questioned if there is another way to do it. It seems to use these to obtain kernel symbols. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jan-Frode Myklebust > Sent: Wednesday, March 12, 2008 12:39 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] rsyslog v3 and selinux > > On 2008-03-11, Johnny Tan wrote: > > > > However, when I rebuilt the RPM for v3.12.1, and used the > > same init script to run it, I get many SELinux errors. > > Eventually, I worked out all the things I had to allow for > > rsyslog v3.12.1 to run properly. It's enclosed below as a > > semanage module. > > These seems very strange: > > > allow syslogd_t boot_t:dir search; > > allow syslogd_t system_map_t:file { read getattr }; > > Why does it want to read /boot/System.map* ? > > > allow syslogd_t auditd_log_t:dir search; > > allow syslogd_t auditd_log_t:file { getattr read }; > > And why read /var/log/audit/* ? > > > > -jf > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From maurizio.rottin at gmail.com Wed Mar 12 14:37:25 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Wed, 12 Mar 2008 14:37:25 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com> Message-ID: 2008/3/12, Rainer Gerhards : > > mmmmm...i'll check that better but at a glance it seems a heavy > > process with big logs. > > > Sorry, I don't really understand what you mean ;) But the problem is > that I probably do not really know how what you do with logger really > works. It is part of the apache config, is this right? Maybe you could > post a full config sample? > but i did... this is the log part in the directive ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com " "combined" in practice i pass the ErrorLog and CustomLog to /usr/bin/logger with facility "-p local*.xxx" with tag "-t www.mysite.com" then i can use the tag to split logs by website in the remote rsyslog machine ;) -- mr From rgerhards at hq.adiscon.com Wed Mar 12 14:44:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 14:44:31 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B70@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Maurizio Rottin > Sent: Wednesday, March 12, 2008 2:37 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog with apache and per vhost log > > 2008/3/12, Rainer Gerhards : > > > mmmmm...i'll check that better but at a glance it seems a heavy > > > process with big logs. > > > > > > Sorry, I don't really understand what you mean ;) But the problem is > > that I probably do not really know how what you do with logger > really > > works. It is part of the apache config, is this right? Maybe you > could > > post a full config sample? > > > but i did... > this is the log part in the directive > ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" > CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com " > "combined" > > in practice i pass the ErrorLog and CustomLog to /usr/bin/logger with > facility "-p local*.xxx" with tag "-t www.mysite.com" > > then i can use the tag to split logs by website in the remote rsyslog > machine ;) Ah, OK, I didn't realize this was actual Apache config. So your original question > just one question, do you know any way to pass the website name in > "/usr/bin/logger -p local5.err -t error_www.mysite.com", even using an > other program...i know only logger... was how you could use the same apache config command for each log, so that you don't need to have separate apache config statements for each virtual server? Please bear with me if I am getting it wrong, I probably have too much rsyslog on my mind. I am trying to help but I don't get a grip at what the real problem is... Rainer From linuxweb at gmail.com Wed Mar 12 15:08:17 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 10:08:17 -0400 Subject: [rsyslog] rsyslog v3 and selinux In-Reply-To: References: <47D6CA86.6060501@gmail.com> Message-ID: <47D7E3D1.9010309@gmail.com> Jan-Frode Myklebust wrote: >> allow syslogd_t auditd_log_t:dir search; >> allow syslogd_t auditd_log_t:file { getattr read }; > > And why read /var/log/audit/* ? This is custom. I prefer to install auditd and have the SELinux logs separated out from /var/log/messages. Then I use imfile to convert the audit logs to syslog in order to send it over to the central logger. Yes, it's a waste of a syslog "local" facility, but I'm not aware of any other way unless I remove auditd and have those go back into /var/log/messages. johnn From linuxweb at gmail.com Wed Mar 12 18:16:56 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 13:16:56 -0400 Subject: [rsyslog] actionqueue in front of tcp forward Message-ID: <47D81008.3090302@gmail.com> I wanted to setup my rsyslog clients such that when the rsyslog server is down and then comes back up, the clients will resume forwarding messages to it. I'm running into this same bug as another user: http://bugzilla.adiscon.com/show_bug.cgi?id=45 According to that bug, the fix is in 3.12.1, which is what I'm using on client and server. But I still can't seem to get it to work. Prior to doing this, I simply had this line: *.* @@127.0.0.1:61514 Am I supposed to replace that line entirely with the ActionQueue stanza, such as this one (the == is for clarity, not actually part of my config): == $WorkDirectory /tmp/rsyslogwork $ActionQueueType LinkedList $ActionQueueFileName dqueue $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 $ActionResumeInterval 30 $ActionResumeRetryCount -1 # infinite retries on insert failure & @@127.0.0.1:61514 == If I do, rsyslog dies immediately on restart. So I changed it so look like this instead: == *.* @@127.0.0.1:61514 $WorkDirectory /tmp/rsyslogwork $ActionQueueType LinkedList $ActionQueueFileName dqueue $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 $ActionResumeInterval 10 $ActionResumeRetryCount -1 & @@127.0.0.1:61514 == This allows rsyslog to work again, but then it doesn't actually do any resume. Just as the OP states in the bug, if the rsyslog server is stopped, the client simply logs messages locally. When the rsyslog server is started again, the client once again logs locally as well as to the server. But all the messages from the client between stop and start on the server are not forwarded. I checked action.c and it does have the code-fix listed in the forum post tied to the bug. I also understand that one message WILL be lost, which is a TCP syslog protocol problem. So I sent multiple messages, none of them came through. johnn From rgerhards at hq.adiscon.com Wed Mar 12 18:22:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 18:22:12 +0100 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <47D81008.3090302@gmail.com> References: <47D81008.3090302@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B81@grfint2.intern.adiscon.com> I need to check for the restart problem (currently working on something else). Please also follow this bug: http://bugzilla.adiscon.com/show_bug.cgi?id=48 This is the "one message lost" problem, and I have created at least some cure for it today (to be verified). Will check your bug report when I verify it. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Wednesday, March 12, 2008 6:17 PM > To: rsyslog-users > Subject: [rsyslog] actionqueue in front of tcp forward > > I wanted to setup my rsyslog clients such that when the > rsyslog server is down and then comes back up, the clients > will resume forwarding messages to it. > > I'm running into this same bug as another user: > http://bugzilla.adiscon.com/show_bug.cgi?id=45 > > According to that bug, the fix is in 3.12.1, which is what > I'm using on client and server. > > > But I still can't seem to get it to work. Prior to doing > this, I simply had this line: > > *.* @@127.0.0.1:61514 > > > > Am I supposed to replace that line entirely with the > ActionQueue stanza, such as this one (the == is for clarity, > not actually part of my config): > > == > $WorkDirectory /tmp/rsyslogwork > $ActionQueueType LinkedList > $ActionQueueFileName dqueue > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 7 > $ActionResumeInterval 30 > $ActionResumeRetryCount -1 # infinite retries on insert failure > & @@127.0.0.1:61514 > == > > If I do, rsyslog dies immediately on restart. So I changed > it so look like this instead: > > == > *.* @@127.0.0.1:61514 > $WorkDirectory /tmp/rsyslogwork > $ActionQueueType LinkedList > $ActionQueueFileName dqueue > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 7 > $ActionResumeInterval 10 > $ActionResumeRetryCount -1 > & @@127.0.0.1:61514 > == > > This allows rsyslog to work again, but then it doesn't > actually do any resume. Just as the OP states in the bug, if > the rsyslog server is stopped, the client simply logs > messages locally. When the rsyslog server is started again, > the client once again logs locally as well as to the server. > But all the messages from the client between stop and start > on the server are not forwarded. > > I checked action.c and it does have the code-fix listed in > the forum post tied to the bug. > > I also understand that one message WILL be lost, which is a > TCP syslog protocol problem. So I sent multiple messages, > none of them came through. > > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Mar 12 18:32:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 18:32:35 +0100 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <47D81008.3090302@gmail.com> References: <47D81008.3090302@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B82@grfint2.intern.adiscon.com> Argh, some config parsing seems to be wrong. There is a problem with the config file. The right config format is this: == $WorkDirectory /tmp/rsyslogwork $ActionQueueType LinkedList $ActionQueueFileName dqueue $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 $ActionResumeInterval 10 $ActionResumeRetryCount -1 *.* @@127.0.0.1:61514 == & in column one is to chain multiple actions together, but it can never be at the start of the chain. Did you take it from some rsyslog documentation? If so and you still know which, please let me know, I'd like to fix that. If I think correctly, the queued action is now chained to the regular action, and probably never executed because the regular action fails ... or something along these lines. Please try the config above, I think it will work (at least it does in my lab). And of course I'll look into the config parser and co ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Wednesday, March 12, 2008 6:17 PM > To: rsyslog-users > Subject: [rsyslog] actionqueue in front of tcp forward > > I wanted to setup my rsyslog clients such that when the > rsyslog server is down and then comes back up, the clients > will resume forwarding messages to it. > > I'm running into this same bug as another user: > http://bugzilla.adiscon.com/show_bug.cgi?id=45 > > According to that bug, the fix is in 3.12.1, which is what > I'm using on client and server. > > > But I still can't seem to get it to work. Prior to doing > this, I simply had this line: > > *.* @@127.0.0.1:61514 > > > > Am I supposed to replace that line entirely with the > ActionQueue stanza, such as this one (the == is for clarity, > not actually part of my config): > > == > $WorkDirectory /tmp/rsyslogwork > $ActionQueueType LinkedList > $ActionQueueFileName dqueue > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 7 > $ActionResumeInterval 30 > $ActionResumeRetryCount -1 # infinite retries on insert failure > & @@127.0.0.1:61514 > == > > If I do, rsyslog dies immediately on restart. So I changed > it so look like this instead: > > == > *.* @@127.0.0.1:61514 > $WorkDirectory /tmp/rsyslogwork > $ActionQueueType LinkedList > $ActionQueueFileName dqueue > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 7 > $ActionResumeInterval 10 > $ActionResumeRetryCount -1 > & @@127.0.0.1:61514 > == > > This allows rsyslog to work again, but then it doesn't > actually do any resume. Just as the OP states in the bug, if > the rsyslog server is stopped, the client simply logs > messages locally. When the rsyslog server is started again, > the client once again logs locally as well as to the server. > But all the messages from the client between stop and start > on the server are not forwarded. > > I checked action.c and it does have the code-fix listed in > the forum post tied to the bug. > > I also understand that one message WILL be lost, which is a > TCP syslog protocol problem. So I sent multiple messages, > none of them came through. > > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Mar 12 18:50:01 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 18:50:01 +0100 Subject: [rsyslog] rsyslog 2.0.3 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B83@grfint2.intern.adiscon.com> Hi all, Rsyslog 2.0.3, a release of the stable branch, has been released today. It is a purely bug-fixing release, solving mostly bugs that occur either seldom or due to mistakes in the configuration file. It fixes some potential segfaults in those cases. Rsyslog 2.0.3 is a recommended update for all users of the rsyslog v2 branch. Download: http://www.rsyslog.com/Downloads-req-getit-lid-85.phtml Changelog: http://www.rsyslog.com/Article189.phtml As always, feedback is appreciated. Rainer Gerhards From linuxweb at gmail.com Wed Mar 12 19:27:38 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 14:27:38 -0400 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B82@grfint2.intern.adiscon.com> References: <47D81008.3090302@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B82@grfint2.intern.adiscon.com> Message-ID: <47D8209A.3070908@gmail.com> Rainer Gerhards wrote: > & in column one is to chain multiple actions together, but it can never > be at the start of the chain. Did you take it from some rsyslog > documentation? If so and you still know which, please let me know, I'd > like to fix that. No, I copied it from the bug report I listed. The OP must've had a regular action that wasn't listed. Anyway, I fixed it, restarted client rsyslog, but the behavior is the same. I don't need any special config on the server side, right? Well, if it helps, here's the top of my client /etc/rsyslog.conf, followed by the top of my server /etc/rsyslog.conf: == client /etc/rsyslog.conf == # Modules $ModLoad imuxsock.so $ModLoad imklog.so $ModLoad imfile.so $WorkDirectory /tmp/rsyslogwork $ActionQueueType LinkedList $ActionQueueFileName dqueue $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 $ActionResumeInterval 10 $ActionResumeRetryCount -1 *.* @@127.0.0.1:61514 # Standard syslog configuration (for local logging) [snip] == == server /etc/rsyslog.conf == # Modules $ModLoad imudp.so $ModLoad imtcp.so $ModLoad imuxsock.so $ModLoad imklog.so $ModLoad imfile.so # TCP port $InputTCPServerRun 61514 # Global Directives $AllowedSender UDP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24 $AllowedSender TCP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24 $DirOwner root $DirGroup root $DirCreateMode 0755 # Templates [snip] # Standard syslog configuration [snip] == From linuxweb at gmail.com Wed Mar 12 19:32:15 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 14:32:15 -0400 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B82@grfint2.intern.adiscon.com> References: <47D81008.3090302@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B82@grfint2.intern.adiscon.com> Message-ID: <47D821AF.9030107@gmail.com> Rainer Gerhards wrote: > $WorkDirectory /tmp/rsyslogwork Also, I should also point out that I NEVER see anything show up in my WorkDirectory. That directory above is owned by root:root and mode 0755. I turned off SELinux. Nothing EVER shows up there. johnn From milton at calnek.com Wed Mar 12 19:22:10 2008 From: milton at calnek.com (Milton Calnek) Date: Wed, 12 Mar 2008 12:22:10 -0600 Subject: [rsyslog] rsyslog 2.0.3 released In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B83@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308B83@grfint2.intern.adiscon.com> Message-ID: <47D81F52.2050701@calnek.com> The link in the web site does not work. The requested URL /rsyslog/rsyslog-2.0.23tar.gz was not found on this server. Thanks. Rainer Gerhards wrote: > Hi all, > > Rsyslog 2.0.3, a release of the stable branch, has been released today. > It is a purely bug-fixing release, solving mostly bugs that occur either > seldom or due to mistakes in the configuration file. It fixes some > potential segfaults in those cases. Rsyslog 2.0.3 is a recommended > update for all users of the rsyslog v2 branch. > > Download: > http://www.rsyslog.com/Downloads-req-getit-lid-85.phtml > > Changelog: > http://www.rsyslog.com/Article189.phtml > > As always, feedback is appreciated. > > Rainer Gerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > -- Milton Calnek BSc, A/Slt(Ret.) milton at calnek.com 306-717-8737 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From rgerhards at hq.adiscon.com Wed Mar 12 19:48:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 19:48:45 +0100 Subject: [rsyslog] rsyslog 2.0.3 released Message-ID: <006101c88471$b85b588f$060013ac@intern.adiscon.com> Oops, as a quick fix http://download.rsyslog.com/rsyslog/rsyslog-2.0.3.tar.gz If i have not mistyped again ;) rainer ----- Urspr?ngliche Nachricht ----- Von: "Milton Calnek" An: "rsyslog-users" Gesendet: 12.03.08 19:37 Betreff: Re: [rsyslog] rsyslog 2.0.3 released The link in the web site does not work. The requested URL /rsyslog/rsyslog-2.0.23tar.gz was not found on this server. Thanks. Rainer Gerhards wrote: > Hi all, > > Rsyslog 2.0.3, a release of the stable branch, has been released today. > It is a purely bug-fixing release, solving mostly bugs that occur either > seldom or due to mistakes in the configuration file. It fixes some > potential segfaults in those cases. Rsyslog 2.0.3 is a recommended > update for all users of the rsyslog v2 branch. > > Download: > http://www.rsyslog.com/Downloads-req-getit-lid-85.phtml > > Changelog: > http://www.rsyslog.com/Article189.phtml > > As always, feedback is appreciated. > > Rainer Gerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > -- Milton Calnek BSc, A/Slt(Ret.) milton at calnek.com 306-717-8737 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Mar 12 19:50:55 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 19:50:55 +0100 Subject: [rsyslog] actionqueue in front of tcp forward Message-ID: <006201c88472$0590e0ed$060013ac@intern.adiscon.com> That's fine, files are only created if the mem queue is filling up. By default thats after 8,000 msgs queued. And, no server needs no special config. More later, i am on a pda and can not really check out anything right now ;) rainer ----- Urspr?ngliche Nachricht ----- Von: "Johnny Tan" An: "rsyslog-users" Gesendet: 12.03.08 19:32 Betreff: Re: [rsyslog] actionqueue in front of tcp forward Rainer Gerhards wrote: > $WorkDirectory /tmp/rsyslogwork Also, I should also point out that I NEVER see anything show up in my WorkDirectory. That directory above is owned by root:root and mode 0755. I turned off SELinux. Nothing EVER shows up there. johnn _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Mar 12 19:52:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 19:52:43 +0100 Subject: [rsyslog] actionqueue in front of tcp forward Message-ID: <006301c88472$46360ab0$060013ac@intern.adiscon.com> Can you send me a ful debug output from the sender when it fails (maybe via mail) - if thats possible... rainer ----- Urspr?ngliche Nachricht ----- Von: "Johnny Tan" An: "rsyslog-users" Gesendet: 12.03.08 19:28 Betreff: Re: [rsyslog] actionqueue in front of tcp forward Rainer Gerhards wrote: > & in column one is to chain multiple actions together, but it can never > be at the start of the chain. Did you take it from some rsyslog > documentation? If so and you still know which, please let me know, I'd > like to fix that. No, I copied it from the bug report I listed. The OP must've had a regular action that wasn't listed. Anyway, I fixed it, restarted client rsyslog, but the behavior is the same. I don't need any special config on the server side, right? Well, if it helps, here's the top of my client /etc/rsyslog.conf, followed by the top of my server /etc/rsyslog.conf: == client /etc/rsyslog.conf == # Modules $ModLoad imuxsock.so $ModLoad imklog.so $ModLoad imfile.so $WorkDirectory /tmp/rsyslogwork $ActionQueueType LinkedList $ActionQueueFileName dqueue $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 $ActionResumeInterval 10 $ActionResumeRetryCount -1 *.* @@127.0.0.1:61514 # Standard syslog configuration (for local logging) [snip] == == server /etc/rsyslog.conf == # Modules $ModLoad imudp.so $ModLoad imtcp.so $ModLoad imuxsock.so $ModLoad imklog.so $ModLoad imfile.so # TCP port $InputTCPServerRun 61514 # Global Directives $AllowedSender UDP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24 $AllowedSender TCP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24 $DirOwner root $DirGroup root $DirCreateMode 0755 # Templates [snip] # Standard syslog configuration [snip] == _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From linuxweb at gmail.com Wed Mar 12 20:07:34 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 15:07:34 -0400 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <006301c88472$46360ab0$060013ac@intern.adiscon.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> Message-ID: <47D829F6.8070404@gmail.com> Rainer Gerhards wrote: > Can you send me a ful debug output from the sender when > it fails (maybe via mail) - if thats possible... I sent it via YouSendIt. Thanks for looking into this! johnn From rgerhards at hq.adiscon.com Wed Mar 12 21:40:22 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 21:40:22 +0100 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <47D829F6.8070404@gmail.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> I have received the file and had a chance to look at it. It's quite confusing. The send fails, but each retry succeeds... It looks like you use stunnel. I currently doubt that stunnel accepts each send and only reports the error after it can not connect to the remote side. So in fact we continously run into the situation that exactly *that* one message is lost. But I am still puzzled. Could you, for a test, run the same without stunnel and tell me if the problem persists or goes away? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Wednesday, March 12, 2008 8:08 PM > To: rsyslog-users > Subject: Re: [rsyslog] actionqueue in front of tcp forward > > Rainer Gerhards wrote: > > Can you send me a ful debug output from the sender when > > it fails (maybe via mail) - if thats possible... > > I sent it via YouSendIt. > Thanks for looking into this! > > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From linuxweb at gmail.com Wed Mar 12 22:05:54 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 17:05:54 -0400 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> Message-ID: <47D845B2.30701@gmail.com> Rainer Gerhards wrote: > I have received the file and had a chance to look at it. It's quite > confusing. The send fails, but each retry succeeds... It looks like you > use stunnel. I currently doubt that stunnel accepts each send and only > reports the error after it can not connect to the remote side. So in > fact we continously run into the situation that exactly *that* one > message is lost. But I am still puzzled. Could you, for a test, run the > same without stunnel and tell me if the problem persists or goes away? Yes, I had the same thought about stunnel being the problem and was running some tests. It works! I now only lose the one message (known problem). But if I go over stunnel, then I lose all messages during the rsyslog server downtime. I can do without the stunnel for now. Thanks once again! johnn From rgerhards at hq.adiscon.com Wed Mar 12 22:20:15 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 12 Mar 2008 22:20:15 +0100 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <47D845B2.30701@gmail.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> <47D845B2.30701@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B85@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Johnny Tan > Sent: Wednesday, March 12, 2008 10:06 PM > To: rsyslog-users > Subject: Re: [rsyslog] actionqueue in front of tcp forward > > Rainer Gerhards wrote: > > I have received the file and had a chance to look at it. It's quite > > confusing. The send fails, but each retry succeeds... It > looks like you > > use stunnel. I currently doubt that stunnel accepts each > send and only > > reports the error after it can not connect to the remote side. So in > > fact we continously run into the situation that exactly *that* one > > message is lost. But I am still puzzled. Could you, for a > test, run the > > same without stunnel and tell me if the problem persists or > goes away? > > Yes, I had the same thought about stunnel being the problem > and was running some tests. > > It works! I now only lose the one message (known problem). > But if I go over stunnel, then I lose all messages during > the rsyslog server downtime. I have the strong feeling that it is time to do something against this plain old ack-less syslog tcp protocol... Maybe I add a half-duplex mode for starters. That's low, but quick to implement and ultra-reliable. I'll also see that I get more serious with RFC 3195 re-enabling. I've already done some basic thinking in regard to 3195 and the new syslog engine and doing it ultra-reliable will require a little bit of work. So there won't be an immediate cure - but defenitely the right route to take. How about half-duplex mode? Would that work for you? It means that each message must be acked before the next one is sent, so tcp's streaming features will almost be disabled. I'd expect a drop to at most 50% (more probable 40%) of the performance compared to what we currently run (half-duplex would obviously need to be an option...). So it would be a large performance hit. > I can do without the stunnel for now. For encryption, you could also look into the GSSAPI modules. It's contributed code, and I currently unfortunately have limited insight into it. But varmojfekoj, the contributor, has done a great job. > Thanks once again! I am very intersted in real-life experience. After all, the engine is still quite new. So I need feedback from the field to make it ultra-solid - a lab is a lab, is a lab ;) So keep the thoughts flowing. A side-note: we are rewriting phpLogCon, the web interface to syslog data. Any chance you happen to have some interest in that? ;) Rainer > johnn > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From linuxweb at gmail.com Wed Mar 12 23:10:22 2008 From: linuxweb at gmail.com (Johnny Tan) Date: Wed, 12 Mar 2008 18:10:22 -0400 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B85@grfint2.intern.adiscon.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> <47D845B2.30701@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B85@grfint2.intern.adiscon.com> Message-ID: <47D854CE.8020209@gmail.com> Rainer Gerhards wrote: > I have the strong feeling that it is time to do something against this > plain old ack-less syslog tcp protocol... Maybe I add a half-duplex mode > for starters. That's low, but quick to implement and ultra-reliable. > I'll also see that I get more serious with RFC 3195 re-enabling. I've > already done some basic thinking in regard to 3195 and the new syslog > engine and doing it ultra-reliable will require a little bit of work. So > there won't be an immediate cure - but defenitely the right route to > take. > > How about half-duplex mode? Would that work for you? It means that each > message must be acked before the next one is sent, so tcp's streaming > features will almost be disabled. I'd expect a drop to at most 50% (more > probable 40%) of the performance compared to what we currently run > (half-duplex would obviously need to be an option...). So it would be a > large performance hit. Just curious: do either of the above solutions (3195 and half-duplex) resolve the issue of losing all messages when the server goes down and you are going over stunnel? It sounds like the half-duplex does, I don't know enough about the 3195 to know. In answer to your question, I probably don't have as heavy a load as some folks out there, so it's easy for me to say, yes, half-duplex works for me, at least for the moment. >> I can do without the stunnel for now. > > For encryption, you could also look into the GSSAPI modules. It's > contributed code, and I currently unfortunately have limited insight > into it. But varmojfekoj, the contributor, has done a great job. Thanks! I'll have to look more closely at this in a staging environment down the road, as I can't afford potentially running into another stumbling block at the moment. > A side-note: we are rewriting phpLogCon, the web interface to syslog > data. Any chance you happen to have some interest in that? ;) It'll be a couple weeks before I really start looking into this, but yes, I was in fact planning to setup phpLogCon, too. johnn From rgerhards at hq.adiscon.com Thu Mar 13 08:11:58 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 13 Mar 2008 08:11:58 +0100 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <47D854CE.8020209@gmail.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> <47D845B2.30701@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B85@grfint2.intern.adiscon.com> <47D854CE.8020209@gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B86@grfint2.intern.adiscon.com> > Rainer Gerhards wrote: > > I have the strong feeling that it is time to do something against > this > > plain old ack-less syslog tcp protocol... Maybe I add a half-duplex > mode > > for starters. That's low, but quick to implement and ultra-reliable. > > I'll also see that I get more serious with RFC 3195 re-enabling. I've > > already done some basic thinking in regard to 3195 and the new syslog > > engine and doing it ultra-reliable will require a little bit of work. > So > > there won't be an immediate cure - but defenitely the right route to > > take. > > > > How about half-duplex mode? Would that work for you? It means that > each > > message must be acked before the next one is sent, so tcp's streaming > > features will almost be disabled. I'd expect a drop to at most 50% > (more > > probable 40%) of the performance compared to what we currently run > > (half-duplex would obviously need to be an option...). So it would be > a > > large performance hit. > > Just curious: do either of the above solutions (3195 and > half-duplex) resolve the issue of losing all messages when > the server goes down and you are going over stunnel? > > It sounds like the half-duplex does, I don't know enough > about the 3195 to know. > > > In answer to your question, I probably don't have as heavy a > load as some folks out there, so it's easy for me to say, > yes, half-duplex works for me, at least for the moment. I had a somewhat sleepless night, which is good news because it usually results in good ideas ;) This problem really bugged me, but I finally made up my mind. Instead of wasting time on fixing broken plain tcp syslog transport mode (e.g. by implementing half-duplex, which isn't standard anyhow), I'll do "the right thing". I thought rfc 3195 is the right thing. But it carries a lot of overhead that I really don't need. And, most importantly, any standard additions takes ages to go through the IETF (I know what I am talking about, have finally succeeded to get a better syslog rfc though it in "just" 4 (or 5?) years -- and it is still waiting to be published...). So instead of adding on any of these existing protocols, I'll do a new, lightweight but capable protocol specifically designed to solve the shortcomings we currently experience. Please welcome RELP, the "reliable event logging protocol" (name based on the quite successful selp [simple event logging protocol] effort: http://www.monitorware.com/en/workinprogress/selp.txt). Relp will evolve over time. I hope to do something useful relatively soon, and it will be extended as the project progresses. The ultimate goal is to have a good, very reliable, protocol for rsyslog-to-rsyslog communications. I'll don't care about the outside world, so I can do the best of breed solution. For the rest of the world, rsyslog will of course continue to support plain tcp syslog and will get better support for rfc 3195. But if you wanna go hardcore on high-reliability, high-performance event logging, relp will be your choice. Technically, I'll split this off into rsyslog relp input and output plugins AND an independent librelp, which provides core protocol functionality (just in case somebody else wants to support it in the long term, so this will not be tied to rsyslog). > > >> I can do without the stunnel for now. > > > > For encryption, you could also look into the GSSAPI modules. It's > > contributed code, and I currently unfortunately have limited insight > > into it. But varmojfekoj, the contributor, has done a great job. > > Thanks! I'll have to look more closely at this in a staging > environment down the road, as I can't afford potentially > running into another stumbling block at the moment. > > > > A side-note: we are rewriting phpLogCon, the web interface to syslog > > data. Any chance you happen to have some interest in that? ;) > > It'll be a couple weeks before I really start looking into > this, but yes, I was in fact planning to setup phpLogCon, too. Excellent. Please keep an eye on its beta announcements - various stages of v2 will probably be available in a few weeks and I promise it will be much better than what is currently there. In fact, it tries to become the best-ever log web viewer. If there are some things you'd like to see in such a viewer, it would be great to hear your opinion. Rainer From maurizio.rottin at gmail.com Thu Mar 13 09:03:16 2008 From: maurizio.rottin at gmail.com (Maurizio Rottin) Date: Thu, 13 Mar 2008 09:03:16 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B70@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308B70@grfint2.intern.adiscon.com> Message-ID: 2008/3/12, Rainer Gerhards : > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Maurizio Rottin > > > Sent: Wednesday, March 12, 2008 2:37 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog with apache and per vhost log > > > > 2008/3/12, Rainer Gerhards : > > > > mmmmm...i'll check that better but at a glance it seems a heavy > > > > process with big logs. > > > > > > > > > Sorry, I don't really understand what you mean ;) But the problem is > > > that I probably do not really know how what you do with logger > > really > > > works. It is part of the apache config, is this right? Maybe you > > could > > > post a full config sample? > > > > > but i did... > > this is the log part in the directive > > ErrorLog "|/usr/bin/logger -p local5.err -t error_www.mysite.com" > > CustomLog "|/usr/bin/logger -p local6.info -t www.mysite.com " > > "combined" > > > > in practice i pass the ErrorLog and CustomLog to /usr/bin/logger with > > facility "-p local*.xxx" with tag "-t www.mysite.com" > > > > then i can use the tag to split logs by website in the remote rsyslog > > machine ;) > > > Ah, OK, I didn't realize this was actual Apache config. So your original > question > > > > just one question, do you know any way to pass the website name in > > "/usr/bin/logger -p local5.err -t error_www.mysite.com", even using an > > other program...i know only logger... > > > was how you could use the same apache config command for each log, so > that you don't need to have separate apache config statements for each > virtual server? > > Please bear with me if I am getting it wrong, I probably have too much > rsyslog on my mind. I am trying to help but I don't get a grip at what > the real problem is... > > Not really...i don't want to use logger, it becomes a heavy process under high traffic load....but this should be a question for the apache ML... -- mr From rgerhards at hq.adiscon.com Thu Mar 13 09:06:09 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 13 Mar 2008 09:06:09 +0100 Subject: [rsyslog] rsyslog with apache and per vhost log In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308B62@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B6A@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B6B@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308B70@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308B87@grfint2.intern.adiscon.com> [sniiiiip] > Not really...i don't want to use logger, it becomes a heavy process > under high traffic load....but this should be a question for the > apache ML... OK, I don't know what makes it a heavy process... anyhow... If you come up with an explanation or what would be less resource-intense (from an apache point of view), please let me know. Maybe I can add a simple little tool to do the job. But obviously I need to know what's wrong with logger before even thinking about that ;) Rainer From Gerrard.Geldenhuis at datacash.com Thu Mar 13 10:04:11 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Thu, 13 Mar 2008 09:04:11 -0000 Subject: [rsyslog] actionqueue in front of tcp forward In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308B86@grfint2.intern.adiscon.com> References: <006301c88472$46360ab0$060013ac@intern.adiscon.com> <47D829F6.8070404@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B84@grfint2.intern.adiscon.com> <47D845B2.30701@gmail.com><577465F99B41C842AAFBE9ED71E70ABA308B85@grfint2.intern.adiscon.com><47D854CE.8020209@gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308B86@grfint2.intern.adiscon.com> Message-ID: > > I had a somewhat sleepless night, which is good news because it usually > results in good ideas ;) This problem really bugged me, but I finally > made up my mind. Instead of wasting time on fixing broken plain tcp > syslog transport mode (e.g. by implementing half-duplex, which isn't > standard anyhow), I'll do "the right thing". I thought rfc 3195 is the > right thing. But it carries a lot of overhead that I really don't need. > And, most importantly, any standard additions takes ages to go through > the IETF (I know what I am talking about, have finally succeeded to get > a better syslog rfc tho