From paul.ruiz at gmail.com Fri Oct 1 23:07:40 2010 From: paul.ruiz at gmail.com (Paul Ruiz) Date: Fri, 1 Oct 2010 14:07:40 -0700 Subject: [rsyslog] imfile duplicate events Message-ID: I'm stumped on a problem involving log consolidation and imfile creating duplicate events in the log and believe it's a bug. I have rsyslog using imfile to forward web access logs to a central server. Each of the source log files is 2.5 to 4.5 GB depending on the traffic that day. The destination ends up being 4 or 5 times that size. The duplicate events occur at very large intervals in the log. Is this possible a 32bit offset problem with files larger than 2GB? I'm doing the same thing with much smaller files and don't have this problem. In all cases rsyslog is restarted every night but at different times to account for log rotation, client and server. Client is Rsyslog 4.4.2 and collector is Rsyslog 5.5.6 both running on Ubuntu linux 8.04.4 x86_64. I stopped the client rsyslog running the imfile module at an hour interval and copied the contents of the control file End . End . Example log lines prepended with lineNumber:byteOffset from grep -xbnF for entire line: 3443:1134236:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 2239059:821710731:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 4976860:1834241112:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 9546150:3535005742:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" >>Relevant rsyslog.conf on client<< $template commonForwardFormat,"<%PRI%>%timereported:::date-rfc3339% %fromhost% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" $ModLoad imfile $InputFileName /opt/wp/log/nginx/wpn_rails_access.log $InputFilePollInterval 5 $InputFileStateFile wpn_rails_access-offset_state-rt_weblog $InputFileFacility local6 $InputFileTag NginxAccess: $InputRunFileMonitor if $programname == 'NginxAccess' then @@syslog1:10518;commonForwardFormat & ~ >>Relevant rsyslog.conf on syslog collector<< # Remove the rsyslog appended timestamp host and tag $template wpnrailsAccessLogFixup,"%msg:15:$:%\n" $template DYNwpnrailsAccess,"/var/log/remote/wpnrails-access/%fromhost%/%fromhost%-access-%timestamp:1:10:date-rfc3339%.log" $template DYNwpnrailsAccessCombined,"/var/log/remote/wpnrails-access-combined/access-combined-%timestamp:1:10:date-rfc3339%.log" $RuleSet wpnrailsAccess $RulesetCreateMainQueue on *.* ?DYNwpnrailsAccess;wpnrailsAccessLogFixup *.* ?DYNwpnrailsAccessCombined;wpnrailsAccessLogFixup $InputTCPServerBindRuleset wpnrailsAccess $InputTCPServerRun 10518 From m.stegmueller at in-put.de Mon Oct 4 12:51:45 2010 From: m.stegmueller at in-put.de (Markus Stegmueller) Date: Mon, 04 Oct 2010 12:51:45 +0200 Subject: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng Message-ID: <4CA9B1C1.6080504@in-put.de> Hello, I am trying to convert my syslog-ng Filter to a working rsyslog Filter Configuration. Here is my syslog-ng Filter: destination d_antispam { file("/var/log/mail_antispam"); }; filter f_antispam { facility(mail) and not match("fax at example.de") and not match("localhost.example.de") and not match("root at example.de") and not match("maildummy@") and not match("nobody at example") and not match("spamdrop") and not match("NOQUEUE") and not match("reject:") and ((match("to=<") and match("status=sent") and not match("relay=procmail") and not match("relay=/var/lib/imap/socket/lmtp") and not match("relay=local") and not match("relay=10.0.50.2")) or match("from=<.*example\.de>")); }; log { source(src); filter(f_antispam); destination(d_antispam); }; This Filter is to get all the mails, that were send from example.de users, so I can re-use them to make a whitelist. I tried to make an expression Based Filter: if $syslogfacility-text == 'mail' and not $msg contains 'fax at example.de' and not $msg contains 'localhost.example.de' and not $msg contains 'root at example.de' and not $msg contains 'maildummy@' and not $msg contains 'nobody at example' and not $msg contains 'spamdrop' and not $msg contains 'NOQUEUE' and not $msg contains 'reject:' and ( $msg contains 'to=<' and $msg contains 'status=sent' and not $msg contains 'relay=procmail' and not $msg contains 'relay=/var/lib/imap/socket/lmtp' and not $msg contains 'relay=local' and not $msg contains 'relay=10.0.50.2' ) or $msg contains 'from=<.*example.de>' then /var/log/mail_antispam And also a property based: :msg, contains, "to=<" ~ :msg, contains, "status=sent" ~ :msg, regex, "from=<*example\.de>" ~ :msg, !contains, "localhost.example.de" ~ :msg, !contains, "root at example.de" ~ :msg, !contains, "maildummy@" ~ :msg, !contains, "nobody at example.de" ~ :msg, !contains, "spamdrop" ~ :msg, !contains, "NOQUEUE" ~ :msg, !contains, "reject:" ~ :msg, !contains, "relay=procmail" ~ :msg, !contains, "relay=/var/lib/imap/socket/lmtp" ~ :msg, !contains, "relay=local" ~ :msg, !contains, "relay=10.0.50.2" ~ mail.* -/var/log/mail_antispam At the Property based Filters - I do not understand how to make an AND or an OR... and at the expression based one, I have totally no Idea, why it isn't working. I hope someone can give me a clue about what my mistake is. Best Regards and thank you very much in Advance, Markus From vilchezr at gmail.com Tue Oct 5 00:40:44 2010 From: vilchezr at gmail.com (rodolfo vilchez) Date: Mon, 4 Oct 2010 16:40:44 -0600 Subject: [rsyslog] [rsyslog + syslog deamon smartconnector] In-Reply-To: References: Message-ID: Hello Folks, I'm struggling in trying to parse the events of my rsyslog v3 with the out-of-the-box log reader "Syslog deamon SmartConnector" of Arcsigth. My question is if there is a way to make work rsyslog as it was syslog, so the Arcsight piece of software would work without problems. Many thanks in advance for the help. Rodolfo From david at lang.hm Tue Oct 5 02:10:25 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 4 Oct 2010 17:10:25 -0700 (PDT) Subject: [rsyslog] [rsyslog + syslog deamon smartconnector] In-Reply-To: References: Message-ID: On Mon, 4 Oct 2010, rodolfo vilchez wrote: > Hello Folks, > > I'm struggling in trying to parse the events of my rsyslog v3 with the > out-of-the-box log reader "Syslog deamon SmartConnector" of Arcsigth. My > question is if there is a way to make work rsyslog as it was syslog, so the > Arcsight piece of software would work without problems. out of the box rsyslog is very much like syslog, what is it that you are seeing as being different that's causing you problems? David Lang > Many thanks in advance for the help. > > Rodolfo > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Oct 5 02:10:51 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 4 Oct 2010 17:10:51 -0700 (PDT) Subject: [rsyslog] FYI: Linux Kongress Nuremberg In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com> Message-ID: di the slides for this get posted anywhere? David Lang On Wed, 22 Sep 2010, Rainer Gerhards wrote: > Date: Wed, 22 Sep 2010 11:46:39 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] FYI: Linux Kongress Nuremberg > > Hi all, > > I am heading out to Linux Kongress N?rnberg soon. I will be there Thursday > and Friday. Should someone be there as well and interested in meeting, please > let me know! For the same reason, I'll probably be very silent on the mailing > list. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 5 07:31:15 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 5 Oct 2010 07:31:15 +0200 Subject: [rsyslog] FYI: Linux Kongress Nuremberg References: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD445@GRFEXC.intern.adiscon.com> Oops, sorry. They are here (Adobe icon beneath the title, right at the bottom): http://www.linux-kongress.org/2010/program.html I'll also see that I post the paper (I will re-check, but I think there are no restrictions to doing this). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, October 05, 2010 2:11 AM > To: rsyslog-users > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > > di the slides for this get posted anywhere? > > David Lang > > On Wed, 22 Sep 2010, Rainer Gerhards wrote: > > > Date: Wed, 22 Sep 2010 11:46:39 +0200 > > From: Rainer Gerhards > > Reply-To: rsyslog-users > > To: rsyslog-users > > Subject: [rsyslog] FYI: Linux Kongress Nuremberg > > > > Hi all, > > > > I am heading out to Linux Kongress N?rnberg soon. I will be there > Thursday > > and Friday. Should someone be there as well and interested in > meeting, please > > let me know! For the same reason, I'll probably be very silent on the > mailing > > list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From friedl at hq.adiscon.com Tue Oct 5 17:13:56 2010 From: friedl at hq.adiscon.com (Florian Riedl) Date: Tue, 5 Oct 2010 17:13:56 +0200 Subject: [rsyslog] rsyslog 5.7.1 (v5-devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD454@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.7.0, the new v5-devel. This is a feature-enhancement release. Rsyslog 5.7.1 offers two exciting areas of enhancement: For imuxsock, we introduced a couple of new features, including the ability to rate-limit the number of log messages a process can generate.The user can specify how many messages in which interval (e.g. 200 messages in 5 seconds) are considered acceptable. If this is exceeded, additional messages are dropped. Rate-limiting does not apply to high priority messages, where it is configurable what "high priority" is. Along the same lines, imuxsock can now obtain the process id from the kernel (via SCM_CREDENTIALS) and be reliably added/replaced in the syslog TAG. This makes sure each TAG contains the process id and contains the CORRECT process id (a malicious process can NOT fake it as it is rewritten by imuxsock). Special thanks to Lennart Poettering for suggesting these enhancements. The second area of enhancement is support for Hadoop's HDFS . There is a new module omhdfs which can be used to write data to the HDFS file system. This module was written from scratch and some of its concept's may be introduced into omfile at a later stage (it contains a better file cache). Building omhdfs is a bit complicated due to Hadoop being written in Java. See omhdfs' documentation for more details. Feedback and experiences from users is more than welcome! More information about Hadoop's HDFS is available here. Other than that, there are also some minor things and a bug fix which affects systems using systemd. ChangeLog: http://www.rsyslog.com/changelog-for-5-7-1-v5-devel/ Download: http://www.rsyslog.com/rsyslog-5-7-1-v5-devel/ As always, feedback is appreciated. Best regards, Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From pgollucci at p6m7g8.com Wed Oct 6 09:41:08 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Wed, 06 Oct 2010 07:41:08 +0000 Subject: [rsyslog] freebsd ports rsyslog overhaul Message-ID: <4CAC2814.9060809@p6m7g8.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/151021 Hope that helps. sysutils/rsyslog5-devel -> 5.7.1 update will happen within minutes. - -- - ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFMrCgUdbiP+9ubjBwRAmKUAJ9IgZgZgZ5I9rEuTA2XBpXag/Rg2QCeIQFA l0AU00Qjs4Wx2C8rbQzjpeY= =7fFu -----END PGP SIGNATURE----- From pgollucci at p6m7g8.com Wed Oct 6 09:42:59 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Wed, 06 Oct 2010 07:42:59 +0000 Subject: [rsyslog] freebsd ports rsyslog overhaul In-Reply-To: <4CAC2814.9060809@p6m7g8.com> References: <4CAC2814.9060809@p6m7g8.com> Message-ID: <4CAC2883.8030902@p6m7g8.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/06/10 07:41, Philip M. Gollucci wrote: > http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/151021 > > Hope that helps. > > sysutils/rsyslog5-devel -> 5.7.1 update will happen within minutes. Or not, can you please patch this and send me a diff to pull locally until 5.7.2. http://people.freebsd.org/~pgollucci/FreeBSD/rsyslog-5.7.1.log - -- - ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFMrCiDdbiP+9ubjBwRAqgJAJwJbXUxAhzHqhUpVvvd+5Nl+XsCSwCgk7RL XMo3He1lpuBpUR8b1sPC6xM= =xjL/ -----END PGP SIGNATURE----- From rgerhards at hq.adiscon.com Wed Oct 6 09:55:43 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 09:55:43 +0200 Subject: [rsyslog] freebsd ports rsyslog overhaul References: <4CAC2814.9060809@p6m7g8.com> <4CAC2883.8030902@p6m7g8.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD45D@GRFEXC.intern.adiscon.com> Args, sorry... I have forgotten to do the configure check for the new functionality... Will fix this soon and let you know then. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > Sent: Wednesday, October 06, 2010 9:43 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] freebsd ports rsyslog overhaul > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 10/06/10 07:41, Philip M. Gollucci wrote: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/151021 > > > > Hope that helps. > > > > sysutils/rsyslog5-devel -> 5.7.1 update will happen within minutes. > > Or not, can you please patch this and send me a diff to pull locally > until 5.7.2. > > http://people.freebsd.org/~pgollucci/FreeBSD/rsyslog-5.7.1.log > > > - -- > - --------------------------------------------------------------------- > --- > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 > VP Apache Infrastructure; Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.16 (FreeBSD) > > iD8DBQFMrCiDdbiP+9ubjBwRAqgJAJwJbXUxAhzHqhUpVvvd+5Nl+XsCSwCgk7RL > XMo3He1lpuBpUR8b1sPC6xM= > =xjL/ > -----END PGP SIGNATURE----- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 6 12:07:47 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 12:07:47 +0200 Subject: [rsyslog] FYI: Linux Kongress Nuremberg References: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD445@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD45F@GRFEXC.intern.adiscon.com> The paper itself is now also available online: http://blog.gerhards.net/2010/10/linux-kongress-2010-rsyslog-paper.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, October 05, 2010 7:31 AM > To: rsyslog-users > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > > Oops, sorry. They are here (Adobe icon beneath the title, right at the > bottom): > > http://www.linux-kongress.org/2010/program.html > > I'll also see that I post the paper (I will re-check, but I think there > are > no restrictions to doing this). > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Tuesday, October 05, 2010 2:11 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > > > > di the slides for this get posted anywhere? > > > > David Lang > > > > On Wed, 22 Sep 2010, Rainer Gerhards wrote: > > > > > Date: Wed, 22 Sep 2010 11:46:39 +0200 > > > From: Rainer Gerhards > > > Reply-To: rsyslog-users > > > To: rsyslog-users > > > Subject: [rsyslog] FYI: Linux Kongress Nuremberg > > > > > > Hi all, > > > > > > I am heading out to Linux Kongress N?rnberg soon. I will be there > > Thursday > > > and Friday. Should someone be there as well and interested in > > meeting, please > > > let me know! For the same reason, I'll probably be very silent on > the > > mailing > > > list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 6 14:17:23 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 14:17:23 +0200 Subject: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng References: <4CA9B1C1.6080504@in-put.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD461@GRFEXC.intern.adiscon.com> Hi Markus, unfortunately I do not have time at the moment to go through this at full length. But a couple of points below: > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Markus Stegmueller > Sent: Monday, October 04, 2010 12:52 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng > > Hello, > > I am trying to convert my syslog-ng Filter to a working rsyslog Filter > Configuration. Here is my > syslog-ng Filter: > > destination d_antispam { file("/var/log/mail_antispam"); }; > > filter f_antispam { facility(mail) and not match("fax at example.de") > and not > match("localhost.example.de") and not match("root at example.de") and not > match("maildummy@") and not > match("nobody at example") and not match("spamdrop") and not > match("NOQUEUE") and not match("reject:") > and ((match("to=<") and match("status=sent") and not > match("relay=procmail") and not > match("relay=/var/lib/imap/socket/lmtp") and not match("relay=local") > and not > match("relay=10.0.50.2")) or match("from=<.*example\.de>")); }; > > log { source(src); filter(f_antispam); destination(d_antispam); }; > > This Filter is to get all the mails, that were send from example.de > users, so I can re-use them to > make a whitelist. I don't know syslog-ng syntax, but guess I got an idea of what you do. > > > I tried to make an expression Based Filter: > > if $syslogfacility-text == 'mail' and not $msg contains > 'fax at example.de' and not $msg contains > 'localhost.example.de' and not $msg contains 'root at example.de' and not > $msg contains 'maildummy@' > and not $msg contains 'nobody at example' and not $msg contains 'spamdrop' > and not $msg contains > 'NOQUEUE' and not $msg contains 'reject:' and ( $msg contains 'to=<' > and $msg contains 'status=sent' > and not $msg contains 'relay=procmail' and not $msg contains > 'relay=/var/lib/imap/socket/lmtp' and > not $msg contains 'relay=local' and not $msg contains 'relay=10.0.50.2' > ) or $msg contains > 'from=<.*example.de>' then /var/log/mail_antispam This sounds basically correct. Is it all on one line? if not, you need to add "\" at the end of each line, an artifact of the current config system. like this: if $syslogfacility-text == 'mail' and not $msg contains \ 'fax at example.de' and not $msg contains \ 'localhost.example.de' and not $msg contains 'root at example.de' and not \ ... If that does not solve the issue, please reduce the logic until it works. Then we can check what the culprit is. Also make sure you do not get any errors while reading the config file. Finally, a look at rsyslog's debug log may help. > And also a property based: > > :msg, contains, "to=<" ~ > :msg, contains, "status=sent" ~ > :msg, regex, "from=<*example\.de>" ~ > :msg, !contains, "localhost.example.de" ~ > :msg, !contains, "root at example.de" ~ > :msg, !contains, "maildummy@" ~ > :msg, !contains, "nobody at example.de" ~ > :msg, !contains, "spamdrop" ~ > :msg, !contains, "NOQUEUE" ~ > :msg, !contains, "reject:" ~ > :msg, !contains, "relay=procmail" ~ > :msg, !contains, "relay=/var/lib/imap/socket/lmtp" ~ > :msg, !contains, "relay=local" ~ > :msg, !contains, "relay=10.0.50.2" ~ > mail.* -/var/log/mail_antispam > > > At the Property based Filters - I do not understand how to make an AND > or an OR... and at the > expression based one, Because that's not possible. For boolean filters, you need to use the if-type of filter. HTH Rainer From rgerhards at hq.adiscon.com Wed Oct 6 14:19:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 14:19:29 +0200 Subject: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng References: <4CA9B1C1.6080504@in-put.de> <9B6E2A8877C38245BFB15CC491A11DA71DD461@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD462@GRFEXC.intern.adiscon.com> Markus, this may also be useful: http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated _by_dynamic_directory Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 06, 2010 2:17 PM > To: rsyslog-users > Subject: Re: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng > > Hi Markus, > > unfortunately I do not have time at the moment to go through this at > full > length. But a couple of points below: > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Markus Stegmueller > > Sent: Monday, October 04, 2010 12:52 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Outgoing Mails Logfile - Ported from Syslog-ng > > > > Hello, > > > > I am trying to convert my syslog-ng Filter to a working rsyslog > Filter > > Configuration. Here is my > > syslog-ng Filter: > > > > destination d_antispam { file("/var/log/mail_antispam"); }; > > > > filter f_antispam { facility(mail) and not match("fax at example.de") > > and not > > match("localhost.example.de") and not match("root at example.de") and > not > > match("maildummy@") and not > > match("nobody at example") and not match("spamdrop") and not > > match("NOQUEUE") and not match("reject:") > > and ((match("to=<") and match("status=sent") and not > > match("relay=procmail") and not > > match("relay=/var/lib/imap/socket/lmtp") and not match("relay=local") > > and not > > match("relay=10.0.50.2")) or match("from=<.*example\.de>")); }; > > > > log { source(src); filter(f_antispam); destination(d_antispam); }; > > > > This Filter is to get all the mails, that were send from example.de > > users, so I can re-use them to > > make a whitelist. > > I don't know syslog-ng syntax, but guess I got an idea of what you do. > > > > > > > I tried to make an expression Based Filter: > > > > if $syslogfacility-text == 'mail' and not $msg contains > > 'fax at example.de' and not $msg contains > > 'localhost.example.de' and not $msg contains 'root at example.de' and > not > > $msg contains 'maildummy@' > > and not $msg contains 'nobody at example' and not $msg contains > 'spamdrop' > > and not $msg contains > > 'NOQUEUE' and not $msg contains 'reject:' and ( $msg contains 'to=<' > > and $msg contains 'status=sent' > > and not $msg contains 'relay=procmail' and not $msg contains > > 'relay=/var/lib/imap/socket/lmtp' and > > not $msg contains 'relay=local' and not $msg contains > 'relay=10.0.50.2' > > ) or $msg contains > > 'from=<.*example.de>' then /var/log/mail_antispam > > This sounds basically correct. Is it all on one line? if not, you need > to add > "\" at the end of each line, an artifact of the current config system. > like > this: > > if $syslogfacility-text == 'mail' and not $msg contains \ > 'fax at example.de' and not $msg contains \ > 'localhost.example.de' and not $msg contains 'root at example.de' and not > \ > ... > > If that does not solve the issue, please reduce the logic until it > works. > Then we can check what the culprit is. Also make sure you do not get > any > errors while reading the config file. Finally, a look at rsyslog's > debug log > may help. > > > And also a property based: > > > > :msg, contains, "to=<" ~ > > :msg, contains, "status=sent" ~ > > :msg, regex, "from=<*example\.de>" ~ > > :msg, !contains, "localhost.example.de" ~ > > :msg, !contains, "root at example.de" ~ > > :msg, !contains, "maildummy@" ~ > > :msg, !contains, "nobody at example.de" ~ > > :msg, !contains, "spamdrop" ~ > > :msg, !contains, "NOQUEUE" ~ > > :msg, !contains, "reject:" ~ > > :msg, !contains, "relay=procmail" ~ > > :msg, !contains, "relay=/var/lib/imap/socket/lmtp" ~ > > :msg, !contains, "relay=local" ~ > > :msg, !contains, "relay=10.0.50.2" ~ > > mail.* -/var/log/mail_antispam > > > > > > At the Property based Filters - I do not understand how to make an > AND > > or an OR... and at the > > expression based one, > > Because that's not possible. For boolean filters, you need to use the > if-type > of filter. > > HTH > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 6 14:26:34 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 14:26:34 +0200 Subject: [rsyslog] freebsd ports rsyslog overhaul References: <4CAC2814.9060809@p6m7g8.com> <4CAC2883.8030902@p6m7g8.com> <9B6E2A8877C38245BFB15CC491A11DA71DD45D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD463@GRFEXC.intern.adiscon.com> I have created a patch, unfortunately in two commits: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=c0de8c6dc4ec4ac270b0c313 ea5a1d2689b44db3 http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=d51d6847a84eeec573ce2dd4 feeee991d3eb0bcc to be applied in above order. These patches effectively disable rate-limiting if SCM_CREDENTIALS is not available. This is far from perfect. I'll think about the situation and either introduce a global rate-limiter or one that works on the TAG value instead. But that's nothing for today. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 06, 2010 9:56 AM > To: rsyslog-users > Subject: Re: [rsyslog] freebsd ports rsyslog overhaul > > Args, sorry... I have forgotten to do the configure check for the new > functionality... Will fix this soon and let you know then. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > > Sent: Wednesday, October 06, 2010 9:43 AM > > To: rsyslog at lists.adiscon.com > > Subject: Re: [rsyslog] freebsd ports rsyslog overhaul > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On 10/06/10 07:41, Philip M. Gollucci wrote: > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/151021 > > > > > > Hope that helps. > > > > > > sysutils/rsyslog5-devel -> 5.7.1 update will happen within minutes. > > > > Or not, can you please patch this and send me a diff to pull locally > > until 5.7.2. > > > > http://people.freebsd.org/~pgollucci/FreeBSD/rsyslog-5.7.1.log > > > > > > - -- > > - ------------------------------------------------------------------- > -- > > --- > > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > > Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 > > VP Apache Infrastructure; Member, Apache Software Foundation > > Committer, FreeBSD Foundation > > Consultant, P6M7G8 Inc. > > Sr. System Admin, Ridecharge Inc. > > > > Work like you don't need the money, > > love like you'll never get hurt, > > and dance like nobody's watching. > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v2.0.16 (FreeBSD) > > > > iD8DBQFMrCiDdbiP+9ubjBwRAqgJAJwJbXUxAhzHqhUpVvvd+5Nl+XsCSwCgk7RL > > XMo3He1lpuBpUR8b1sPC6xM= > > =xjL/ > > -----END PGP SIGNATURE----- > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From epiphani at gmail.com Wed Oct 6 15:13:22 2010 From: epiphani at gmail.com (Aaron Wiebe) Date: Wed, 6 Oct 2010 09:13:22 -0400 Subject: [rsyslog] FYI: Linux Kongress Nuremberg In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD45F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD445@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD45F@GRFEXC.intern.adiscon.com> Message-ID: Interesting paper Rainer - good read! -Aaron On Wed, Oct 6, 2010 at 6:07 AM, Rainer Gerhards wrote: > The paper itself is now also available online: > > http://blog.gerhards.net/2010/10/linux-kongress-2010-rsyslog-paper.html > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, October 05, 2010 7:31 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg >> >> Oops, sorry. They are here (Adobe icon beneath the title, right at the >> bottom): >> >> http://www.linux-kongress.org/2010/program.html >> >> I'll also see that I post the paper (I will re-check, but I think there >> are >> no restrictions to doing this). >> >> Rainer >> >> > -----Original Message----- >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> > Sent: Tuesday, October 05, 2010 2:11 AM >> > To: rsyslog-users >> > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg >> > >> > di the slides for this get posted anywhere? >> > >> > David Lang >> > >> > On Wed, 22 Sep 2010, Rainer Gerhards wrote: >> > >> > > Date: Wed, 22 Sep 2010 11:46:39 +0200 >> > > From: Rainer Gerhards >> > > Reply-To: rsyslog-users >> > > To: rsyslog-users >> > > Subject: [rsyslog] FYI: Linux Kongress Nuremberg >> > > >> > > Hi all, >> > > >> > > I am heading out to Linux Kongress N?rnberg soon. I will be there >> > Thursday >> > > and Friday. Should someone be there as well and interested in >> > meeting, please >> > > let me know! For the same reason, I'll probably be very silent on >> the >> > mailing >> > > list. >> > > >> > > Rainer >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com >> > > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Oct 6 15:59:48 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 6 Oct 2010 15:59:48 +0200 Subject: [rsyslog] FYI: Linux Kongress Nuremberg References: <9B6E2A8877C38245BFB15CC491A11DA71DD3D4@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD445@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD45F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD468@GRFEXC.intern.adiscon.com> Thanks, Aaron. I hope to be able to put some more data up once I can begin with the next performance optimization project. I'd really appreciate all discussions around what I plan and I think we will probably be able to do quite a bit better than we already do ;) But for the next month or so I'll need to dig into some very exiciting normalizing effort, which I think will be very useful for rsyslog. I'll provide some more details as soon as I have managed to finalize them - hopefully soon. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Wednesday, October 06, 2010 3:13 PM > To: rsyslog-users > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > > Interesting paper Rainer - good read! > > -Aaron > > On Wed, Oct 6, 2010 at 6:07 AM, Rainer Gerhards > wrote: > > The paper itself is now also available online: > > > > http://blog.gerhards.net/2010/10/linux-kongress-2010-rsyslog- > paper.html > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Tuesday, October 05, 2010 7:31 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > >> > >> Oops, sorry. They are here (Adobe icon beneath the title, right at > the > >> bottom): > >> > >> http://www.linux-kongress.org/2010/program.html > >> > >> I'll also see that I post the paper (I will re-check, but I think > there > >> are > >> no restrictions to doing this). > >> > >> Rainer > >> > >> > -----Original Message----- > >> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> > Sent: Tuesday, October 05, 2010 2:11 AM > >> > To: rsyslog-users > >> > Subject: Re: [rsyslog] FYI: Linux Kongress Nuremberg > >> > > >> > di the slides for this get posted anywhere? > >> > > >> > David Lang > >> > > >> > On Wed, 22 Sep 2010, Rainer Gerhards wrote: > >> > > >> > > Date: Wed, 22 Sep 2010 11:46:39 +0200 > >> > > From: Rainer Gerhards > >> > > Reply-To: rsyslog-users > >> > > To: rsyslog-users > >> > > Subject: [rsyslog] FYI: Linux Kongress Nuremberg > >> > > > >> > > Hi all, > >> > > > >> > > I am heading out to Linux Kongress N?rnberg soon. I will be > there > >> > Thursday > >> > > and Friday. Should someone be there as well and interested in > >> > meeting, please > >> > > let me know! For the same reason, I'll probably be very silent > on > >> the > >> > mailing > >> > > list. > >> > > > >> > > Rainer > >> > > _______________________________________________ > >> > > rsyslog mailing list > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > http://www.rsyslog.com > >> > > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Oct 7 17:26:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 7 Oct 2010 17:26:18 +0200 Subject: [rsyslog] Introducing liblognorm Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> Hi folks, I am happy to finally announce the liblognorm project, a facility to normalize all kinds of log messages. My initial announcement is here: http://blog.gerhards.net/2010/10/introducing-liblognorm.html I'd appreciate if you follow this effort and share the news. I plan to do a couple of more blog posts within the next days. Rainer From jean.luc.lacroix at regains.be Fri Oct 8 09:09:01 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Fri, 08 Oct 2010 09:09:01 +0200 Subject: [rsyslog] [compile] could not create Makefile Message-ID: <4CAEC38D.5040809@regains.be> Hi, I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the configure script throw awk errors when trying to create the make file: CODE: SELECT ALL config.status: creating Makefile gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" gawk: ./confgA6g9j/subs.awk:2: ^ syntax error gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" gawk: ./confgA6g9j/subs.awk:3: ^ syntax error gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# "#" gawk: ./confgA6g9j/subs.awk:4: ^ syntax error gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" gawk: ./confgA6g9j/subs.awk:4: ^ syntax error gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" gawk: ./confgA6g9j/subs.awk:5: ^ syntax error ... many more of these errors gawk version installed on my system: Package: gawk Version: 3.1.8-1 -- Jean-Luc Lacroix Regains SCS 0477 43 72 72 From rgerhards at hq.adiscon.com Fri Oct 8 10:06:12 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 10:06:12 +0200 Subject: [rsyslog] [compile] could not create Makefile References: <4CAEC38D.5040809@regains.be> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> I am sorry, I am far from being an autotools expert. So I can not help here. Michael maybe? Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix > Sent: Friday, October 08, 2010 9:09 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] [compile] could not create Makefile > > Hi, > > I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the > configure script > throw awk errors when trying to create the make file: > > CODE: SELECT ALL > config.status: creating Makefile > gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" > gawk: ./confgA6g9j/subs.awk:2: ^ syntax > error > gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" > gawk: ./confgA6g9j/subs.awk:3: ^ > syntax error > gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# > "#" > gawk: ./confgA6g9j/subs.awk:4: ^ syntax > error > gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" > gawk: ./confgA6g9j/subs.awk:4: ^ syntax error > gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" > gawk: ./confgA6g9j/subs.awk:5: ^ syntax > error > > ... many more of these errors > > > > gawk version installed on my system: > Package: gawk Version: 3.1.8-1 > -- > Jean-Luc Lacroix > Regains SCS > 0477 43 72 72 > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From karsten.heymann at telecolumbus.net Fri Oct 8 10:50:02 2010 From: karsten.heymann at telecolumbus.net (Karsten Heymann) Date: Fri, 08 Oct 2010 10:50:02 +0200 Subject: [rsyslog] 5.4.0 hangs after several hours Message-ID: <87fwwhax1h.fsf@ara.blue-cable.net> Hello, I'm currently implementing a new central log server with 5.4.0 on Debian Linux for our company and am running into severe stability problems. I hope this list is the right place to report and discuss these, if not, please point me to the right direction. Our logserver receives logs via udp and tcp on several ports and handles them with different rulesets (this is why we upgraded to 5.4.0): %- /etc/rsyslog.conf $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $IncludeConfig /etc/rsyslog.d/*.conf auth,authpriv.* /var/log/auth.log [... more file rules omitted] %- %- /etc/rsyslog.d/remote.conf $RuleSet udp514 local0.* -/var/log/cisco/local0.log [... more file rules omitted] $RuleSet tcp514 $RuleSet tcp10514 auth,authpriv.* /var/log/server/auth.log [... more file rules omitted] $RuleSet tcp20514 $ModLoad imudp $InputUDPServerBindRuleset udp514 $UDPServerRun 514 $ModLoad imtcp $InputTCPServerBindRuleset tcp514 $InputTCPServerRun 514 $InputTCPServerBindRuleset tcp10514 $INPUTTCPServerRun 10514 $InputTCPServerBindRuleset tcp20514 $INPUTTCPServerRun 20514 $RuleSet RSYSLOG_DefaultRuleset %- rsyslog is started with "/usr/sbin/rsyslogd -c5". The Problem: After several hours, one rsyslogd process starts running at 100% cpu and uses more and more memory, also it completely stops writing to the logfiles (hence no rsyslog error messages too). If I run strace -p i get a constant stream of write(3, "Oct 8 09:40:42 loghost1-01 kerne"..., 266) = -1 EAGAIN (Resource temporarily unavailable) system calls. Can you give me any hints how to debug this further? Yours Karsten From david at lang.hm Fri Oct 8 12:30:00 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 8 Oct 2010 03:30:00 -0700 (PDT) Subject: [rsyslog] 5.4.0 hangs after several hours In-Reply-To: <87fwwhax1h.fsf@ara.blue-cable.net> References: <87fwwhax1h.fsf@ara.blue-cable.net> Message-ID: it sounds as if an output is getting stuck and then the queue is filling up. re all your rules writing to local files, or are some of them writing to something that could block? in top, use the 'H' option to turn on per-thread reporting, I suspect that there is one thread that is getting stuck. It may beworth dong an strace on the threads to see what they are doing (especially if there is one stuck at 100% cpu) David Lang On Fri, 8 Oct 2010, Karsten Heymann wrote: > Date: Fri, 08 Oct 2010 10:50:02 +0200 > From: Karsten Heymann > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] 5.4.0 hangs after several hours > > Hello, > > I'm currently implementing a new central log server with 5.4.0 on Debian > Linux for our company and am running into severe stability problems. I > hope this list is the right place to report and discuss these, if not, > please point me to the right direction. > > Our logserver receives logs via udp and tcp on several ports and handles > them with different rulesets (this is why we upgraded to 5.4.0): > > %- /etc/rsyslog.conf > $ModLoad imuxsock # provides support for local system logging > $ModLoad imklog # provides kernel logging support (previously done by rklogd) > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > $FileOwner root > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $IncludeConfig /etc/rsyslog.d/*.conf > auth,authpriv.* /var/log/auth.log > [... more file rules omitted] > %- > > %- /etc/rsyslog.d/remote.conf > $RuleSet udp514 > local0.* -/var/log/cisco/local0.log > [... more file rules omitted] > $RuleSet tcp514 > $RuleSet tcp10514 > auth,authpriv.* /var/log/server/auth.log > [... more file rules omitted] > $RuleSet tcp20514 > $ModLoad imudp > $InputUDPServerBindRuleset udp514 > $UDPServerRun 514 > $ModLoad imtcp > $InputTCPServerBindRuleset tcp514 > $InputTCPServerRun 514 > $InputTCPServerBindRuleset tcp10514 > $INPUTTCPServerRun 10514 > $InputTCPServerBindRuleset tcp20514 > $INPUTTCPServerRun 20514 > $RuleSet RSYSLOG_DefaultRuleset > %- > > rsyslog is started with "/usr/sbin/rsyslogd -c5". > > The Problem: > > After several hours, one rsyslogd process starts running at 100% cpu and > uses more and more memory, also it completely stops writing to the > logfiles (hence no rsyslog error messages too). If I run > > strace -p > > i get a constant stream of > > write(3, "Oct 8 09:40:42 loghost1-01 kerne"..., 266) = -1 EAGAIN > (Resource temporarily unavailable) > > system calls. > > Can you give me any hints how to debug this further? > > Yours > Karsten > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 8 13:48:28 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 13:48:28 +0200 Subject: [rsyslog] 5.4.0 hangs after several hours References: <87fwwhax1h.fsf@ara.blue-cable.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD482@GRFEXC.intern.adiscon.com> I'd not bother looking into the 5.4.0 issue. The current beta most probably fixes the issue (5.4.0 was not a very good build). The beta is scheduled to become the new stable next week or so. Sorry that version did not work out well... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, October 08, 2010 12:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] 5.4.0 hangs after several hours > > it sounds as if an output is getting stuck and then the queue is > filling > up. > > re all your rules writing to local files, or are some of them writing > to > something that could block? > > in top, use the 'H' option to turn on per-thread reporting, I suspect > that > there is one thread that is getting stuck. It may beworth dong an > strace > on the threads to see what they are doing (especially if there is one > stuck at 100% cpu) > > David Lang > > On Fri, 8 Oct 2010, Karsten Heymann wrote: > > > Date: Fri, 08 Oct 2010 10:50:02 +0200 > > From: Karsten Heymann > > Reply-To: rsyslog-users > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] 5.4.0 hangs after several hours > > > > Hello, > > > > I'm currently implementing a new central log server with 5.4.0 on > Debian > > Linux for our company and am running into severe stability problems. > I > > hope this list is the right place to report and discuss these, if > not, > > please point me to the right direction. > > > > Our logserver receives logs via udp and tcp on several ports and > handles > > them with different rulesets (this is why we upgraded to 5.4.0): > > > > %- /etc/rsyslog.conf > > $ModLoad imuxsock # provides support for local system logging > > $ModLoad imklog # provides kernel logging support (previously done > by rklogd) > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $FileOwner root > > $FileGroup adm > > $FileCreateMode 0640 > > $DirCreateMode 0755 > > $Umask 0022 > > $IncludeConfig /etc/rsyslog.d/*.conf > > auth,authpriv.* /var/log/auth.log > > [... more file rules omitted] > > %- > > > > %- /etc/rsyslog.d/remote.conf > > $RuleSet udp514 > > local0.* -/var/log/cisco/local0.log > > [... more file rules omitted] > > $RuleSet tcp514 > > $RuleSet tcp10514 > > auth,authpriv.* /var/log/server/auth.log > > [... more file rules omitted] > > $RuleSet tcp20514 > > $ModLoad imudp > > $InputUDPServerBindRuleset udp514 > > $UDPServerRun 514 > > $ModLoad imtcp > > $InputTCPServerBindRuleset tcp514 > > $InputTCPServerRun 514 > > $InputTCPServerBindRuleset tcp10514 > > $INPUTTCPServerRun 10514 > > $InputTCPServerBindRuleset tcp20514 > > $INPUTTCPServerRun 20514 > > $RuleSet RSYSLOG_DefaultRuleset > > %- > > > > rsyslog is started with "/usr/sbin/rsyslogd -c5". > > > > The Problem: > > > > After several hours, one rsyslogd process starts running at 100% cpu > and > > uses more and more memory, also it completely stops writing to the > > logfiles (hence no rsyslog error messages too). If I run > > > > strace -p > > > > i get a constant stream of > > > > write(3, "Oct 8 09:40:42 loghost1-01 kerne"..., 266) = -1 EAGAIN > > (Resource temporarily unavailable) > > > > system calls. > > > > Can you give me any hints how to debug this further? > > > > Yours > > Karsten > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From champ at softwink.com Fri Oct 8 14:41:41 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 08:41:41 -0400 Subject: [rsyslog] Sagan + Rsyslog [$template question] Message-ID: <20101008124141.GA966@bundy.vistech.net> Hello all! I've been working on a project known as "Sagan" for some time. Basically, Sagan uses a 'Snort like' (IDS/IPS) rule set and can detect when "back things" are happening and correlate that information with your IDS/IPS Snort back end. For more information about Sagan, please see: http://sagan.softwink.com When I started writing Sagan, it was syslog-ng centric. However, considering the amount of rsyslog users, I decided to take a shot at making Sagan work with rsyslog. T'was a breeze, and over all, I'm dang impressed with rsyslog's functionality. Sagan 'read' a named pipe (FIFO) for events, then uses it's rule set to 'parse' these events for 'bad things' happening (attacks, system problems, etc). In the syslog-ng world, my template for Sagan is: --------------------------------------- destination sagan { pipe("/var/run/sagan.fifo" template("$SOURCEIP|$FACILITY|$PRIORITY|$LEVEL|$TAG|$YEAR-$MONTH-$DAY|$HOUR:$MIN:$SEC|$PROGRAM|$MSG\n") template-escape(no)); }; --------------------------------------- I'm attempting to mimic the this template with rsyslog. This is the template that I have so far: --------------------------------------- # Template for Sagan (for the FIFO) $template sagan,"%fromhost-ip%|%syslogfacility-text%|%syslogpriority-text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-rfc3339%|%timegenerated:12:19:date-rfc3339%|%programname%|%msg%\n" # Calling the named pipe... *.* |/var/run/sagan.fifo;sagan --------------------------------------- Sagan seems to be able to read the events from the named pipe fine. I was wonder, based on the functionality, if anyone might see a better way to do this? I noticed between rsyslog and syslog-ng, the $TAG and %syslogtag% functionality seems to be a bit different. For example: Syslog-ng "$TAG" == 1e rsyslog-ng %syslogtag% == the program name (for example: 'rsync:') Other than this, it seems to function as normal. Does any one know how syslog-ng verses rsyslog treats 'tags'. Thanks in advance. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Fri Oct 8 14:55:09 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 08:55:09 -0400 Subject: [rsyslog] liblognorm Message-ID: <20101008125509.GA1416@bundy.vistech.net> On Thu, Oct 07, 2010 at 05:26:18PM +0200, Rainer Gerhards wrote: > Hi folks, > > I am happy to finally announce the liblognorm project, a facility to > normalize all kinds of log messages. My initial announcement is here: > > http://blog.gerhards.net/2010/10/introducing-liblognorm.html > > I'd appreciate if you follow this effort and share the news. I plan to do a > couple of more blog posts within the next days. Very cool idea, and the strange thing is.... I was look, yesterday, for the exact same idea of a library. Needless to say, I found no such project. I have a couple of questions: What license do you plan on distributing liblognorm under? The example formating (in your blog post) looks just about exactly what I was thinking. Will the log formats, that is, what to look for, be in a seperate text file that can be updated? As you mention in the blog post, "much like a virus scanner". If that is the case, will programs that use liblognorm pull that into an array first, and let liblognorm do it's work? Or will this file have to be repeatedly referenced? I'd love to help out in this, be it code wise and/or log examples. Thans for rsyslog, and keep up the great work. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Fri Oct 8 15:00:05 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 09:00:05 -0400 Subject: [rsyslog] Rsyslog with Sagan. Message-ID: <20101008130005.GA1583@bundy.vistech.net> Hello all! I've been working on a project known as "Sagan" for some time. Basically, Sagan uses a 'Snort like' (IDS/IPS) rule set and can detect when "back things" are happening and correlate that information with your IDS/IPS Snort back end. For more information about Sagan, please see: http://sagan.softwink.com When I started writing Sagan, it was syslog-ng centric. However, considering the amount of rsyslog users, I decided to take a shot at making Sagan work with rsyslog. T'was a breeze, and over all, I'm dang impressed with rsyslog's functionality. Sagan 'read' a named pipe (FIFO) for events, then uses it's rule set to 'parse' these events for 'bad things' happening (attacks, system problems, etc). In the syslog-ng world, my template for Sagan is: --------------------------------------- destination sagan { pipe("/var/run/sagan.fifo" template("$SOURCEIP|$FACILITY|$PRIORITY|$LEVEL|$TAG|$YEAR-$MONTH-$DAY|$HOUR:$MIN:$SEC|$PROGRAM|$MSG\n") template-escape(no)); }; --------------------------------------- I'm attempting to mimic the this template with rsyslog. This is the template that I have so far: --------------------------------------- # Template for Sagan (for the FIFO) $template sagan,"%fromhost-ip%|%syslogfacility-text%|%syslogpriority-text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date-rfc3339%|%timegenerated:12:19:date-rfc3339%|%programname%|%msg%\n" # Calling the named pipe... *.* |/var/run/sagan.fifo;sagan --------------------------------------- Sagan seems to be able to read the events from the named pipe fine. I was wonder, based on the functionality, if anyone might see a better way to do this? I noticed between rsyslog and syslog-ng, the $TAG and %syslogtag% functionality seems to be a bit different. For example: Syslog-ng "$TAG" == 1e rsyslog-ng %syslogtag% == the program name (for example: 'rsync:') Other than this, it seems to function as normal. Does any one know how syslog-ng verses rsyslog treats 'tags'. Thanks in advance. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Fri Oct 8 15:07:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 15:07:37 +0200 Subject: [rsyslog] Sagan + Rsyslog [$template question] References: <20101008124141.GA966@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD483@GRFEXC.intern.adiscon.com> Hi, first things first: good news! :) On to the meat... > I've been working on a project known as "Sagan" for some time. > Basically, Sagan uses a 'Snort like' (IDS/IPS) rule set and can detect > when "back things" are happening and correlate that information with > your > IDS/IPS Snort back end. For more information about Sagan, please see: > > http://sagan.softwink.com > > When I started writing Sagan, it was syslog-ng centric. > However, > considering the amount of rsyslog users, I decided to take a shot at > making Sagan work with rsyslog. T'was a breeze, and over all, I'm > dang impressed with rsyslog's functionality. > > Sagan 'read' a named pipe (FIFO) for events, then uses it's rule > set to 'parse' these events for 'bad things' happening (attacks, system > problems, etc). In the syslog-ng world, my template for Sagan is: > > --------------------------------------- > > destination sagan { > pipe("/var/run/sagan.fifo" > template("$SOURCEIP|$FACILITY|$PRIORITY|$LEVEL|$TAG|$YEAR-$MONTH- > $DAY|$HOUR:$MIN:$SEC|$PROGRAM|$MSG\n") template-escape(no)); }; > > --------------------------------------- > > I'm attempting to mimic the this template with rsyslog. This is > the > template that I have so far: > > --------------------------------------- > > # Template for Sagan (for the FIFO) > > $template sagan,"%fromhost-ip%|%syslogfacility-text%|%syslogpriority- > text%|%syslogseverity-text%|%syslogtag%|%timegenerated:1:10:date- > rfc3339%|%timegenerated:12:19:date-rfc3339%|%programname%|%msg%\n" > > # Calling the named pipe... > > *.* |/var/run/sagan.fifo;sagan > > --------------------------------------- > > Sagan seems to be able to read the events from the named pipe > fine. > I was wonder, based on the functionality, if anyone might see a > better > way to do this? This looks good. The only thing I see instantly is that we could spare a view CPU cycles (with recent v5 builds) if we use a strgen module. A strgen effectively builds the template string via C code instead of the template processor and thus is a bit faster and more flexible. Nothing to really care about for low to medium number of messages, but makes a small difference if you have severl ten-thousand messages per second. It isn't hard to write one and I think I could contribute it some time next week if you find it useful. > I noticed between rsyslog and syslog-ng, the $TAG > and %syslogtag% functionality seems to be a bit different. For > example: > > Syslog-ng "$TAG" == 1e > rsyslog-ng %syslogtag% == the program name (for example: 'rsync:') > I really don't know what syslog-ng does here. Is it really the string 1e or the Ascii char 1e? If so, this is very strange to me. SYSLOG-TAG is defined in RFC5424 and traditional meaning is described in RFC 3164. rsyslog uses these definitions. But I don't know syslog-ng at all, so I do not know what they call the "tag". But obviously they have applied a non-standard meaning to this name. Rainer From rgerhards at hq.adiscon.com Fri Oct 8 15:17:34 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 15:17:34 +0200 Subject: [rsyslog] liblognorm References: <20101008125509.GA1416@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Friday, October 08, 2010 2:55 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] liblognorm > > > On Thu, Oct 07, 2010 at 05:26:18PM +0200, Rainer Gerhards wrote: > > Hi folks, > > > > I am happy to finally announce the liblognorm project, a facility to > > normalize all kinds of log messages. My initial announcement is here: > > > > http://blog.gerhards.net/2010/10/introducing-liblognorm.html > > > > I'd appreciate if you follow this effort and share the news. I plan > to do a > > couple of more blog posts within the next days. > > Very cool idea, and the strange thing is.... I was look, > yesterday, for the exact same idea of a library. Needless to say, > I found no such project. I have a couple of questions: > > What license do you plan on distributing liblognorm under? It's not my sole decision, but I think it will most probably by LGPL v2 or v3. BSD may make some sense as we try to get it into wide-spread use, but I may not get along with this. My bet is that LGPL v2 will be the compromise. > The example formating (in your blog post) looks just about exactly what > I was thinking. :-) Starting this morning I have begun to wade in-depth through a pile of CEE docs (the upcoming standard). We may need to make some format adjustments, but I'd like to stick with somewhat as simple as possible to write for the samples. > Will the log formats, that is, what to look for, > be in a seperate text file that can be updated? As you mention in the > blog post, "much like a virus scanner". Actually, at this point nothing is really finalized. For the initial effort, it will definitely be a text file. But in the long term I can also envision that we should be able to pull it e.g. from web servers or load multiple files, or... actually something that should be configurable. > If that is the case, will programs that use liblognorm pull that into > an array first, and let liblognorm do it's work? Or will this file > have to be repeatedly referenced? In the long term I think it makes sense to provide a way to stuff samples in via either a dedicated API OR (and?) have them loaded based on a config file. The latter is what I have on my mind for the first step. > I'd love to help out in this, be it code wise and/or log examples. Any help is deeply appreciated and these questions already help! Log samples and trying out the library, once it materializes, would be a very very great aid. I will write a couple of more blog posts with more details, and everything is really open for change now. It would also be very useful if you could spread the news. An as wide audience as possible would be very good. Rainer > Thans for rsyslog, and keep up the great work. > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From epiphani at gmail.com Fri Oct 8 15:23:04 2010 From: epiphani at gmail.com (Aaron Wiebe) Date: Fri, 8 Oct 2010 09:23:04 -0400 Subject: [rsyslog] Introducing liblognorm In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> Message-ID: Rainer - when do you expect to be publishing a git repository? -Aaron On Thu, Oct 7, 2010 at 11:26 AM, Rainer Gerhards wrote: > Hi folks, > > I am happy to finally announce the liblognorm project, a facility to > normalize all kinds of log messages. My initial announcement is here: > > http://blog.gerhards.net/2010/10/introducing-liblognorm.html > > I'd appreciate if you follow this effort and share the news. I plan to do a > couple of more blog posts within the next days. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Oct 8 15:42:07 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 15:42:07 +0200 Subject: [rsyslog] Introducing liblognorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD487@GRFEXC.intern.adiscon.com> Aaron, as soon as I have something to publish. This depends a bit on the question if I do a PoC first or begin with a conceptual write-up (in which case the actual code comes a bit behind). I am wading through CEE docs right now, so that I see what actually fits and what not. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Wiebe > Sent: Friday, October 08, 2010 3:23 PM > To: rsyslog-users > Subject: Re: [rsyslog] Introducing liblognorm > > Rainer - when do you expect to be publishing a git repository? > > -Aaron > > On Thu, Oct 7, 2010 at 11:26 AM, Rainer Gerhards > wrote: > > Hi folks, > > > > I am happy to finally announce the liblognorm project, a facility to > > normalize all kinds of log messages. My initial announcement is here: > > > > http://blog.gerhards.net/2010/10/introducing-liblognorm.html > > > > I'd appreciate if you follow this effort and share the news. I plan > to do a > > couple of more blog posts within the next days. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From karsten.heymann at telecolumbus.net Fri Oct 8 15:19:17 2010 From: karsten.heymann at telecolumbus.net (Karsten Heymann) Date: Fri, 08 Oct 2010 15:19:17 +0200 Subject: [rsyslog] 5.4.0 hangs after several hours In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD482@GRFEXC.intern.adiscon.com> (Rainer Gerhards's message of "Fri, 8 Oct 2010 13:48:28 +0200") References: <87fwwhax1h.fsf@ara.blue-cable.net> <9B6E2A8877C38245BFB15CC491A11DA71DD482@GRFEXC.intern.adiscon.com> Message-ID: <87aamoakja.fsf@ara.blue-cable.net> Hi "Rainer Gerhards" writes: > I'd not bother looking into the 5.4.0 issue. The current beta most > probably fixes the issue (5.4.0 was not a very good build). The beta > is scheduled to become the new stable next week or so. Great news and it seems my workaround of not using /dev/xconsole does the trick (will be sure on monday). > Sorry that version did not work out well... No problem :-) Yours Karsten From champ at softwink.com Fri Oct 8 17:04:34 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 11:04:34 -0400 Subject: [rsyslog] liblognorm In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com> References: <20101008125509.GA1416@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com> Message-ID: <20101008150434.GB6987@bundy.vistech.net> > Starting this morning I have begun to wade in-depth through a pile of CEE > docs (the upcoming standard). We may need to make some format adjustments, > but I'd like to stick with somewhat as simple as possible to write for the > samples. Do you have a link to the CEE documentation for the upcoming standard? I'd be interested in taking a look. > > Will the log formats, that is, what to look for, > > be in a seperate text file that can be updated? As you mention in the > > blog post, "much like a virus scanner". > > Actually, at this point nothing is really finalized. For the initial effort, > it will definitely be a text file. But in the long term I can also envision > that we should be able to pull it e.g. from web servers or load multiple > files, or... actually something that should be configurable. IMHO, I think leaving it a text file would be best. Let some other process (wget, whatever) pull the definitions. Or let the user do it manually. Again, IMHO, I think having the actual library pull the definitions might be outside of the scope. > > > If that is the case, will programs that use liblognorm pull that into > > an array first, and let liblognorm do it's work? Or will this file > > have to be repeatedly referenced? > > In the long term I think it makes sense to provide a way to stuff samples in > via either a dedicated API OR (and?) have them loaded based on a config file. > The latter is what I have on my mind for the first step. Sounds great. > > I'd love to help out in this, be it code wise and/or log examples. > > Any help is deeply appreciated and these questions already help! Log samples > and trying out the library, once it materializes, would be a very very great > aid. > > I will write a couple of more blog posts with more details, and everything is > really open for change now. It would also be very useful if you could spread > the news. An as wide audience as possible would be very good. Sounds great as well. I know on this end of the wire, you've already got one beta tester :) I'm really itching to try it out, but I know it's not even near that point! I'll be more than happy to help out where I can (code, log examples, etc). -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Fri Oct 8 17:18:03 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 11:18:03 -0400 Subject: [rsyslog] Sagan + Rsyslog [$template question] In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> Message-ID: <20101008151803.GA7702@bundy.vistech.net> > This looks good. The only thing I see instantly is that we could spare a view > CPU cycles (with recent v5 builds) if we use a strgen module. A strgen > effectively builds the template string via C code instead of the template > processor and thus is a bit faster and more flexible. Nothing to really care > about for low to medium number of messages, but makes a small difference if > you have severl ten-thousand messages per second. It isn't hard to write one > and I think I could contribute it some time next week if you find it useful. I don't know if you've looked at the Sagan page, but I'm always about sparing a few CPU cycles :) I also read your paper on taking rsyslog from 40k to 250k of mps (can't recall the exact name). It's a nice write up, and certainly gave me some ideas about improving the preformance of Sagan. If you have some spare time to contribute the strgen processor, that'd be great! Do you have any 'example strgen' code I could look at? I might take a wack at it :) >> and %syslogtag% functionality seems to be a bit different. For >> example: >> >> Syslog-ng "$TAG" == 1e >> rsyslog-ng %syslogtag% == the program name (for example: 'rsync:') > I really don't know what syslog-ng does here. Is it really the string 1e or > the Ascii char 1e? If so, this is very strange to me. SYSLOG-TAG is defined > in RFC5424 and traditional meaning is described in RFC 3164. rsyslog uses > these definitions. But I don't know syslog-ng at all, so I do not know what > they call the "tag". But obviously they have applied a non-standard meaning > to this name. It's the string "1e". It's not always that, but it's basically a hex looking 'string'. It's not terribly important, as I don't think I have any rules for Sagan that actually use the 'tag'. I've been confused about that for a while, and I'm glad I'm not the only one :) -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Fri Oct 8 18:34:11 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 18:34:11 +0200 Subject: [rsyslog] liblognorm References: <20101008125509.GA1416@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com> <20101008150434.GB6987@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD489@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Friday, October 08, 2010 5:05 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm > > > Starting this morning I have begun to wade in-depth through a pile of > CEE > > docs (the upcoming standard). We may need to make some format > adjustments, > > but I'd like to stick with somewhat as simple as possible to write > for the > > samples. > > Do you have a link to the CEE documentation for the upcoming > standard? I'd be interested in taking a look. Unfortunately the public site currently has only a limited selection. I need to check if I can share some of the current drafts. The Dictionary and Taxonmy stuff is most interesting. Anyhow, the public site definitely helps getting the idea: http://cee.mitre.org/ > > > Will the log formats, that is, what to look for, > > > be in a seperate text file that can be updated? As you mention in > the > > > blog post, "much like a virus scanner". > > > > Actually, at this point nothing is really finalized. For the initial > effort, > > it will definitely be a text file. But in the long term I can also > envision > > that we should be able to pull it e.g. from web servers or load > multiple > > files, or... actually something that should be configurable. > > IMHO, I think leaving it a text file would be best. Let some > other process (wget, whatever) pull the definitions. Or let the user > do it manually. Again, IMHO, I think having the actual library > pull > the definitions might be outside of the scope. That sounds good. The only thing that I am pretty sure about is that - at some stage - we must support *multiple* files. That is because I envision that some may be pulled from a global repository but some local-only may also exist. I think it is easier to manage those if they can be kept in different files. > > > > > If that is the case, will programs that use liblognorm pull that > into > > > an array first, and let liblognorm do it's work? Or will this > file > > > have to be repeatedly referenced? > > > > In the long term I think it makes sense to provide a way to stuff > samples in > > via either a dedicated API OR (and?) have them loaded based on a > config file. > > The latter is what I have on my mind for the first step. > > Sounds great. > > > > I'd love to help out in this, be it code wise and/or log examples. > > > > Any help is deeply appreciated and these questions already help! Log > samples > > and trying out the library, once it materializes, would be a very > very great > > aid. > > > > I will write a couple of more blog posts with more details, and > everything is > > really open for change now. It would also be very useful if you could > spread > > the news. An as wide audience as possible would be very good. > > Sounds great as well. I know on this end of the wire, you've > already got one beta tester :) I'm really itching to try it out, but > I > know it's not even near that point! I'll be more than happy to help > out where I can (code, log examples, etc). Looks like we are in business ;) I need to digest what I have read today, but it looks like I will begin to create some skeleton code next week, starting with the build system and followed by some important definitions (e.g. for tags, fields and so on). Feedback on the organization of this material is also appreciated. I'll populate the public git as soon as I have some lines of code ;) Rainer From matteobattistini at gmail.com Fri Oct 8 18:34:56 2010 From: matteobattistini at gmail.com (Matteo Battistini) Date: Fri, 8 Oct 2010 18:34:56 +0200 Subject: [rsyslog] mysql log rotation best practices Message-ID: I all, my log (db) is every day 200MB bigger i need 4 month rotation Suggested best practices? delete ... from ... where or table rotation (create/drop) or ? anyone with sone simple crontab script? thanks From rgerhards at hq.adiscon.com Fri Oct 8 18:37:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 8 Oct 2010 18:37:54 +0200 Subject: [rsyslog] Sagan + Rsyslog [$template question] References: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> <20101008151803.GA7702@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD48A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Friday, October 08, 2010 5:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] Sagan + Rsyslog [$template question] > > > > This looks good. The only thing I see instantly is that we could > spare a view > > CPU cycles (with recent v5 builds) if we use a strgen module. A > strgen > > effectively builds the template string via C code instead of the > template > > processor and thus is a bit faster and more flexible. Nothing to > really care > > about for low to medium number of messages, but makes a small > difference if > > you have severl ten-thousand messages per second. It isn't hard to > write one > > and I think I could contribute it some time next week if you find it > useful. > > I don't know if you've looked at the Sagan page, but I'm always > about sparing a few CPU cycles :) I did, and this was the reason I mentioned strgens ;) > I also read your paper on taking > rsyslog from 40k to 250k of mps (can't recall the exact name). It's > a nice write up, and certainly gave me some ideas about improving the > preformance of Sagan. Thanks! If you think it is useful, please pass it on. My hope is that it will be useful especially in this transition phase from single- to multi-threaded paradigms. > > If you have some spare time to contribute the strgen processor, > that'd be great! Do you have any 'example strgen' code I could look > at? > I might take a wack at it :) They are *very* simple. The one used for the regular flat file format is here: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=tools/smtradfile.c;h=eff2f99a6 4f8f5da69855a4df1bf8d5e8ea2fb02;hb=HEAD This is a built-in module, the plumbing is very slightly different for loadable ones - but I can do a conversion in 2 minutes ;) > >> and %syslogtag% functionality seems to be a bit different. For > >> example: > >> > >> Syslog-ng "$TAG" == 1e > >> rsyslog-ng %syslogtag% == the program name (for example: 'rsync:') > > > > I really don't know what syslog-ng does here. Is it really the string > 1e or > > the Ascii char 1e? If so, this is very strange to me. SYSLOG-TAG is > defined > > in RFC5424 and traditional meaning is described in RFC 3164. rsyslog > uses > > these definitions. But I don't know syslog-ng at all, so I do not > know what > > they call the "tag". But obviously they have applied a non-standard > meaning > > to this name. > > It's the string "1e". It's not always that, but it's basically > a hex looking 'string'. It's not terribly important, as I don't > think > I have any rules for Sagan that actually use the 'tag'. I've been > confused > about that for a while, and I'm glad I'm not the only one :) Lol ;) Rainer > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Fri Oct 8 18:57:04 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 8 Oct 2010 12:57:04 -0400 Subject: [rsyslog] liblognorm In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD489@GRFEXC.intern.adiscon.com> References: <20101008125509.GA1416@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com> <20101008150434.GB6987@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD489@GRFEXC.intern.adiscon.com> Message-ID: <20101008165704.GA10906@bundy.vistech.net> > http://cee.mitre.org/ Yep.. I've read through that. I was looking for something more "meaty". :) > That sounds good. The only thing that I am pretty sure about is that - at > some stage - we must support *multiple* files. That is because I envision > that some may be pulled from a global repository but some local-only may also > exist. I think it is easier to manage those if they can be kept in different > files. That's a interesting concept, and pretty much how we do it with Sagan/Snort. In the Sagan configuration file, you have lines like this: include $RULE_PATH/rsync.rules include $RULE_PATH/samba.rules include $RULE_PATH/sendmail.rules If you don't use "sendmail", you can "# out" that rule. There's not much need to "monitor" for things that you don't expect to see. The same could apply to liblognorm ... That way, you could also include "local" definitions. Here's how I'm looking to use something like liblognorm. I'd actually already started on some simple parsers, but would rather see something like liblognorm (keeps from re-inventing the wheel, and useful for many projects). Take the following "openssh.rules" line: alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (msg:"[OPENSSH] Authentication failure for root"; content: "Authentication failure for root"; classtype: unsuccessful-admin;program: sshd; threshold:type limit, track by_src, count 5, seconds 300; parse_ip_simple; parse_port_simple; reference: url,wiki.softwink.com/bin/view/Main/5000017; sid: 5000017; rev:1;) Note the parse_ip_simple and parse_port_simple. Those are my current, simple, parsers to pull IP address and TCP source port information (when applicable). Replace those calls with the liblognorm. That's the goal, across many different log sets (Cisco, Fortigate firewalls, Linux boxes) that I'm looking for. Basically, "ip_parse_simple" becomes a rule flag I can pass to liblognorm, which "tells" liblognorm", "this is a openssh message" and "extract the source IP address and source port". Does that seem on track? Sorry for the rant.... > Looks like we are in business ;) I need to digest what I have read today, but > it looks like I will begin to create some skeleton code next week, starting > with the build system and followed by some important definitions (e.g. for > tags, fields and so on). Feedback on the organization of this material is > also appreciated. I'll populate the public git as soon as I have some lines > of code ;) No problem. These things take time :) -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Sun Oct 10 11:53:06 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 10 Oct 2010 11:53:06 +0200 Subject: [rsyslog] liblognorm vs. libeventnorm Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> Hi all, I think I'll start with the libeventnorm name for the normalizing library instead of liblognorm. Reason here: http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html Further name suggestions or arguments are very welcome! Rainer From rgerhards at hq.adiscon.com Mon Oct 11 09:01:02 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 09:01:02 +0200 Subject: [rsyslog] liblognorm vs. libeventnorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD491@GRFEXC.intern.adiscon.com> I would like to add as an argument pro liblognorm, that many people probably better understand what "log normalization" is whereas "event normalization" may sound strange. In that sense, liblognorm may be a better name. Feedback is appreciated. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Sunday, October 10, 2010 11:53 AM > To: rsyslog-users > Subject: [rsyslog] liblognorm vs. libeventnorm > > Hi all, > > I think I'll start with the libeventnorm name for the normalizing > library > instead of liblognorm. Reason here: > > http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html > > Further name suggestions or arguments are very welcome! > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Oct 11 12:37:22 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 12:37:22 +0200 Subject: [rsyslog] liblognorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD491@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD496@GRFEXC.intern.adiscon.com> I have just written another post on the normalization library. It looks like the design tends to favor a split into two libraries: http://blog.gerhards.net/2010/10/splitting-up-normalization-library.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, October 11, 2010 9:01 AM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm vs. libeventnorm > > I would like to add as an argument pro liblognorm, that many people > probably > better understand what "log normalization" is whereas "event > normalization" > may sound strange. In that sense, liblognorm may be a better name. > Feedback > is appreciated. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Sunday, October 10, 2010 11:53 AM > > To: rsyslog-users > > Subject: [rsyslog] liblognorm vs. libeventnorm > > > > Hi all, > > > > I think I'll start with the libeventnorm name for the normalizing > > library > > instead of liblognorm. Reason here: > > > > http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html > > > > Further name suggestions or arguments are very welcome! > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Mon Oct 11 12:42:04 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 12:42:04 +0200 Subject: [rsyslog] liblognorm References: <20101008125509.GA1416@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD484@GRFEXC.intern.adiscon.com><20101008150434.GB6987@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD489@GRFEXC.intern.adiscon.com> <20101008165704.GA10906@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD498@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Friday, October 08, 2010 6:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm > > > http://cee.mitre.org/ > > Yep.. I've read through that. I was looking for something more > "meaty". :) > I understand, but right now I cannot provide more due to NDA's. I am even going on a border when I begin to write the class definitions (and so I will check with Mitre first). > > That sounds good. The only thing that I am pretty sure about is that > - at > > some stage - we must support *multiple* files. That is because I > envision > > that some may be pulled from a global repository but some local-only > may also > > exist. I think it is easier to manage those if they can be kept in > different > > files. > > That's a interesting concept, and pretty much how we do it with > Sagan/Snort. In the Sagan configuration file, you have lines like > this: > > include $RULE_PATH/rsync.rules > include $RULE_PATH/samba.rules > include $RULE_PATH/sendmail.rules > > If you don't use "sendmail", you can "# out" that rule. > There's not much need to "monitor" for things that you don't expect to > see. > The same could apply to liblognorm ... That way, you could also > include > "local" definitions. Yup > > Here's how I'm looking to use something like liblognorm. I'd > actually already started on some simple parsers, but would rather see > something like liblognorm (keeps from re-inventing the wheel, and > useful for many projects). > > Take the following "openssh.rules" line: > > alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (msg:"[OPENSSH] > Authentication failure for root"; content: "Authentication failure for > root"; classtype: unsuccessful-admin;program: sshd; threshold:type > limit, track by_src, count 5, seconds 300; parse_ip_simple; > parse_port_simple; reference: > url,wiki.softwink.com/bin/view/Main/5000017; sid: 5000017; rev:1;) > > Note the parse_ip_simple and parse_port_simple. Those are my > current, simple, parsers to pull IP address and TCP source port > information (when applicable). Replace those calls with the > liblognorm. That's the goal, across many different log sets (Cisco, > Fortigate firewalls, Linux boxes) that I'm looking for. > > Basically, "ip_parse_simple" becomes a rule flag I can pass to > liblognorm, which "tells" liblognorm", "this is a openssh message" > and > "extract the source IP address and source port". > > Does that seem on track? Sorry for the rant.... Exactly the same idea. Maybe we can share some of the code! Rainer From marcin at mejor.pl Mon Oct 11 13:05:02 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 13:05:02 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin Message-ID: <4CB2EF5E.3060504@mejor.pl> Hello! I'm using git version of rsyslog. I have problem with imuxsock, i'm getting this message: " could not load module '/usr/lib/rsyslog/imuxsock.so', dlopen: /usr/lib/rsyslog/imuxsock.so: undefined symbol: ceilf " BTW, is this something to worry?: Message from portage: " QA Notice: Package has poor programming practices which may compile * fine but exhibit random runtime failures. * net.c:1191: warning: dereferencing pointer ?ipv6? does break strict-aliasing rules " Thanks for help, Marcin From rgerhards at hq.adiscon.com Mon Oct 11 13:15:39 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 13:15:39 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 1:05 PM > To: rsyslog-users > Subject: [rsyslog] Problem with loading imuxsock plugin > > Hello! > I'm using git version of rsyslog. > I have problem with imuxsock, i'm getting this message: " > could not load module '/usr/lib/rsyslog/imuxsock.so', dlopen: > /usr/lib/rsyslog/imuxsock.so: undefined symbol: ceilf > " This looks like a the math library was not part of the link process. Do you use an older commit? (This problem was there, but I fixed it -- at least it looked so ;)). > > BTW, is this something to worry?: > Message from portage: > " QA Notice: Package has poor programming practices which may compile > * fine but exhibit random runtime failures. > * net.c:1191: warning: dereferencing pointer ?ipv6? does break > strict-aliasing rules > " Well, whomever's QA would be better of providing a patch than writing this ;) Actually I tried to get rid of the compiler warning, but could not find any solution to this use case. I'd appreciate I you pressed whoever wrote this ro provide some real assistance instead of spreading FUD ;) Rainer > > Thanks for help, > Marcin > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marcin at mejor.pl Mon Oct 11 13:33:25 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 13:33:25 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> Message-ID: <4CB2F605.6050405@mejor.pl> W dniu 11.10.2010 13:15, Rainer Gerhards pisze: >> could not load module '/usr/lib/rsyslog/imuxsock.so', dlopen: >> /usr/lib/rsyslog/imuxsock.so: undefined symbol: ceilf >> " > > This looks like a the math library was not part of the link process. Do you > use an older commit? (This problem was there, but I fixed it -- at least it > looked so ;)). # rsyslogd -v rsyslogd 6.1.1, compiled with: [...] commit: f60fd65378d4bfc110e8ae1b55197d73c3293f0c It should be newest commit. >> BTW, is this something to worry?: >> Message from portage: >> " QA Notice: Package has poor programming practices which may compile >> * fine but exhibit random runtime failures. >> * net.c:1191: warning: dereferencing pointer ?ipv6? does break >> strict-aliasing rules >> " > > Well, whomever's QA would be better of providing a patch than writing this ;) > Actually I tried to get rid of the compiler warning, but could not find any > solution to this use case. I'd appreciate I you pressed whoever wrote this ro > provide some real assistance instead of spreading FUD ;) This is automatic QA warning, we can't expect any help. Sorry:) From mbiebl at gmail.com Mon Oct 11 13:39:14 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 11 Oct 2010 13:39:14 +0200 Subject: [rsyslog] liblognorm vs. libeventnorm In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> Message-ID: 2010/10/10 Rainer Gerhards : > Further name suggestions or arguments are very welcome! libcee -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Mon Oct 11 13:58:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 13:58:37 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 1:33 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Problem with loading imuxsock plugin > > W dniu 11.10.2010 13:15, Rainer Gerhards pisze: > >> could not load module '/usr/lib/rsyslog/imuxsock.so', dlopen: > >> /usr/lib/rsyslog/imuxsock.so: undefined symbol: ceilf > >> " > > > > This looks like a the math library was not part of the link process. > Do you > > use an older commit? (This problem was there, but I fixed it -- at > least it > > looked so ;)). > > # rsyslogd -v > rsyslogd 6.1.1, compiled with: > [...] mhhh... which platform? > > commit: f60fd65378d4bfc110e8ae1b55197d73c3293f0c > It should be newest commit. > > >> BTW, is this something to worry?: > >> Message from portage: > >> " QA Notice: Package has poor programming practices which may > compile > >> * fine but exhibit random runtime failures. > >> * net.c:1191: warning: dereferencing pointer ?ipv6? does break > >> strict-aliasing rules > >> " > > > > Well, whomever's QA would be better of providing a patch than writing > this ;) > > Actually I tried to get rid of the compiler warning, but could not > find any > > solution to this use case. I'd appreciate I you pressed whoever wrote > this ro > > provide some real assistance instead of spreading FUD ;) > > This is automatic QA warning, we can't expect any help. Sorry:) what a shame -- I thought I finally found someone able to fix that thing ;) Rainer From rgerhards at hq.adiscon.com Mon Oct 11 13:59:56 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 13:59:56 +0200 Subject: [rsyslog] liblognorm vs. libeventnorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49C@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Monday, October 11, 2010 1:39 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm vs. libeventnorm > > 2010/10/10 Rainer Gerhards : > > Further name suggestions or arguments are very welcome! > > libcee sounds decent, especially for the CEE part. Need to check with mitre, which owns a TM on the term CEE. Maybe the plan to permit this only based on conformance level (what is hard to guess when the standard is not yet out ;)). Rainer From marcin at mejor.pl Mon Oct 11 14:06:48 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 14:06:48 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> Message-ID: <4CB2FDD8.60802@mejor.pl> W dniu 11.10.2010 13:58, Rainer Gerhards pisze: >> # rsyslogd -v >> rsyslogd 6.1.1, compiled with: >> [...] > > mhhh... which platform? $ uname -srmo Linux 2.6.35-hardened-r2 i686 GNU/Linux gcc version 4.4.4 (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) From rgerhards at hq.adiscon.com Mon Oct 11 14:08:32 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 14:08:32 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 2:07 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Problem with loading imuxsock plugin > > W dniu 11.10.2010 13:58, Rainer Gerhards pisze: > >> # rsyslogd -v > >> rsyslogd 6.1.1, compiled with: > >> [...] > > > > mhhh... which platform? > > $ uname -srmo > Linux 2.6.35-hardened-r2 i686 GNU/Linux > > gcc version 4.4.4 (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) where can I download this? Rainer From marcin at mejor.pl Mon Oct 11 14:18:13 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 14:18:13 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> Message-ID: <4CB30085.6030800@mejor.pl> W dniu 11.10.2010 14:08, Rainer Gerhards pisze: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw >> Sent: Monday, October 11, 2010 2:07 PM >> To: rsyslog at lists.adiscon.com >> Subject: Re: [rsyslog] Problem with loading imuxsock plugin >> >> W dniu 11.10.2010 13:58, Rainer Gerhards pisze: >>>> # rsyslogd -v >>>> rsyslogd 6.1.1, compiled with: >>>> [...] >>> >>> mhhh... which platform? >> >> $ uname -srmo >> Linux 2.6.35-hardened-r2 i686 GNU/Linux >> >> gcc version 4.4.4 (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) > where can I download this? This is gentoo, you need built gcc from sources, plus patches: http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.3-specs-0.2.0.tar.bz2 http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-piepatches-v0.4.5.tar.bz2 http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-patches-1.2.tar.bz2 http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-uclibc-patches-1.0.tar.bz2 or i can create tarrball with compilled version, from live system. From rgerhards at hq.adiscon.com Mon Oct 11 14:21:09 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 14:21:09 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com> OK, if it is basically Gentoo, I'll start from there. What happens here is that ceil() from the math lib is used. This is being utilized by the new hash table code to calculate a table size which has 65% free space (via floats) if the table needs to be extended. I could probably remove that from the hash table code that I use, but so far I found it useful (and the extra overhead should not really be noticable, I thought). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 2:18 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Problem with loading imuxsock plugin > > W dniu 11.10.2010 14:08, Rainer Gerhards pisze: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > >> Sent: Monday, October 11, 2010 2:07 PM > >> To: rsyslog at lists.adiscon.com > >> Subject: Re: [rsyslog] Problem with loading imuxsock plugin > >> > >> W dniu 11.10.2010 13:58, Rainer Gerhards pisze: > >>>> # rsyslogd -v > >>>> rsyslogd 6.1.1, compiled with: > >>>> [...] > >>> > >>> mhhh... which platform? > >> > >> $ uname -srmo > >> Linux 2.6.35-hardened-r2 i686 GNU/Linux > >> > >> gcc version 4.4.4 (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) > > where can I download this? > > This is gentoo, you need built gcc from sources, plus patches: > http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.3-specs- > 0.2.0.tar.bz2 > http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-piepatches- > v0.4.5.tar.bz2 > http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-patches- > 1.2.tar.bz2 > http://trumpetti.atm.tut.fi/gentoo/distfiles/gcc-4.4.4-uclibc-patches- > 1.0.tar.bz2 > > or i can create tarrball with compilled version, from live system. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From marcin at mejor.pl Mon Oct 11 14:48:20 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 14:48:20 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com> Message-ID: <4CB30794.8030909@mejor.pl> W dniu 11.10.2010 14:21, Rainer Gerhards pisze: > I could probably remove that from the hash > table code that I use, but so far I found it useful (and the extra overhead > should not really be noticable, I thought). This is quite drastic solution;) Isn't there another way? From dev.fanooos at gmail.com Mon Oct 11 14:58:24 2010 From: dev.fanooos at gmail.com (=?UTF-8?B?2KPZhtizINin2YTZhNmK2KvZig==?=) Date: Mon, 11 Oct 2010 10:58:24 -0200 Subject: [rsyslog] Question about using rsyslog in a central logging system In-Reply-To: References: Message-ID: Dears: I have a little question I hope you can help me I am using rsyslog in a central logging system for my company. rsyslog is installed on various machine and send their logs via RELP to a central server. The central server stores the logs into a mysql DB. In the DB I can identify which record belongs to a specific machine using the FromHost column. My question: Is there a way to configure rsyslog on the client machine to send their logs with a specific host name and get that name stored in the FromHost column in the DB ? Thanks in advance and best regards -- Anas Rabei Software developer EDC(Egypt Deveopment Center) anas.rabei at egyptdc.com From rgerhards at hq.adiscon.com Mon Oct 11 15:47:09 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 15:47:09 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com> <4CB30794.8030909@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD49F@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 2:48 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Problem with loading imuxsock plugin > > W dniu 11.10.2010 14:21, Rainer Gerhards pisze: > > I could probably remove that from the hash > > table code that I use, but so far I found it useful (and the extra > overhead > > should not really be noticable, I thought). > > This is quite drastic solution;) Isn't there another way? I meant this as a last resort, if the math lib is not available. But thinking twice about it, I think this can really be done without any loss of functionality. Let me check... From rgerhards at hq.adiscon.com Mon Oct 11 16:04:58 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 16:04:58 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com><4CB30794.8030909@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4A0@GRFEXC.intern.adiscon.com> > > > I could probably remove that from the hash > > > table code that I use, but so far I found it useful (and the extra > > overhead > > > should not really be noticable, I thought). > > > > This is quite drastic solution;) Isn't there another way? > > I meant this as a last resort, if the math lib is not available. But > thinking > twice about it, I think this can really be done without any loss of > functionality. Let me check... yup, simple calculations do the job. The code currently is: ceil(unsigned * 0.65) which equals ((unsigned long long) unsigned * 65) / 100 where "/" is the integer division which ignores the reminder. So it is easy to do this without float's, what obviously removes the need for ceil and thus the math lib. Will patch soon... Thanks for bringing this to my attention! Rainer From marcin at mejor.pl Mon Oct 11 16:10:55 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 11 Oct 2010 16:10:55 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD4A0@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com><4CB30794.8030909@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD4A0@GRFEXC.intern.adiscon.com> Message-ID: <4CB31AEF.10705@mejor.pl> W dniu 11.10.2010 16:04, Rainer Gerhards pisze: >> I meant this as a last resort, if the math lib is not available. Shouldn't be any error while compilling? > yup, simple calculations do the job. The code currently is: > > ceil(unsigned * 0.65) > > which equals > > ((unsigned long long) unsigned * 65) / 100 heh, let's go back to math from primary school :D > Will patch soon... I'm waiting. Thanks for quick response! Regards From rgerhards at hq.adiscon.com Mon Oct 11 17:13:39 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 11 Oct 2010 17:13:39 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com><4CB30794.8030909@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD4A0@GRFEXC.intern.adiscon.com> <4CB31AEF.10705@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4A1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Monday, October 11, 2010 4:11 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Problem with loading imuxsock plugin > > W dniu 11.10.2010 16:04, Rainer Gerhards pisze: > >> I meant this as a last resort, if the math lib is not available. > > Shouldn't be any error while compilling? no, the compiler generates the proper function call, but the linker can not find the member. > > > yup, simple calculations do the job. The code currently is: > > > > ceil(unsigned * 0.65) > > > > which equals > > > > ((unsigned long long) unsigned * 65) / 100 > > heh, let's go back to math from primary school :D > > > Will patch soon... > > I'm waiting. Thanks for quick response! http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8c3d40b798e3ed68fb03629d 87b55140cb6bc044 This is for v5-devel, but should apply to v6 as well. I have not yet merged this up to not clutter the git log so much. Let me know if you have problems, I'll then merge. Rainer From marcin at mejor.pl Mon Oct 11 18:14:50 2010 From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=) Date: Mon, 11 Oct 2010 18:14:50 +0200 Subject: [rsyslog] Problem with loading imuxsock plugin In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD4A1@GRFEXC.intern.adiscon.com> References: <4CB2EF5E.3060504@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49A@GRFEXC.intern.adiscon.com> <4CB2F605.6050405@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49B@GRFEXC.intern.adiscon.com> <4CB2FDD8.60802@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49D@GRFEXC.intern.adiscon.com> <4CB30085.6030800@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA71DD49E@GRFEXC.intern.adiscon.com><4CB30794.8030909@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD49F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD4A0@GRFEXC.intern.adiscon.com> <4CB31AEF.10705@mejor.pl> <9B6E2A8877C38245BFB15CC491A11DA71DD4A1@GRFEXC.intern.adiscon.com> Message-ID: <4CB337FA.2010302@mejor.pl> W dniu 2010-10-11 17:13, Rainer Gerhards pisze: > This is for v5-devel, but should apply to v6 as well. I have not yet merged > this up to not clutter the git log so much. Let me know if you have problems, > I'll then merge. Works without any problem:) Thank You! From david at lang.hm Mon Oct 11 22:33:18 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 11 Oct 2010 13:33:18 -0700 (PDT) Subject: [rsyslog] liblognorm In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD496@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD491@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD496@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 11 Oct 2010, Rainer Gerhards wrote: > I have just written another post on the normalization library. It looks like > the design tends to favor a split into two libraries: > > http://blog.gerhards.net/2010/10/splitting-up-normalization-library.html this seems like a good idea. there is a definate need for a good, efficient parsing tool that can be used for high volume sites. There are a lot of tools that heavily use regex matching, but those tend to collapse at high volumes. you can create your own parser with lex, yacc, bison, or flex, but th work needed to create the input config file for these (with their specific syntax) is daunting. a tool that could take it's configuration in something that looks very similar to log lines (with some sort of syntax to show the variable part), that would then compile into something very effient like the tools above would be very useful for a lot of different tools. this may just need to be a configuration generator for the tools listed above that can take the list of annotated lines and create the appropriate config file to build the parser. If this can accept regex lines and then compile them down to a parser tree it would be wonderful. so once there is a high performance parser to pull the data apart, then the question is what to do with it. some people will want to write it to various places, others will want to make decisions based on what is matched. for those who are wanting to write the normalized output to various places, a plugin structure like rsyslog has (with the ability to format the messages based on the various properties that are discovered) is very appealing, and it may make a lot of sense to see what can be done to re-use that work. If so, there will need to be a 'format string' that creates the output with all the properties that are known tagged, but without including ones that didn't have any matches in this log message. for those who are wanting to then implement logic based on what it gets, thing get much more interesting. I suspect that the thing to do here will be to make the event normalization engine be something that can be a library included in other programs (in various languages), something so that you can have the config file be something along the lines of documentation (hopefully including a sample raw line) line-to-match function to call when matched there are a log of programs out there written to do good and interesting stuff with lines that it receives, if there was an ability to replace their sequential 'does it match rule 1, does it match rule 2' logic with a more efficient parser it would be a huge win. I don't think you are wanting to tackle that portion of the task. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Monday, October 11, 2010 9:01 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] liblognorm vs. libeventnorm >> >> I would like to add as an argument pro liblognorm, that many people >> probably >> better understand what "log normalization" is whereas "event >> normalization" >> may sound strange. In that sense, liblognorm may be a better name. >> Feedback >> is appreciated. >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>> Sent: Sunday, October 10, 2010 11:53 AM >>> To: rsyslog-users >>> Subject: [rsyslog] liblognorm vs. libeventnorm >>> >>> Hi all, >>> >>> I think I'll start with the libeventnorm name for the normalizing >>> library >>> instead of liblognorm. Reason here: >>> >>> http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html >>> >>> Further name suggestions or arguments are very welcome! >>> >>> Rainer >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Oct 12 13:41:38 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 12 Oct 2010 13:41:38 +0200 Subject: [rsyslog] Introducing liblognorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD477@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4B1@GRFEXC.intern.adiscon.com> > Rainer - when do you expect to be publishing a git repository? An initial repository is now online: http://git.adiscon.com/?p=liblognorm.git;a=summary Most importantly, is has a glimpse of the API available at http://git.adiscon.com/?p=liblognorm.git;a=blob;f=src/liblognorm.h;h=97279a8f 0729fa58229e81512eda4f26c7e3e4b2;hb=422c4cf7e848f18a09ea1857177a7ce5731f6077 Almost all real plumbing is still missing ;) Rainer From karsten.heymann at telecolumbus.net Tue Oct 12 16:59:16 2010 From: karsten.heymann at telecolumbus.net (Karsten Heymann) Date: Tue, 12 Oct 2010 16:59:16 +0200 Subject: [rsyslog] rsyslog hangs when tcp loghost is unreachable Message-ID: <87bp6zv4mz.fsf@ara.blue-cable.net> Hi, today I stumbled over a strange problem: When forwarding logs via tcp to a remote logserver, and the logserver is unreachable, local logging is delayed up to an half hour. Config: $ModLoad imuxsock # provides support for local system logging $ModLoad imklog # provides kernel logging support (previously done by rklogd) $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 *.* @@remote-loghost-ip:10514 $ActionExecOnlyWhenPreviousIsSuspended on & /var/log/localbuffer $ActionExecOnlyWhenPreviousIsSuspended off auth,authpriv.* /var/log/auth.log [... standard local log rules snipped ...] I tested with both 4.6.4-1~bpo50+1 and 3.18.6-4, which are the versions easily available on debian lenny (+backport). Yours Karsten From rgerhards at hq.adiscon.com Wed Oct 13 12:56:58 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 13 Oct 2010 12:56:58 +0200 Subject: [rsyslog] liblognorm will use UTF-8 / passive mode Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4C7@GRFEXC.intern.adiscon.com> Some information here: http://blog.gerhards.net/2010/10/liblognorm-will-use-passive-unicode.html Once this effort is done, I will also do a review of rsyslog on where (if at all) I need to make changes for full passive support of UTF-8. Probably the only one required is BOM support for RFC5424. Rainer From rgerhards at hq.adiscon.com Wed Oct 13 14:44:43 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 13 Oct 2010 14:44:43 +0200 Subject: [rsyslog] liblognorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD491@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD496@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4CA@GRFEXC.intern.adiscon.com> Hi David, as usual, many thanks for your great thoughts. I had a day of heavy hacking yesterday, thus the late response. See below... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Monday, October 11, 2010 10:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm > > On Mon, 11 Oct 2010, Rainer Gerhards wrote: > > > I have just written another post on the normalization library. It > looks like > > the design tends to favor a split into two libraries: > > > > http://blog.gerhards.net/2010/10/splitting-up-normalization- > library.html > > this seems like a good idea. > > there is a definate need for a good, efficient parsing tool that can be > used for high volume sites. There are a lot of tools that heavily use > regex matching, but those tend to collapse at high volumes. > > you can create your own parser with lex, yacc, bison, or flex, but th > work > needed to create the input config file for these (with their specific > syntax) is daunting. > > a tool that could take it's configuration in something that looks very > similar to log lines (with some sort of syntax to show the variable > part), > that would then compile into something very effient like the tools > above > would be very useful for a lot of different tools. That's the basic idea, instead that I do not intend to create e.g. lex source but rather have the engine do that part itself. The main advantage is that this could be done dynamically. I think this will be possible in almost constant time, as long as all fields can be parsed via primitive types (which do not require too much effort to back off). I will be working on the parse tree the next days, so you'll hopefully be able to get an idea of it by looking at the code. At it's heart, it simply is a radix tree, with constants and field syntaxes definig how the tree is traversed. > > this may just need to be a configuration generator for the tools listed > above that can take the list of annotated lines and create the > appropriate > config file to build the parser. If this can accept regex lines and > then compile them down to a parser tree it would be wonderful. Regex is a different beast, because for it you need to create a full-blown DFA, which also explains the slowness of regexes. I'll not tackle that beast. For some fields, I will support regex matches, but when they are used, performance is affected. The overall idea is that you usually do not need any regex/DFA at all. > > so once there is a high performance parser to pull the data apart, then > the question is what to do with it. > > some people will want to write it to various places, others will want > to > make decisions based on what is matched. > > for those who are wanting to write the normalized output to various > places, a plugin structure like rsyslog has (with the ability to format > the messages based on the various properties that are discovered) is > very > appealing, and it may make a lot of sense to see what can be done to > re-use that work. If so, there will need to be a 'format string' that > creates the output with all the properties that are known tagged, but > without including ones that didn't have any matches in this log > message. One thing I definitely intend to do is utilize the library in rsyslog. I envision a parser module that works based on the library. That also means rsyslog's core engine must be extended to support the additional fields, but something that can definitely be done. With that approach, no complex output engine is needed - one can just use the rsyslog plugin. And with a near O(1) algorithm, we can probably expect that this happens in real-time even for very large traffic loads (but probably not for the largest ones). It is important to know here that the current parsers also have some limited backout needs, for example for the date and tag/hostname fields. So this can be done quickly. > > for those who are wanting to then implement logic based on what it > gets, > thing get much more interesting. I suspect that the thing to do here > will > be to make the event normalization engine be something that can be a > library included in other programs (in various languages), something so > that you can have the config file be something along the lines of that's actually the idea. An initial sketch of the API is already in git and I hope to get some better-readbly doxgen-generated interface spec up later today. > > documentation (hopefully including a sample raw line) > line-to-match > function to call when matched > > there are a log of programs out there written to do good and > interesting > stuff with lines that it receives, if there was an ability to replace > their sequential 'does it match rule 1, does it match rule 2' logic > with a > more efficient parser it would be a huge win. > > I don't think you are wanting to tackle that portion of the task. The lib part yes, but that obviously requires a lot of changes for applications using it. Rainer > > David Lang > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >> Sent: Monday, October 11, 2010 9:01 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] liblognorm vs. libeventnorm > >> > >> I would like to add as an argument pro liblognorm, that many people > >> probably > >> better understand what "log normalization" is whereas "event > >> normalization" > >> may sound strange. In that sense, liblognorm may be a better name. > >> Feedback > >> is appreciated. > >> > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>> Sent: Sunday, October 10, 2010 11:53 AM > >>> To: rsyslog-users > >>> Subject: [rsyslog] liblognorm vs. libeventnorm > >>> > >>> Hi all, > >>> > >>> I think I'll start with the libeventnorm name for the normalizing > >>> library > >>> instead of liblognorm. Reason here: > >>> > >>> http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html > >>> > >>> Further name suggestions or arguments are very welcome! > >>> > >>> Rainer > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 13 15:29:02 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 13 Oct 2010 15:29:02 +0200 Subject: [rsyslog] liblognorm References: <9B6E2A8877C38245BFB15CC491A11DA71DD48B@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD491@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD496@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD4CA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4CC@GRFEXC.intern.adiscon.com> I have put a copy of the liblognorm API doc (temporarily) up on: http://www.gerhards.net/liblognorm/liblognorm_8h.html Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 13, 2010 2:45 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm > > Hi David, > > as usual, many thanks for your great thoughts. I had a day of heavy > hacking > yesterday, thus the late response. See below... > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Monday, October 11, 2010 10:33 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] liblognorm > > > > On Mon, 11 Oct 2010, Rainer Gerhards wrote: > > > > > I have just written another post on the normalization library. It > > looks like > > > the design tends to favor a split into two libraries: > > > > > > http://blog.gerhards.net/2010/10/splitting-up-normalization- > > library.html > > > > this seems like a good idea. > > > > there is a definate need for a good, efficient parsing tool that can > be > > used for high volume sites. There are a lot of tools that heavily use > > regex matching, but those tend to collapse at high volumes. > > > > you can create your own parser with lex, yacc, bison, or flex, but th > > work > > needed to create the input config file for these (with their specific > > syntax) is daunting. > > > > a tool that could take it's configuration in something that looks > very > > similar to log lines (with some sort of syntax to show the variable > > part), > > that would then compile into something very effient like the tools > > above > > would be very useful for a lot of different tools. > > That's the basic idea, instead that I do not intend to create e.g. lex > source > but rather have the engine do that part itself. The main advantage is > that > this could be done dynamically. I think this will be possible in almost > constant time, as long as all fields can be parsed via primitive types > (which > do not require too much effort to back off). > > I will be working on the parse tree the next days, so you'll hopefully > be > able to get an idea of it by looking at the code. At it's heart, it > simply is > a radix tree, with constants and field syntaxes definig how the tree is > traversed. > > > > > this may just need to be a configuration generator for the tools > listed > > above that can take the list of annotated lines and create the > > appropriate > > config file to build the parser. If this can accept regex lines and > > then compile them down to a parser tree it would be wonderful. > > Regex is a different beast, because for it you need to create a full- > blown > DFA, which also explains the slowness of regexes. I'll not tackle that > beast. > For some fields, I will support regex matches, but when they are used, > performance is affected. The overall idea is that you usually do not > need any > regex/DFA at all. > > > > > so once there is a high performance parser to pull the data apart, > then > > the question is what to do with it. > > > > some people will want to write it to various places, others will want > > to > > make decisions based on what is matched. > > > > for those who are wanting to write the normalized output to various > > places, a plugin structure like rsyslog has (with the ability to > format > > the messages based on the various properties that are discovered) is > > very > > appealing, and it may make a lot of sense to see what can be done to > > re-use that work. If so, there will need to be a 'format string' that > > creates the output with all the properties that are known tagged, but > > without including ones that didn't have any matches in this log > > message. > > One thing I definitely intend to do is utilize the library in rsyslog. > I > envision a parser module that works based on the library. That also > means > rsyslog's core engine must be extended to support the additional > fields, but > something that can definitely be done. With that approach, no complex > output > engine is needed - one can just use the rsyslog plugin. And with a near > O(1) > algorithm, we can probably expect that this happens in real-time even > for > very large traffic loads (but probably not for the largest ones). > > It is important to know here that the current parsers also have some > limited > backout needs, for example for the date and tag/hostname fields. So > this can > be done quickly. > > > > > for those who are wanting to then implement logic based on what it > > gets, > > thing get much more interesting. I suspect that the thing to do here > > will > > be to make the event normalization engine be something that can be a > > library included in other programs (in various languages), something > so > > that you can have the config file be something along the lines of > > that's actually the idea. An initial sketch of the API is already in > git and > I hope to get some better-readbly doxgen-generated interface spec up > later > today. > > > > documentation (hopefully including a sample raw line) > > line-to-match > > function to call when matched > > > > there are a log of programs out there written to do good and > > interesting > > stuff with lines that it receives, if there was an ability to replace > > their sequential 'does it match rule 1, does it match rule 2' logic > > with a > > more efficient parser it would be a huge win. > > > > I don't think you are wanting to tackle that portion of the task. > > The lib part yes, but that obviously requires a lot of changes for > applications using it. > > Rainer > > > > David Lang > > > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > >> Sent: Monday, October 11, 2010 9:01 AM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] liblognorm vs. libeventnorm > > >> > > >> I would like to add as an argument pro liblognorm, that many > people > > >> probably > > >> better understand what "log normalization" is whereas "event > > >> normalization" > > >> may sound strange. In that sense, liblognorm may be a better name. > > >> Feedback > > >> is appreciated. > > >> > > >> Rainer > > >> > > >>> -----Original Message----- > > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > >>> Sent: Sunday, October 10, 2010 11:53 AM > > >>> To: rsyslog-users > > >>> Subject: [rsyslog] liblognorm vs. libeventnorm > > >>> > > >>> Hi all, > > >>> > > >>> I think I'll start with the libeventnorm name for the normalizing > > >>> library > > >>> instead of liblognorm. Reason here: > > >>> > > >>> http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html > > >>> > > >>> Further name suggestions or arguments are very welcome! > > >>> > > >>> Rainer > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Wed Oct 13 15:55:32 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 13 Oct 2010 15:55:32 +0200 Subject: [rsyslog] library for XML Regular Expressions? Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4CF@GRFEXC.intern.adiscon.com> Hi all, does someone know a decent library to process XML regular expressions, as specified in http://www.w3.org/TR/xmlschema-2/#regexs I am NOT looking for some XML validator or XML toolset that can process them, but for a generic library that I can throw a string at and have that string be checked against a XML regex. I need this to implement some of the plumbing for the normalizing lib. Interestingly, I did not find a dedicated library to do the jobs, so I thought if someone over here has an idea. A fall-back scenario may be to extract something from a XML toolkit, but I'd prefer to avoid that, if possible. Thanks, Rainer From rgerhards at hq.adiscon.com Wed Oct 13 20:58:21 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 13 Oct 2010 20:58:21 +0200 Subject: [rsyslog] library for XML Regular Expressions? References: <9B6E2A8877C38245BFB15CC491A11DA71DD4CF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4D6@GRFEXC.intern.adiscon.com> I was hinted that libxml provides the necessary functionality, and indeed it does :) I seem to have overlooked that. This resolves my last perquisite and so I am very happy to be able to look at the CEE part initial interface soon. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Wednesday, October 13, 2010 3:56 PM > To: rsyslog-users > Subject: [rsyslog] library for XML Regular Expressions? > > Hi all, > > does someone know a decent library to process XML regular expressions, > as > specified in > > http://www.w3.org/TR/xmlschema-2/#regexs > > I am NOT looking for some XML validator or XML toolset that can process > them, > but for a generic library that I can throw a string at and have that > string > be checked against a XML regex. > > I need this to implement some of the plumbing for the normalizing lib. > Interestingly, I did not find a dedicated library to do the jobs, so I > thought if someone over here has an idea. > > A fall-back scenario may be to extract something from a XML toolkit, > but I'd > prefer to avoid that, if possible. > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jean.luc.lacroix at regains.be Thu Oct 14 14:42:16 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Thu, 14 Oct 2010 14:42:16 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> Message-ID: <4CB6FAA8.8080300@regains.be> Sorry to bump this but I get no assistance from Qnap folks and user group. In the mean time, I am forced to tail -f syslog to a mount which is far from elegant. It looks like a simple and repetitive awk syntax error but I don't know where to look. Le 08/10/10 10:06, Rainer Gerhards a ?crit : > I am sorry, I am far from being an autotools expert. So I can not help here. > Michael maybe? > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >> Sent: Friday, October 08, 2010 9:09 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] [compile] could not create Makefile >> >> Hi, >> >> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the >> configure script >> throw awk errors when trying to create the make file: >> >> CODE: SELECT ALL >> config.status: creating Makefile >> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >> gawk: ./confgA6g9j/subs.awk:2: ^ syntax >> error >> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >> gawk: ./confgA6g9j/subs.awk:3: ^ >> syntax error >> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# >> "#" >> gawk: ./confgA6g9j/subs.awk:4: ^ syntax >> error >> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >> gawk: ./confgA6g9j/subs.awk:4: ^ syntax error >> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >> gawk: ./confgA6g9j/subs.awk:5: ^ syntax >> error >> >> ... many more of these errors >> >> >> >> gawk version installed on my system: >> Package: gawk Version: 3.1.8-1 >> -- >> Jean-Luc Lacroix >> Regains SCS >> 0477 43 72 72 >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Oct 14 14:49:13 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 14 Oct 2010 14:49:13 +0200 Subject: [rsyslog] [compile] could not create Makefile References: <4CAEC38D.5040809@regains.be><9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD4EC@GRFEXC.intern.adiscon.com> If nobody over here has a clue, you could try the autoconf mailing list. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix > Sent: Thursday, October 14, 2010 2:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] [compile] could not create Makefile > > Sorry to bump this but I get no assistance from Qnap folks and user > group. In the mean > time, I am forced to tail -f syslog to a mount which is far from > elegant. > > It looks like a simple and repetitive awk syntax error but I don't know > where to look. > > Le 08/10/10 10:06, Rainer Gerhards a ?crit : > > I am sorry, I am far from being an autotools expert. So I can not > help here. > > Michael maybe? > > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix > >> Sent: Friday, October 08, 2010 9:09 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] [compile] could not create Makefile > >> > >> Hi, > >> > >> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and > the > >> configure script > >> throw awk errors when trying to create the make file: > >> > >> CODE: SELECT ALL > >> config.status: creating Makefile > >> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# > "" > >> gawk: ./confgA6g9j/subs.awk:2: ^ > syntax > >> error > >> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# > "" > >> gawk: ./confgA6g9j/subs.awk:3: ^ > >> syntax error > >> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# > >> "#" > >> gawk: ./confgA6g9j/subs.awk:4: ^ > syntax > >> error > >> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" > >> gawk: ./confgA6g9j/subs.awk:4: ^ syntax > error > >> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" > >> gawk: ./confgA6g9j/subs.awk:5: ^ syntax > >> error > >> > >> ... many more of these errors > >> > >> > >> > >> gawk version installed on my system: > >> Package: gawk Version: 3.1.8-1 > >> -- > >> Jean-Luc Lacroix > >> Regains SCS > >> 0477 43 72 72 > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From heinbockel at mitre.org Thu Oct 14 18:45:17 2010 From: heinbockel at mitre.org (Heinbockel, Bill) Date: Thu, 14 Oct 2010 12:45:17 -0400 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <4CB6FAA8.8080300@regains.be> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> Message-ID: <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> It looks like the issue is that you are not using the GNU version of awk and sed Make sure that both `gawk --version` and `sed --version` state that the programs are "GNU awk" and "GNU sed", respectively William Heinbockel The MITRE Corporation >-----Original Message----- >From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >Sent: Thursday, 14 October 2010 08:42 >To: rsyslog-users >Subject: Re: [rsyslog] [compile] could not create Makefile > >Sorry to bump this but I get no assistance from Qnap folks and user group. >In the mean >time, I am forced to tail -f syslog to a mount which is far from elegant. > >It looks like a simple and repetitive awk syntax error but I don't know >where to look. > >Le 08/10/10 10:06, Rainer Gerhards a ?crit : >> I am sorry, I am far from being an autotools expert. So I can not help >here. >> Michael maybe? >> >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>> Sent: Friday, October 08, 2010 9:09 AM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] [compile] could not create Makefile >>> >>> Hi, >>> >>> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the >>> configure script >>> throw awk errors when trying to create the make file: >>> >>> CODE: SELECT ALL >>> config.status: creating Makefile >>> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>> gawk: ./confgA6g9j/subs.awk:2: ^ syntax >>> error >>> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>> gawk: ./confgA6g9j/subs.awk:3: ^ >>> syntax error >>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# >>> "#" >>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax >>> error >>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax error >>> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>> gawk: ./confgA6g9j/subs.awk:5: ^ syntax >>> error >>> >>> ... many more of these errors >>> >>> >>> >>> gawk version installed on my system: >>> Package: gawk Version: 3.1.8-1 >>> -- >>> Jean-Luc Lacroix >>> Regains SCS >>> 0477 43 72 72 >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3520 bytes Desc: not available URL: From jean.luc.lacroix at regains.be Fri Oct 15 13:12:26 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Fri, 15 Oct 2010 13:12:26 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> Message-ID: <4CB8371A.6020800@regains.be> You are probably right but my NAS is using rather recent versions: sed - 4.2.1-1 - Stream editor gawk - 3.1.8-1 - Gnu AWK interpreter I think I will try a xcompile on my Debian box. Never did it but I will give it a try. Anything I should know before I do it? Jean-Luc Lacroix Regains SCS 0477 43 72 72 Le 14/10/10 18:45, Heinbockel, Bill a ?crit : > It looks like the issue is that you are not using the GNU version of awk and sed > > Make sure that both `gawk --version` and `sed --version` state that the programs > are "GNU awk" and "GNU sed", respectively > > > William Heinbockel > The MITRE Corporation > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >> Sent: Thursday, 14 October 2010 08:42 >> To: rsyslog-users >> Subject: Re: [rsyslog] [compile] could not create Makefile >> >> Sorry to bump this but I get no assistance from Qnap folks and user group. >> In the mean >> time, I am forced to tail -f syslog to a mount which is far from elegant. >> >> It looks like a simple and repetitive awk syntax error but I don't know >> where to look. >> >> Le 08/10/10 10:06, Rainer Gerhards a ?crit : >>> I am sorry, I am far from being an autotools expert. So I can not help >> here. >>> Michael maybe? >>> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>>> Sent: Friday, October 08, 2010 9:09 AM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] [compile] could not create Makefile >>>> >>>> Hi, >>>> >>>> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the >>>> configure script >>>> throw awk errors when trying to create the make file: >>>> >>>> CODE: SELECT ALL >>>> config.status: creating Makefile >>>> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>> gawk: ./confgA6g9j/subs.awk:2: ^ syntax >>>> error >>>> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>> gawk: ./confgA6g9j/subs.awk:3: ^ >>>> syntax error >>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# >>>> "#" >>>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax >>>> error >>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax error >>>> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>> gawk: ./confgA6g9j/subs.awk:5: ^ syntax >>>> error >>>> >>>> ... many more of these errors >>>> >>>> >>>> >>>> gawk version installed on my system: >>>> Package: gawk Version: 3.1.8-1 >>>> -- >>>> Jean-Luc Lacroix >>>> Regains SCS >>>> 0477 43 72 72 >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com From mbiebl at gmail.com Fri Oct 15 13:57:56 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 15 Oct 2010 13:57:56 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <4CB8371A.6020800@regains.be> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> <4CB8371A.6020800@regains.be> Message-ID: Are you sure, you are not using awk as provided by busybox? I found a related bug report: https://bugs.launchpad.net/hplip/+bug/577209 HTH, Michael 2010/10/15 Jean-Luc Lacroix : > You are probably right but my NAS is using rather recent versions: > sed - 4.2.1-1 - Stream editor > gawk - 3.1.8-1 - Gnu AWK interpreter > > I think I will try a xcompile on my Debian box. Never did it but I will give > it a try. Anything I should know before I do it? > > Jean-Luc Lacroix > Regains SCS > 0477 43 72 72 > > Le 14/10/10 18:45, Heinbockel, Bill a ?crit : >> >> It looks like the issue is that you are not using the GNU version of awk >> and sed >> >> Make sure that both `gawk --version` and `sed --version` state that the >> programs >> are "GNU awk" and "GNU sed", respectively >> >> >> William Heinbockel >> The MITRE Corporation >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>> Sent: Thursday, 14 October 2010 08:42 >>> To: rsyslog-users >>> Subject: Re: [rsyslog] [compile] could not create Makefile >>> >>> Sorry to bump this but I get no assistance from Qnap folks and user >>> group. >>> In the mean >>> time, I am forced to tail -f syslog to a mount which is far from elegant. >>> >>> It looks like a simple and repetitive awk syntax error but I don't know >>> where to look. >>> >>> Le 08/10/10 10:06, Rainer Gerhards a ?crit : >>>> >>>> I am sorry, I am far from being an autotools expert. So I can not help >>> >>> here. >>>> >>>> Michael maybe? >>>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>>>> Sent: Friday, October 08, 2010 9:09 AM >>>>> To: rsyslog at lists.adiscon.com >>>>> Subject: [rsyslog] [compile] could not create Makefile >>>>> >>>>> Hi, >>>>> >>>>> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the >>>>> configure script >>>>> throw awk errors when trying to create the make file: >>>>> >>>>> CODE: SELECT ALL >>>>> config.status: creating Makefile >>>>> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>>> gawk: ./confgA6g9j/subs.awk:2: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ syntax >>>>> error >>>>> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>>> gawk: ./confgA6g9j/subs.awk:3: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^ >>>>> syntax error >>>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# >>>>> "#" >>>>> gawk: ./confgA6g9j/subs.awk:4: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ syntax >>>>> error >>>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>>> gawk: ./confgA6g9j/subs.awk:4: ? ? ? ? ? ? ? ? ? ? ? ? ?^ syntax error >>>>> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>>> gawk: ./confgA6g9j/subs.awk:5: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ syntax >>>>> error >>>>> >>>>> ... many more of these errors >>>>> >>>>> >>>>> >>>>> gawk version installed on my system: >>>>> Package: gawk Version: 3.1.8-1 >>>>> -- >>>>> Jean-Luc Lacroix >>>>> Regains SCS >>>>> 0477 43 72 72 >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From jean.luc.lacroix at regains.be Fri Oct 15 20:12:06 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Fri, 15 Oct 2010 20:12:06 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> <4CB8371A.6020800@regains.be> Message-ID: <4CB89976.9010004@regains.be> Hi Michael, That was it! awk and sed pointed to the busybox binary. I symlinked sed and awk to their GNU version and the make file was created flawlessly. Thanks! Jean-Luc Lacroix Le 15/10/10 13:57, Michael Biebl a ?crit : > Are you sure, you are not using awk as provided by busybox? > > I found a related bug report: https://bugs.launchpad.net/hplip/+bug/577209 > > HTH, > Michael > > 2010/10/15 Jean-Luc Lacroix: >> You are probably right but my NAS is using rather recent versions: >> sed - 4.2.1-1 - Stream editor >> gawk - 3.1.8-1 - Gnu AWK interpreter >> >> I think I will try a xcompile on my Debian box. Never did it but I will give >> it a try. Anything I should know before I do it? >> >> Jean-Luc Lacroix >> Regains SCS >> 0477 43 72 72 >> >> Le 14/10/10 18:45, Heinbockel, Bill a ?crit : >>> >>> It looks like the issue is that you are not using the GNU version of awk >>> and sed >>> >>> Make sure that both `gawk --version` and `sed --version` state that the >>> programs >>> are "GNU awk" and "GNU sed", respectively >>> >>> >>> William Heinbockel >>> The MITRE Corporation >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>>> Sent: Thursday, 14 October 2010 08:42 >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] [compile] could not create Makefile >>>> >>>> Sorry to bump this but I get no assistance from Qnap folks and user >>>> group. >>>> In the mean >>>> time, I am forced to tail -f syslog to a mount which is far from elegant. >>>> >>>> It looks like a simple and repetitive awk syntax error but I don't know >>>> where to look. >>>> >>>> Le 08/10/10 10:06, Rainer Gerhards a ?crit : >>>>> >>>>> I am sorry, I am far from being an autotools expert. So I can not help >>>> >>>> here. >>>>> >>>>> Michael maybe? >>>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Jean-Luc Lacroix >>>>>> Sent: Friday, October 08, 2010 9:09 AM >>>>>> To: rsyslog at lists.adiscon.com >>>>>> Subject: [rsyslog] [compile] could not create Makefile >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to compile rsyslog on a Qnap NAS (proc Marvell ARM) and the >>>>>> configure script >>>>>> throw awk errors when trying to create the make file: >>>>>> >>>>>> CODE: SELECT ALL >>>>>> config.status: creating Makefile >>>>>> gawk: ./confgA6g9j/subs.awk:2: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>>>> gawk: ./confgA6g9j/subs.awk:2: ^ syntax >>>>>> error >>>>>> gawk: ./confgA6g9j/subs.awk:3: S["/; s/!.*/"]=am__EXEEXT_FALSE!%!_!# "" >>>>>> gawk: ./confgA6g9j/subs.awk:3: ^ >>>>>> syntax error >>>>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=am__EXEEXT_TRUE!#%!_!# >>>>>> "#" >>>>>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax >>>>>> error >>>>>> gawk: ./confgA6g9j/subs.awk:4: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>>>> gawk: ./confgA6g9j/subs.awk:4: ^ syntax error >>>>>> gawk: ./confgA6g9j/subs.awk:5: S["/; s/!.*/"]=LTLIBOBJS!%!_!# "" >>>>>> gawk: ./confgA6g9j/subs.awk:5: ^ syntax >>>>>> error >>>>>> >>>>>> ... many more of these errors >>>>>> >>>>>> >>>>>> >>>>>> gawk version installed on my system: >>>>>> Package: gawk Version: 3.1.8-1 >>>>>> -- >>>>>> Jean-Luc Lacroix >>>>>> Regains SCS >>>>>> 0477 43 72 72 >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > From jean.luc.lacroix at regains.be Sat Oct 16 10:25:57 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Sat, 16 Oct 2010 10:25:57 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <4CB89976.9010004@regains.be> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> <4CB8371A.6020800@regains.be> <4CB89976.9010004@regains.be> Message-ID: <4CB96195.7070108@regains.be> Well, the configuration script is working file now but the make and make install are both generating errors. Too long to be posted here but the log files can be found here: http://regains.be/repo/rsyslog/ Any help will be appreciated by the Qnap community as syslog server is a feature for which there is a strong demand. Jean-Luc Le 15/10/10 20:12, Jean-Luc Lacroix a ?crit : > Hi Michael, > > That was it! awk and sed pointed to the busybox binary. I symlinked sed > and awk to their GNU version and the make file was created flawlessly. > > Thanks! > > Jean-Luc > > Le 15/10/10 13:57, Michael Biebl a ?crit : >> Are you sure, you are not using awk as provided by busybox? >> >> I found a related bug report: >> https://bugs.launchpad.net/hplip/+bug/577209 >> >> HTH, >> Michael >> From mbiebl at gmail.com Sat Oct 16 14:17:42 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Sat, 16 Oct 2010 14:17:42 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <4CB96195.7070108@regains.be> References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> <4CB8371A.6020800@regains.be> <4CB89976.9010004@regains.be> <4CB96195.7070108@regains.be> Message-ID: 2010/10/16 Jean-Luc Lacroix : > Well, the configuration script is working file now but the make and make > install are both generating errors. Too long to be posted here but the log > files can be found here: > http://regains.be/repo/rsyslog/ Which versions of gcc and libc6 do you use? Does it help if you upgrade gcc? I've seen similar problems on Debian [1], [2] when compiling for lenny, which has gcc 4.3.2. Michael [1] https://buildd.debian.org/status/package.php?p=rsyslog&suite=lenny-backports [2] http://lists.debian.org/debian-backports/2010/09/msg00154.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Sat Oct 16 15:00:34 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 16 Oct 2010 15:00:34 +0200 Subject: [rsyslog] [compile] could not create Makefile References: <4CAEC38D.5040809@regains.be><9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com><4CB6FAA8.8080300@regains.be><93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG><4CB8371A.6020800@regains.be><4CB89976.9010004@regains.be> <4CB96195.7070108@regains.be> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD507@GRFEXC.intern.adiscon.com> I think he needs to upgrade to 4.7.2, because of the atomic instructions: http://www.rsyslog.com/article457/ The make log shows issues with them, and I think for 4.6.4 they MUST be available. If this is in a recent Intel CPU, adding -march=i586 (or so) to CFLAGS will also remove the issues (AND is the strongly recommended solution, as it leads to faster code). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Saturday, October 16, 2010 2:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] [compile] could not create Makefile > > 2010/10/16 Jean-Luc Lacroix : > > Well, the configuration script is working file now but the make and > make > > install are both generating errors. Too long to be posted here but > the log > > files can be found here: > > http://regains.be/repo/rsyslog/ > > Which versions of gcc and libc6 do you use? Does it help if you upgrade > gcc? > > I've seen similar problems on Debian [1], [2] when compiling for > lenny, which has gcc > 4.3.2. > > > Michael > > [1] https://buildd.debian.org/status/package.php?p=rsyslog&suite=lenny- > backports > [2] http://lists.debian.org/debian-backports/2010/09/msg00154.html > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jean.luc.lacroix at regains.be Sat Oct 16 16:08:37 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Sat, 16 Oct 2010 16:08:37 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: References: <4CAEC38D.5040809@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com> <4CB6FAA8.8080300@regains.be> <93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG> <4CB8371A.6020800@regains.be> <4CB89976.9010004@regains.be> <4CB96195.7070108@regains.be> Message-ID: <4CB9B1E5.1030203@regains.be> All I can find is this: Package: gcc Version: 4.2.3-1 Depends: binutils, libc-dev Package: libc-dev Version: 2.5-5 Depends: libnsl Package: libnsl Version: 2.5-4 The gcc package is the only one available on that platform. Don't know how to update. I could try to do a x-compilation (my debian box has a gcc 4:4.4.4-2 compiler) but here again I don't know how to do that so that the libraries are installed in the right directories etc... Jean-Luc Lacroix Le 16/10/10 14:17, Michael Biebl a ?crit : >> Well, the configuration script is working file now but the make and make >> install are both generating errors. Too long to be posted here but the log >> files can be found here: >> http://regains.be/repo/rsyslog/ > > Which versions of gcc and libc6 do you use? Does it help if you upgrade gcc? > > I've seen similar problems on Debian [1], [2] when compiling for > lenny, which has gcc > 4.3.2. > > > Michael > > [1] https://buildd.debian.org/status/package.php?p=rsyslog&suite=lenny-backports > [2] http://lists.debian.org/debian-backports/2010/09/msg00154.html From jean.luc.lacroix at regains.be Sat Oct 16 21:44:39 2010 From: jean.luc.lacroix at regains.be (Jean-Luc Lacroix) Date: Sat, 16 Oct 2010 21:44:39 +0200 Subject: [rsyslog] [compile] could not create Makefile In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD507@GRFEXC.intern.adiscon.com> References: <4CAEC38D.5040809@regains.be><9B6E2A8877C38245BFB15CC491A11DA71DD47E@GRFEXC.intern.adiscon.com><4CB6FAA8.8080300@regains.be><93ED0A84F9A1D74FA65021D940AA5884053E2ACD7D@IMCMBX3.MITRE.ORG><4CB8371A.6020800@regains.be><4CB89976.9010004@regains.be> <4CB96195.7070108@regains.be> <9B6E2A8877C38245BFB15CC491A11DA71DD507@GRFEXC.intern.adiscon.com> Message-ID: <4CBA00A7.9010509@regains.be> Oh joy! Compilation went flawlessly with the 4.7.2 sources. rsyslog is up and running. Thank you Rainer and Michael and check this thread on the Qnap forum: http://forum.qnap.com/viewtopic.php?f=24&t=11507&p=155879#p156004 Jean-Luc Lacroix Le 16/10/10 15:00, Rainer Gerhards a ?crit : > I think he needs to upgrade to 4.7.2, because of the atomic instructions: > > http://www.rsyslog.com/article457/ > > The make log shows issues with them, and I think for 4.6.4 they MUST be > available. If this is in a recent Intel CPU, adding -march=i586 (or so) to > CFLAGS will also remove the issues (AND is the strongly recommended solution, > as it leads to faster code). > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Michael Biebl >> Sent: Saturday, October 16, 2010 2:18 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] [compile] could not create Makefile >> >> 2010/10/16 Jean-Luc Lacroix: >>> Well, the configuration script is working file now but the make and >> make >>> install are both generating errors. Too long to be posted here but >> the log >>> files can be found here: >>> http://regains.be/repo/rsyslog/ >> >> Which versions of gcc and libc6 do you use? Does it help if you upgrade >> gcc? >> >> I've seen similar problems on Debian [1], [2] when compiling for >> lenny, which has gcc >> 4.3.2. >> >> >> Michael >> >> [1] https://buildd.debian.org/status/package.php?p=rsyslog&suite=lenny- >> backports >> [2] http://lists.debian.org/debian-backports/2010/09/msg00154.html >> -- >> Why is it that all of the instruments seeking intelligent life in the >> universe are pointed away from Earth? >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From tbergfeld at hq.adiscon.com Tue Oct 19 16:14:27 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Tue, 19 Oct 2010 16:14:27 +0200 Subject: [rsyslog] rsyslog 5.6.0 (v5-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD51E@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.6.0, the new v5-stable. This release brings all changes and enhancements of the 5.5.x series to the v5-stable branch. It is a recommended update for all users of the devel branch. See Changelog for more details. ChangeLog: http://www.rsysl og.com/changelog-for-5-6-0-v5-stable/ Download: http://www.rsyslog.com /rsyslog-5-6-0-v5-stable/ As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From champ at softwink.com Wed Oct 20 19:59:51 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 20 Oct 2010 13:59:51 -0400 Subject: [rsyslog] Building liblognorm.... Message-ID: <20101020175951.GA13070@bundy.vistech.net> Rainer, I pulled down the (git clone) the liblognorm stuff this morning. This might be a bit premature, but I'd like to start getting my hands dirty :) I noticed you have the configure.ac and Makefile.am out there. Is it correct to assume I'll need to manually run autoconf / automake / aclocal and all that. I'm just wondering how you're getting your auto tools to setup the build system. Thanks. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Wed Oct 20 20:47:38 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 20 Oct 2010 20:47:38 +0200 Subject: [rsyslog] Building liblognorm.... References: <20101020175951.GA13070@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD539@GRFEXC.intern.adiscon.com> It's just $ autoreconf -vfi (if you like verbose output ;)) Than you can do ./configure. But, honestly, there is not much that is built today. A simple test program is there, plus a very basic (and memory-hungry) radix tree. I now need some CEE stuff for parsing and this is what I am currently looking at. Unfortunately, there are a couple of smaller things that - together - take a larger part of my day than I had hoped for. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Wednesday, October 20, 2010 8:00 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Building liblognorm.... > > > Rainer, > > I pulled down the (git clone) the liblognorm stuff this morning. > This might be a bit premature, but I'd like to start getting my hands > dirty :) > > I noticed you have the configure.ac and Makefile.am out there. > Is it correct to assume I'll need to manually run autoconf / automake / > aclocal and all that. I'm just wondering how you're getting your auto > tools > to setup the build system. Thanks. > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Wed Oct 20 20:59:12 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 20 Oct 2010 14:59:12 -0400 Subject: [rsyslog] Building liblognorm.... In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD539@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD539@GRFEXC.intern.adiscon.com> Message-ID: <20101020185912.GA15527@bundy.vistech.net> On Wed, Oct 20, 2010 at 08:47:38PM +0200, Rainer Gerhards wrote: > It's just > > $ autoreconf -vfi (if you like verbose output ;)) > > Than you can do ./configure. But, honestly, there is not much that is built > today. A simple test program is there, plus a very basic (and memory-hungry) > radix tree. I now need some CEE stuff for parsing and this is what I am > currently looking at. Unfortunately, there are a couple of smaller things > that - together - take a larger part of my day than I had hoped for. Okay. Looking at the code, online, it didn't look like a lot was there yet. BTW: autoreconf didn't work for me, but I can wait on that. No worries. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From timuckun at gmail.com Thu Oct 21 04:34:00 2010 From: timuckun at gmail.com (Tim Uckun) Date: Thu, 21 Oct 2010 15:34:00 +1300 Subject: [rsyslog] postgres logging to three places.. Message-ID: I have the following setup in the rsyslog.d directory. # local0.* -/var/log/postgresql/postgresql-8.4-main.log # # add local0 to the ignore field for the normal messages so it wont' get logged twice. # *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none;\ local0.none -/var/log/messages The problem is that postgres is now logging to /var/log/postgres... and /var/log/messages and /var/log/syslog. What do I have to do in order to exclude the postgres logs from messages and syslog? Thanks. From rgerhards at hq.adiscon.com Thu Oct 21 07:26:08 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 21 Oct 2010 07:26:08 +0200 Subject: [rsyslog] postgres logging to three places.. References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> do local0.* -/var/log/postgresql/postgresql-8.4-main.log & ~ Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tim Uckun > Sent: Thursday, October 21, 2010 4:34 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] postgres logging to three places.. > > I have the following setup in the rsyslog.d directory. > > # > local0.* -/var/log/postgresql/postgresql-8.4-main.log > # > # add local0 to the ignore field for the normal messages so it wont' > get logged twice. > # > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none;\ > local0.none -/var/log/messages > > > The problem is that postgres is now logging to /var/log/postgres... > and /var/log/messages and /var/log/syslog. > > What do I have to do in order to exclude the postgres logs from > messages and syslog? > > Thanks. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From timuckun at gmail.com Thu Oct 21 12:40:26 2010 From: timuckun at gmail.com (Tim Uckun) Date: Thu, 21 Oct 2010 23:40:26 +1300 Subject: [rsyslog] postgres logging to three places.. In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> Message-ID: On Thu, Oct 21, 2010 at 6:26 PM, Rainer Gerhards wrote: > do > > local0.* ? ? ? ? ? ? ? ?-/var/log/postgresql/postgresql-8.4-main.log > & ~ Thanks. From marcin at mejor.pl Thu Oct 21 16:23:58 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Thu, 21 Oct 2010 16:23:58 +0200 Subject: [rsyslog] Could be message "imuxsock begins to drop messages" be a little more verbose? Message-ID: <4CC04CFE.2060707@mejor.pl> Hello! I found in message.log following messages:" 2010-10-19T22:24:58.641707+02:00 localhost suhosin[1669]: ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value (attacker '2001:470:1f0b:1ab 3:1ce3:a6fc:750f:fde1', file '/dane/domeny/xxxxx', line 96) 2010-10-20T09:23:28+02:00 localhost sshd[23129]: error: PAM: Authentication failure for marcin from 127-goc-33.acn.waw.pl 2010-10-20T09:23:30+02:00 localhost sshd[23129]: Accepted keyboard-interactive/pam for marcin from 94.75.108.127 port 49875 ssh2 2010-10-20T10:27:54.593338+02:00 localhost kernel: [167143.457207] deliver[667]: segfault at 48 ip 9af8c707 sp b220b910 error 6 in libdovecot-storage.so.0.0.0[9af2f000+a3000] 2010-10-20T15:37:25.404441+02:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 12703 due to rate-limiting 2010-10-20T15:37:27.006681+02:00 localhost rsyslogd-2177: imuxsock lost 147 messages from pid 12703 due to rate-limiting 2010-10-20T15:37:28.850821+02:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 12703 due to rate-limiting 2010-10-20T15:37:33.003283+02:00 localhost rsyslogd-2177: imuxsock lost 462 messages from pid 12703 due to rate-limiting " It's fine rate-limit cut off many messages, but is it possible that imuxsock provide what message is dropped? From this log i don't know which application floods log (probably php-cgi) and what was the message. Could be message from rate-limit extended with info about: name of pid and message? E.g.: "2010-10-20T15:37:28.850821+02:00 localhost rsyslogd-2177: imuxsock begins to drop messages from pid 12703 due to rate-limiting, *last mesasge was*: localhost suhosin[1669]: ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value (attacker '2001:470:1f0b:1ab 3:1ce3:a6fc:750f:fde1', file '/dane/domeny/xxxxx', line 96)" Regards! From rgerhards at hq.adiscon.com Thu Oct 21 16:32:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 21 Oct 2010 16:32:59 +0200 Subject: [rsyslog] Could be message "imuxsock begins to drop messages" be a little more verbose? References: <4CC04CFE.2060707@mejor.pl> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD55B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw > Sent: Thursday, October 21, 2010 4:24 PM > To: rsyslog-users > Subject: [rsyslog] Could be message "imuxsock begins to drop messages" > be a little more verbose? > > Hello! > I found in message.log following messages:" > 2010-10-19T22:24:58.641707+02:00 localhost suhosin[1669]: ALERT - > script > tried to increase memory_limit to 268435456 bytes which is above the > allowed value (attacker '2001:470:1f0b:1ab > 3:1ce3:a6fc:750f:fde1', file '/dane/domeny/xxxxx', line 96) > 2010-10-20T09:23:28+02:00 localhost sshd[23129]: error: PAM: > Authentication failure for marcin from 127-goc-33.acn.waw.pl > 2010-10-20T09:23:30+02:00 localhost sshd[23129]: Accepted > keyboard-interactive/pam for marcin from 94.75.108.127 port 49875 ssh2 > 2010-10-20T10:27:54.593338+02:00 localhost kernel: [167143.457207] > deliver[667]: segfault at 48 ip 9af8c707 sp b220b910 error 6 in > libdovecot-storage.so.0.0.0[9af2f000+a3000] > 2010-10-20T15:37:25.404441+02:00 localhost rsyslogd-2177: imuxsock > begins to drop messages from pid 12703 due to rate-limiting > 2010-10-20T15:37:27.006681+02:00 localhost rsyslogd-2177: imuxsock lost > 147 messages from pid 12703 due to rate-limiting > 2010-10-20T15:37:28.850821+02:00 localhost rsyslogd-2177: imuxsock > begins to drop messages from pid 12703 due to rate-limiting > 2010-10-20T15:37:33.003283+02:00 localhost rsyslogd-2177: imuxsock lost > 462 messages from pid 12703 due to rate-limiting > " > > It's fine rate-limit cut off many messages, but is it possible that > imuxsock provide what message is dropped? No, because that would force me to store the last message for every possible pid. The idea is that you have the pid, so you can check what was the last message from that pid inside the log file. Rainer > From this log i don't know > which application floods log (probably php-cgi) and what was the > message. Could be message from rate-limit extended with info about: > name > of pid and message? > E.g.: > "2010-10-20T15:37:28.850821+02:00 localhost rsyslogd-2177: imuxsock > begins to drop messages from pid 12703 due to rate-limiting, *last > mesasge was*: localhost suhosin[1669]: ALERT - script tried to > increase > memory_limit to 268435456 bytes which is above the allowed value > (attacker '2001:470:1f0b:1ab > 3:1ce3:a6fc:750f:fde1', file '/dane/domeny/xxxxx', line 96)" > > Regards! > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Oct 21 16:59:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 21 Oct 2010 16:59:54 +0200 Subject: [rsyslog] Building liblognorm.... References: <20101020175951.GA13070@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD539@GRFEXC.intern.adiscon.com> <20101020185912.GA15527@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD55C@GRFEXC.intern.adiscon.com> It may be worth looking at libee, which I was able to commit quite some code to today. Still in its infancy, but growing up ;) Links in Blog post: http://blog.gerhards.net/2010/10/libee-first-peek-preview-available.html I have completed necessary tag handling code, I will probably now do some small changes to liblognorm to utilize that before continuing on in libee. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Wednesday, October 20, 2010 8:59 PM > To: rsyslog-users > Subject: Re: [rsyslog] Building liblognorm.... > > On Wed, Oct 20, 2010 at 08:47:38PM +0200, Rainer Gerhards wrote: > > It's just > > > > $ autoreconf -vfi (if you like verbose output ;)) > > > > Than you can do ./configure. But, honestly, there is not much that is > built > > today. A simple test program is there, plus a very basic (and memory- > hungry) > > radix tree. I now need some CEE stuff for parsing and this is what I > am > > currently looking at. Unfortunately, there are a couple of smaller > things > > that - together - take a larger part of my day than I had hoped for. > > Okay. Looking at the code, online, it didn't look like a lot > was there yet. BTW: autoreconf didn't work for me, but I can wait > on that. No worries. > > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From pgollucci at p6m7g8.com Fri Oct 22 00:27:31 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Thu, 21 Oct 2010 22:27:31 +0000 Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-descr pkg-plist Message-ID: <4CC0BE53.6090300@p6m7g8.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -------- Original Message -------- Subject: cvs commit: ports/sysutils/rsyslog5 Makefile distinfo pkg-descr pkg-plist Date: Thu, 21 Oct 2010 21:46:49 +0000 (UTC) From: Philip M. Gollucci To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org pgollucci 2010-10-21 21:46:49 UTC FreeBSD ports repository Modified files: sysutils/rsyslog5 Makefile distinfo pkg-descr pkg-plist Log: - Update to 5.6.0 marking the start of a 5-stable. Changes: http://www.rsyslog.com/changelog-for-5-6-0-v5-stable/ Revision Changes Path 1.21 +1 -1 ports/sysutils/rsyslog5/Makefile 1.15 +3 -3 ports/sysutils/rsyslog5/distinfo 1.7 +1 -1 ports/sysutils/rsyslog5/pkg-descr 1.10 +1 -0 ports/sysutils/rsyslog5/pkg-plist http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.20&r2=1.21&f=h | --- ports/sysutils/rsyslog5/Makefile 2010/10/06 00:26:23 1.20 | +++ ports/sysutils/rsyslog5/Makefile 2010/10/21 21:46:49 1.21 | @@ -2,11 +2,11 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.20 2010/10/06 00:26:23 pgollucci Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.21 2010/10/21 21:46:49 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 5.5.7 | +PORTVERSION= 5.6.0 | CATEGORIES= sysutils | MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.14&r2=1.15&f=h | --- ports/sysutils/rsyslog5/distinfo 2010/10/06 00:11:53 1.14 | +++ ports/sysutils/rsyslog5/distinfo 2010/10/21 21:46:49 1.15 | @@ -1,3 +1,3 @@ | -MD5 (rsyslog-5.5.7.tar.gz) = bd38106a467f26ea1b73d62b05ea2c0c | -SHA256 (rsyslog-5.5.7.tar.gz) = 593cb2b471b865ddeb7ee8d33391db3894c70db79fb55824384bfa8b68d36a84 | -SIZE (rsyslog-5.5.7.tar.gz) = 2232657 | +MD5 (rsyslog-5.6.0.tar.gz) = 62c8edad559a39ca8c124ca87908075b | +SHA256 (rsyslog-5.6.0.tar.gz) = 54dde8f250aa641d810cafc6745403a753bc2822b6aeef0df094cb7103a3c505 | +SIZE (rsyslog-5.6.0.tar.gz) = 2253026 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-descr.diff?r1=1.6&r2=1.7&f=h | --- ports/sysutils/rsyslog5/pkg-descr 2010/10/06 00:22:41 1.6 | +++ ports/sysutils/rsyslog5/pkg-descr 2010/10/21 21:46:49 1.7 | @@ -14,6 +14,6 @@ suitable for enterprise-class, encryptio | chains while at the same time being very easy to setup for the | novice user. | | -Tracking v5-beta | +Tracking v5-stable | | WWW: http://www.rsyslog.com/ http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/pkg-plist.diff?r1=1.9&r2=1.10&f=h | --- ports/sysutils/rsyslog5/pkg-plist 2010/10/06 00:11:53 1.9 | +++ ports/sysutils/rsyslog5/pkg-plist 2010/10/21 21:46:49 1.10 | @@ -55,6 +55,7 @@ sbin/rsyslogd | %%PORTDOCS%%%%DOCSDIR%%/imtcp.html | %%PORTDOCS%%%%DOCSDIR%%/imsolaris.html | %%PORTDOCS%%%%DOCSDIR%%/imuxsock.html | +%%PORTDOCS%%%%DOCSDIR%%/imptcp.html | %%PORTDOCS%%%%DOCSDIR%%/index.html | %%PORTDOCS%%%%DOCSDIR%%/install.html | %%PORTDOCS%%%%DOCSDIR%%/ipv6.html -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFMwL5TdbiP+9ubjBwRApYuAJ96u3xxm76MSWB5Sicx9FyqM51dIQCeJ5Bm cYZ0Ot1+EgrDQOPPmhkFqgA= =7HZR -----END PGP SIGNATURE----- From pgollucci at p6m7g8.com Fri Oct 22 00:51:59 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Thu, 21 Oct 2010 22:51:59 +0000 Subject: [rsyslog] rsyslog 5.6.0 (v5-stable) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD51E@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD51E@GRFEXC.intern.adiscon.com> Message-ID: <4CC0C40F.8000400@p6m7g8.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 $ date Thu Oct 21 22:50:59 UTC 2010 $ ls -l configure - -rwxr-xr-x 1 pgollucci wheel - 529K Oct 22 14:20:13 2010 configure* Somebody's clock is scewed by several days (like 3-4). ===> Configuring for rsyslog-relp-5.6.0 checking for a BSD-compatible install... /usr/bin/install -c -o root -g wheel checking whether build environment is sane... configure: error: newly created file is older than distributed files! Check your system clock On 10/19/10 14:14, Tom Bergfeld wrote: > Hi all, > > We have just released rsyslog 5.6.0, the new v5-stable. This release brings > all changes and enhancements of the 5.5.x series to the v5-stable branch. > > It is a recommended update for all users of the devel branch. See Changelog > for more details. > > ChangeLog: > > href="http://www.rsyslog.com/changelog-for-5-6-0-v5-stable/">http://www.rsysl > og.com/changelog-for-5-6-0-v5-stable/ > > Download: > > href="http://www.rsyslog.com/rsyslog-5-6-0-v5-stable/">http://www.rsyslog.com > /rsyslog-5-6-0-v5-stable/ > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > -- > Support > ======= > > Improving rsyslog is costly, but you can help! We are looking for > organizations that find rsyslog useful and wish to contribute back. You can > contribute by reporting bugs, improve the software, or donate money or > equipment. > > Commercial support contracts for rsyslog are available, and they help finance > continued maintenance. Adiscon GmbH, a privately held German company, is > currently funding rsyslog development. We are always looking for interesting > development projects. For details on how to help, please see > http://www.rsyslog.com/doc-how2help.html . > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > - -- - ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFMwMQPdbiP+9ubjBwRAjM/AJ9pLjm+08vz94yv4svj8NCxbwgH5wCfQDjb AHzLXS0DVZxBZ0vepksTfi0= =cCCI -----END PGP SIGNATURE----- From rgerhards at hq.adiscon.com Fri Oct 22 07:20:42 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 22 Oct 2010 07:20:42 +0200 Subject: [rsyslog] rsyslog 5.6.0 (v5-stable) released References: <9B6E2A8877C38245BFB15CC491A11DA71DD51E@GRFEXC.intern.adiscon.com> <4CC0C40F.8000400@p6m7g8.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD561@GRFEXC.intern.adiscon.com> Hi Philip, thanks for letting me know. I had indeed a very weired issue on my main development VM where the timesync went wild. Bad that the tarball is affected. However, it is now just 25 hours to go, so I think I will not update it. This would require us to make a number of changes to various sites aka quite some work. Given the fact that the issue is removed tomorrow and existed for a couple of days already, I'd like to save that time for some other things. I'll be cautioned to look at the timestamp in more depth the next time. I did a check on all platforms to make sure I had no hidden issues, but I did so via the git version. Looks like I found another way to screw up :( Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Philip M. Gollucci > Sent: Friday, October 22, 2010 12:52 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.6.0 (v5-stable) released > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > $ date > Thu Oct 21 22:50:59 UTC 2010 > $ ls -l configure > - -rwxr-xr-x 1 pgollucci wheel - 529K Oct 22 14:20:13 2010 > configure* > > Somebody's clock is scewed by several days (like 3-4). > > ===> Configuring for rsyslog-relp-5.6.0 > checking for a BSD-compatible install... /usr/bin/install -c -o root -g > wheel > checking whether build environment is sane... configure: error: newly > created file is older than distributed files! > Check your system clock > > > > > On 10/19/10 14:14, Tom Bergfeld wrote: > > Hi all, > > > > We have just released rsyslog 5.6.0, the new v5-stable. This release > brings > > all changes and enhancements of the 5.5.x series to the v5-stable > branch. > > > > It is a recommended update for all users of the devel branch. See > Changelog > > for more details. > > > > ChangeLog: > > > > > href="http://www.rsyslog.com/changelog-for-5-6-0-v5- > stable/">http://www.rsysl > > og.com/changelog-for-5-6-0-v5-stable/ > > > > Download: > > > > > href="http://www.rsyslog.com/rsyslog-5-6-0-v5- > stable/">http://www.rsyslog.com > > /rsyslog-5-6-0-v5-stable/ > > > > As always, feedback is appreciated. > > > > Best regards, > > Tom Bergfeld > > -- > > Support > > ======= > > > > Improving rsyslog is costly, but you can help! We are looking for > > organizations that find rsyslog useful and wish to contribute back. > You can > > contribute by reporting bugs, improve the software, or donate money > or > > equipment. > > > > Commercial support contracts for rsyslog are available, and they help > finance > > continued maintenance. Adiscon GmbH, a privately held German > company, is > > currently funding rsyslog development. We are always looking for > interesting > > development projects. For details on how to help, please see > > http://www.rsyslog.com/doc-how2help.html . > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > - -- > - --------------------------------------------------------------------- > --- > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354 > VP Apache Infrastructure; Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.16 (FreeBSD) > > iD8DBQFMwMQPdbiP+9ubjBwRAjM/AJ9pLjm+08vz94yv4svj8NCxbwgH5wCfQDjb > AHzLXS0DVZxBZ0vepksTfi0= > =cCCI > -----END PGP SIGNATURE----- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From timuckun at gmail.com Fri Oct 22 12:35:48 2010 From: timuckun at gmail.com (Tim Uckun) Date: Fri, 22 Oct 2010 23:35:48 +1300 Subject: [rsyslog] postgres logging to three places.. In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> Message-ID: On Thu, Oct 21, 2010 at 6:26 PM, Rainer Gerhards wrote: > do > > local0.* ? ? ? ? ? ? ? ?-/var/log/postgresql/postgresql-8.4-main.log > & ~ > I am afraid this did not work. To give you more details. I am running on ubuntu with postgres 8.4. Postgres is configured like this. log_destination = 'syslog' syslog_facility = 'LOCAL0' I created a file called postgresql.conf in the /etc/rsyslog.d/ directory and now it looks like this. local0.* -/var/log/postgresql/postgresql-8.4-main.log & ~ # # add local0 to the ignore field for the normal messages so it wont' get logged twice. # *.=info;*.=notice;*.=warn;\ local0.none -/var/log/messages *.=info;*.=notice;*.=warn;\ local0.none -/var/log/syslog I tried it with and without the bottom part (after the comments). No matter what I do every entry is written into /var/log/messages , /var/log/syslog and /var/log/postgresql/postgresql-8.4-main.log I am reloading rsyslog with "reload rsyslog" when I do that both syslog and messages have rsyslog messages on them which is also a bit puzzling. The rest of the syslog configuration is in 50-default.conf which I didn't touch but I have included here for reference. Default rules for rsyslog. # # For more information see rsyslog.conf(5) and /etc/rsyslog.conf # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # # Logging for INN news system. # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice # # Some "catch-all" log files. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages # # Emergencies are sent to everybody logged in. # *.emerg * # # I like to have messages displayed on the console, but only on a virtual # console I usually leave idle. # #daemon,mail.*;\ # news.=crit;news.=err;news.=notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn /dev/tty8 # The named pipe /dev/xconsole is for the `xconsole' utility. To use it, # you must invoke `xconsole' with the `-file' option: # # $ xconsole -file /dev/xconsole [...] # # NOTE: adjust the list below, or you'll go crazy if you have a reasonably # busy site.. # daemon.*;mail.*;\ news.err;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole From rgerhards at hq.adiscon.com Fri Oct 22 12:40:04 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 22 Oct 2010 12:40:04 +0200 Subject: [rsyslog] postgres logging to three places.. References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> I guess the answer is simply that your file is included at the wrong spot into the main config file. Move that code to the top of the main file and it will work. Position-dependent rules like this and including via a directory does not work well together. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Tim Uckun > Sent: Friday, October 22, 2010 12:36 PM > To: rsyslog-users > Subject: Re: [rsyslog] postgres logging to three places.. > > On Thu, Oct 21, 2010 at 6:26 PM, Rainer Gerhards > wrote: > > do > > > > local0.* ? ? ? ? ? ? ? ?-/var/log/postgresql/postgresql-8.4-main.log > > & ~ > > > > I am afraid this did not work. > > To give you more details. > > I am running on ubuntu with postgres 8.4. Postgres is configured like > this. > > log_destination = 'syslog' > syslog_facility = 'LOCAL0' > > I created a file called postgresql.conf in the /etc/rsyslog.d/ > directory and now it looks like this. > > local0.* -/var/log/postgresql/postgresql-8.4-main.log > & ~ > > # > # add local0 to the ignore field for the normal messages so it wont' > get logged twice. > # > > *.=info;*.=notice;*.=warn;\ > local0.none -/var/log/messages > *.=info;*.=notice;*.=warn;\ > local0.none -/var/log/syslog > > > > I tried it with and without the bottom part (after the comments). > > No matter what I do every entry is written into /var/log/messages , > /var/log/syslog and /var/log/postgresql/postgresql-8.4-main.log > > I am reloading rsyslog with "reload rsyslog" when I do that both > syslog and messages have rsyslog messages on them which is also a bit > puzzling. > > The rest of the syslog configuration is in 50-default.conf which I > didn't touch but I have included here for reference. > > > > Default rules for rsyslog. > # > # For more information see rsyslog.conf(5) and > /etc/rsyslog.conf > > # > # First some standard log files. Log by facility. > # > auth,authpriv.* /var/log/auth.log > *.*;auth,authpriv.none -/var/log/syslog > #cron.* /var/log/cron.log > daemon.* -/var/log/daemon.log > kern.* -/var/log/kern.log > lpr.* -/var/log/lpr.log > mail.* -/var/log/mail.log > user.* -/var/log/user.log > > # > # Logging for the mail system. Split it up so that > # it is easy to write scripts to parse these files. > # > mail.info -/var/log/mail.info > mail.warn -/var/log/mail.warn > mail.err /var/log/mail.err > > # > # Logging for INN news system. > # > news.crit /var/log/news/news.crit > news.err /var/log/news/news.err > news.notice -/var/log/news/news.notice > > # > # Some "catch-all" log files. > # > *.=debug;\ > auth,authpriv.none;\ > news.none;mail.none -/var/log/debug > *.=info;*.=notice;*.=warn;\ > auth,authpriv.none;\ > cron,daemon.none;\ > mail,news.none -/var/log/messages > > # > # Emergencies are sent to everybody logged in. > # > *.emerg * > > # > # I like to have messages displayed on the console, but only on a > virtual > # console I usually leave idle. > # > #daemon,mail.*;\ > # news.=crit;news.=err;news.=notice;\ > # *.=debug;*.=info;\ > # *.=notice;*.=warn /dev/tty8 > > # The named pipe /dev/xconsole is for the `xconsole' utility. To use > it, > # you must invoke `xconsole' with the `-file' option: > # > # $ xconsole -file /dev/xconsole [...] > # > # NOTE: adjust the list below, or you'll go crazy if you have a > reasonably > # busy site.. > # > daemon.*;mail.*;\ > news.err;\ > *.=debug;*.=info;\ > *.=notice;*.=warn |/dev/xconsole > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From timuckun at gmail.com Fri Oct 22 12:58:14 2010 From: timuckun at gmail.com (Tim Uckun) Date: Fri, 22 Oct 2010 23:58:14 +1300 Subject: [rsyslog] postgres logging to three places.. In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: On Fri, Oct 22, 2010 at 11:40 PM, Rainer Gerhards wrote: > I guess the answer is simply that your file is included at the wrong spot > into the main config file. Move that code to the top of the main file and it > will work. Position-dependent rules like this and including via a directory > does not work well together. > OK I'll try that next. I also noticed that postfix is logging to both mail.log and syslog too. Very strange.. > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Tim Uckun >> Sent: Friday, October 22, 2010 12:36 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] postgres logging to three places.. >> >> On Thu, Oct 21, 2010 at 6:26 PM, Rainer Gerhards >> wrote: >> > do >> > >> > local0.* ? ? ? ? ? ? ? ?-/var/log/postgresql/postgresql-8.4-main.log >> > & ~ >> > >> >> I am afraid this did not work. >> >> To give you more details. >> >> I am running on ubuntu with postgres 8.4. Postgres is configured like >> this. >> >> log_destination = 'syslog' >> syslog_facility = 'LOCAL0' >> >> I created a file called postgresql.conf in the /etc/rsyslog.d/ >> directory and now it looks like this. >> >> local0.* ? ? ? ? ? ? ? ?-/var/log/postgresql/postgresql-8.4-main.log >> & ~ >> >> # >> # add local0 to the ignore field for the normal messages so it wont' >> get logged twice. >> # >> >> *.=info;*.=notice;*.=warn;\ >> ? ? ? ? local0.none ?-/var/log/messages >> *.=info;*.=notice;*.=warn;\ >> ? ? ? ? local0.none -/var/log/syslog >> >> >> >> I tried it with and without the bottom part (after the comments). >> >> No matter what I do every entry is written into /var/log/messages , >> /var/log/syslog and /var/log/postgresql/postgresql-8.4-main.log >> >> I am reloading rsyslog with "reload rsyslog" ?when I do that both >> syslog and messages have rsyslog messages on them which is also a bit >> puzzling. >> >> The rest of the syslog configuration is in 50-default.conf which I >> didn't touch but I have included here for reference. >> >> >> >> ? Default rules for rsyslog. >> # >> # ? ? ? ? ? ? ? ? ? ? ? For more information see rsyslog.conf(5) and >> /etc/rsyslog.conf >> >> # >> # First some standard log files. ?Log by facility. >> # >> auth,authpriv.* ? ? ? ? ? ? ? ? /var/log/auth.log >> *.*;auth,authpriv.none ? ? ? ? ?-/var/log/syslog >> #cron.* ? ? ? ? ? ? ? ? ? ? ? ? /var/log/cron.log >> daemon.* ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/daemon.log >> kern.* ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/kern.log >> lpr.* ? ? ? ? ? ? ? ? ? ? ? ? ? -/var/log/lpr.log >> mail.* ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/mail.log >> user.* ? ? ? ? ? ? ? ? ? ? ? ? ?-/var/log/user.log >> >> # >> # Logging for the mail system. ?Split it up so that >> # it is easy to write scripts to parse these files. >> # >> mail.info ? ? ? ? ? ? ? ? ? ? ? -/var/log/mail.info >> mail.warn ? ? ? ? ? ? ? ? ? ? ? -/var/log/mail.warn >> mail.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/mail.err >> >> # >> # Logging for INN news system. >> # >> news.crit ? ? ? ? ? ? ? ? ? ? ? /var/log/news/news.crit >> news.err ? ? ? ? ? ? ? ? ? ? ? ?/var/log/news/news.err >> news.notice ? ? ? ? ? ? ? ? ? ? -/var/log/news/news.notice >> >> # >> # Some "catch-all" log files. >> # >> *.=debug;\ >> ? ? ? ? auth,authpriv.none;\ >> ? ? ? ? news.none;mail.none ? ? -/var/log/debug >> *.=info;*.=notice;*.=warn;\ >> ? ? ? ? auth,authpriv.none;\ >> ? ? ? ? cron,daemon.none;\ >> ? ? ? ? mail,news.none ? ? ? ? ?-/var/log/messages >> >> # >> # Emergencies are sent to everybody logged in. >> # >> *.emerg ? ? ? ? ? ? ? ? ? ? ? ? * >> >> # >> # I like to have messages displayed on the console, but only on a >> virtual >> # console I usually leave idle. >> # >> #daemon,mail.*;\ >> # ? ? ? news.=crit;news.=err;news.=notice;\ >> # ? ? ? *.=debug;*.=info;\ >> # ? ? ? *.=notice;*.=warn ? ? ? /dev/tty8 >> >> # The named pipe /dev/xconsole is for the `xconsole' utility. ?To use >> it, >> # you must invoke `xconsole' with the `-file' option: >> # >> # ? ?$ xconsole -file /dev/xconsole [...] >> # >> # NOTE: adjust the list below, or you'll go crazy if you have a >> reasonably >> # ? ? ?busy site.. >> # >> daemon.*;mail.*;\ >> ? ? ? ? news.err;\ >> ? ? ? ? *.=debug;*.=info;\ >> ? ? ? ? *.=notice;*.=warn ? ? ? |/dev/xconsole >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From timuckun at gmail.com Fri Oct 22 13:21:51 2010 From: timuckun at gmail.com (Tim Uckun) Date: Sat, 23 Oct 2010 00:21:51 +1300 Subject: [rsyslog] postgres logging to three places.. In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: On Fri, Oct 22, 2010 at 11:58 PM, Tim Uckun wrote: > On Fri, Oct 22, 2010 at 11:40 PM, Rainer Gerhards > wrote: >> I guess the answer is simply that your file is included at the wrong spot >> into the main config file. Move that code to the top of the main file and it >> will work. Position-dependent rules like this and including via a directory >> does not work well together. >> > > OK I'll try that next. > > I also noticed that postfix is logging to both mail.log and syslog > too. ?Very strange.. > putting on the top of the file worked but renaming the config file 00-postfix.conf didn't even though theoretically it should have loaded the 00-postfix conf before the 50-default.conf Other strange happenings.... rsyslog itself is logging to both syslog and messages but postfix is logging to both syslog and mail.log, cron is logging only to syslog. Rsyslog logs twice into messages like this Oct 22 11:02:22 momentumdb rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="9853" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'. Oct 22 11:02:22 momentumdb rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="9853" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'. and once into syslog Oct 22 11:02:22 momentumdb rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="9853" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'. weird.. At least the postgres logging is sorted. Thanks. From david at lang.hm Fri Oct 22 21:00:32 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 22 Oct 2010 12:00:32 -0700 (PDT) Subject: [rsyslog] postgres logging to three places.. In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: On Sat, 23 Oct 2010, Tim Uckun wrote: > On Fri, Oct 22, 2010 at 11:58 PM, Tim Uckun wrote: >> On Fri, Oct 22, 2010 at 11:40 PM, Rainer Gerhards >> wrote: >>> I guess the answer is simply that your file is included at the wrong spot >>> into the main config file. Move that code to the top of the main file and it >>> will work. Position-dependent rules like this and including via a directory >>> does not work well together. >>> >> >> OK I'll try that next. >> >> I also noticed that postfix is logging to both mail.log and syslog >> too. ?Very strange.. >> > > putting on the top of the file worked but renaming the config file > 00-postfix.conf didn't even though theoretically it should have loaded > the 00-postfix conf before the 50-default.conf > > > Other strange happenings.... > Tim, rsyslog will log a message to however many different destinations that you tell it to. It doesn't log it to the first match and then stop processing the log, it continues to see if there is anywhere else that matches the message. if you want to have rsyslog stop processing a message after it's matched, you have to tell it to do so explicitly. the easy way to do this is to add a line & ~ after the line that matches the logs you don't want to see anywere else. the & says to use the same criteria as the last match, and the ~ says to stop processing this message. David Lang From timuckun at gmail.com Fri Oct 22 22:58:41 2010 From: timuckun at gmail.com (Tim Uckun) Date: Sat, 23 Oct 2010 09:58:41 +1300 Subject: [rsyslog] postgres logging to three places.. In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: > > rsyslog will log a message to however many different destinations that you > tell it to. It doesn't log it to the first match and then stop processing > the log, it continues to see if there is anywhere else that matches the > message. > I am surprised the default setup in ubuntu does this thouhg. Maybe they just took the syslog conf file and carried it across. > if you want to have rsyslog stop processing a message after it's matched, > you have to tell it to do so explicitly. > > the easy way to do this is to add a line > & ~ > after the line that matches the logs you don't want to see anywere else. > How come adding local0.none didn't help? Maybe that's because it was in a different file? From david at lang.hm Fri Oct 22 23:02:07 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 22 Oct 2010 14:02:07 -0700 (PDT) Subject: [rsyslog] postgres logging to three places.. In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: On Sat, 23 Oct 2010, Tim Uckun wrote: >> rsyslog will log a message to however many different destinations that you >> tell it to. It doesn't log it to the first match and then stop processing >> the log, it continues to see if there is anywhere else that matches the >> message. >> > > I am surprised the default setup in ubuntu does this thouhg. Maybe > they just took the syslog conf file and carried it across. syslog did the same thing, if you listed multiple lines that matched a log message, it would put it in multiple files. >> if you want to have rsyslog stop processing a message after it's matched, >> you have to tell it to do so explicitly. >> >> the easy way to do this is to add a line >> & ~ >> after the line that matches the logs you don't want to see anywere else. >> > > How come adding local0.none didn't help? Maybe that's because it was > in a different file? That would be a question for Rainer. David Lang From rgerhards at hq.adiscon.com Sat Oct 23 10:07:21 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sat, 23 Oct 2010 10:07:21 +0200 Subject: [rsyslog] postgres logging to three places.. References: <9B6E2A8877C38245BFB15CC491A11DA71DD53F@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD56A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD57C@GRFEXC.intern.adiscon.com> > > How come adding local0.none didn't help? Maybe that's because it was > > in a different file? > > That would be a question for Rainer. Please create a debug log oft he startup, that will tell us which filters were created. I don't' need any message flow for that, just the startup. Rainer From me at gavitron.com Tue Oct 26 06:19:47 2010 From: me at gavitron.com (Gavin McDonald) Date: Mon, 25 Oct 2010 21:19:47 -0700 Subject: [rsyslog] rsyslog + tls fails under load. Message-ID: Greetings, I recently tried installing rsyslog on a small server farm to do centralized logging over TLS. I am now seeing some unexpected behavior, and was hoping the list would have some insight; When I start/restart rsyslog on the central server, I get less than a minute or two of logs from the web farm, then nothing. the central server just stops receiving remote messages. (they don't stop being sent though, which leads to a compounding issue...) In an effort to debug the problem, I stopped the service, and ran rsyslogd manually, adding the -d option. Rsyslog appears to run ok. and I see no explicit errors, but the log clients themselves will start to seize after about 30 minutes, forcing at minimum an apache restart. (which I have since learned they will do, as long as there is no server to connect to.) If I run 'rsyslog -r -c4 -d > /dev/null' the original problem manifests. In test conditions under minimal load, (one log client, dev traffic only,) this doesn't happen at all. (or at least, takes far too long to happen.) I have found multiple bug-reports relating to the clients' halting problem, and understand that it may have even been fixed in the latest 5.x branch. Which is why I'm not asking about why the log clients need to be rebooted if the logging server goes down. What I am more interested in is how quickly rsyslog dies when not throttled by writing to the console. - We run a modest web farm, of about 60 LAMP servers, doing a sustained 10 access/s, with peaks around 30/s. I direct the apache access.log at rsyslog using the O'reilly recipe, and the messages appear to all make it through at pace until rsyslogd falls over on the central server. in -d mode, estimating 10-15 output lines per log message, I'd expect to be seeing ~10k debug lines per second of debug output given fictionally optimal console throughput. I know that my connection is incapable of this, thus I suspect that the lag introduced by actually viewing the debug messages through an ssh session is enough to delay the leak/race that makes rsyslog fall over so quickly when run as a proper daemon. This is further supported by the behaviour when dumping the debug output to /dev/null. As this issue manifested in production, I've rolled the configs back to simple insecure logging over tcp, and everything has been stable since. I ask, Is tls support _supposed_ to work in the 4.2.0-2 release, or am I just tilting at windmills? as for version, etc; All log clients are ubuntu 8.04 LTS, with rsyslog 4.2.0-2 installed from alessandro bono's ppa archive. ( https://launchpad.net/~a.bono/+archive/rsyslog) The log server is also running 4.2.0-2 but from the official repositories, as it is ubuntu 9.10. I have created the certificates for each machine, and authentication is (initially) successful. my client rsyslog.conf is as follows: (comments etc stripped for brevity, and obviously sanitized) $DefaultNetstreamDriver gtls $ModLoad imuxsock $ActionSendStreamDriverAuthMode x509/name $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem $DefaultNetstreamDriverCertFile /etc/tls/cert/client1.pem $DefaultNetstreamDriverKeyFile /etc/tls/hkey/client1.pem $ActionSendStreamDriverPermittedPeer logserver.mydomain.com $ActionSendStreamDriverMode 1 $RepeatedMsgReduction on $ActionFileDefaultTemplate RSYSLOG_ForwardFormat auth,authpriv.* /var/log/auth.log *.* @@logserver.mydomain.com:10514 The server config is nearly identical, save for the ruleset definitions in an include file, and the relevant cert changes. $ModLoad imuxsock $ModLoad imklog $KLogPath /var/run/rsyslog/kmsg $ModLoad imtcp $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem $DefaultNetstreamDriverCertFile /etc/tls/cert/server.pem $DefaultNetstreamDriverKeyFile /etc/tls/hkey/server.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.mydomain.com $InputTCPServerStreamDriverMode 1 $InputTCPServerRun 10514 $RepeatedMsgReduction on $CreateDirs on $FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $IncludeConfig /etc/rsyslog.d/*.conf Regards, -G Gavin McDonald EVI Logistic Enterprises From rgerhards at hq.adiscon.com Tue Oct 26 07:38:12 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 26 Oct 2010 07:38:12 +0200 Subject: [rsyslog] rsyslog + tls fails under load. References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD599@GRFEXC.intern.adiscon.com> Thanks for the long post, the answer is unfortunately very short: I know there is an issue, but I am trying for quite a while no to reproduce it without any aid. Some people have reported the issue, but nobody so far had the time to go through the (probably considerable) hassle of helping me to debug this (that requires multiple iterations of instrumentation and new tries as I do not have a solid idea what the cause is). Many folks who wanted to help also have not been able to reproduce the issue reliably enough (or it took them days). If you can help finding the cause, I would be more than happy to address it. (but unfortunately I will be out of office most part of tomorrow and Thursday). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gavin McDonald > Sent: Tuesday, October 26, 2010 6:20 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog + tls fails under load. > > Greetings, > > I recently tried installing rsyslog on a small server farm to do > centralized > logging over TLS. I am now seeing some unexpected behavior, and was > hoping > the list would have some insight; > > When I start/restart rsyslog on the central server, I get less than a > minute > or two of logs from the web farm, then nothing. the central server just > stops receiving remote messages. (they don't stop being sent though, > which > leads to a compounding issue...) In an effort to debug the problem, I > stopped the service, and ran rsyslogd manually, adding the -d option. > Rsyslog appears to run ok. and I see no explicit errors, but the log > clients themselves will start to seize after about 30 minutes, forcing > at > minimum an apache restart. (which I have since learned they will do, > as > long as there is no server to connect to.) If I run 'rsyslog -r -c4 -d > > > /dev/null' the original problem manifests. In test conditions under > minimal > load, (one log client, dev traffic only,) this doesn't happen at all. > (or at > least, takes far too long to happen.) > > I have found multiple bug-reports relating to the clients' halting > problem, > and understand that it may have even been fixed in the latest 5.x > branch. > Which is why I'm not asking about why the log clients need to be > rebooted > if the logging server goes down. What I am more interested in is how > quickly rsyslog dies when not throttled by writing to the console. - We > run > a modest web farm, of about 60 LAMP servers, doing a sustained 10 > access/s, > with peaks around 30/s. I direct the apache access.log at rsyslog > using the > O'reilly recipe, and the messages appear to all make it through at pace > until rsyslogd falls over on the central server. in -d mode, > estimating > 10-15 output lines per log message, I'd expect to be seeing ~10k debug > lines > per second of debug output given fictionally optimal console > throughput. I > know that my connection is incapable of this, thus I suspect that the > lag > introduced by actually viewing the debug messages through an ssh > session is > enough to delay the leak/race that makes rsyslog fall over so quickly > when > run as a proper daemon. This is further supported by the behaviour when > dumping the debug output to /dev/null. > > As this issue manifested in production, I've rolled the configs back to > simple insecure logging over tcp, and everything has been stable since. > I > ask, Is tls support _supposed_ to work in the 4.2.0-2 release, or am I > just > tilting at windmills? > > as for version, etc; All log clients are ubuntu 8.04 LTS, with rsyslog > 4.2.0-2 installed from alessandro bono's ppa archive. ( > https://launchpad.net/~a.bono/+archive/rsyslog) The log server is also > running 4.2.0-2 but from the official repositories, as it is ubuntu > 9.10. > > I have created the certificates for each machine, and authentication is > (initially) successful. my client rsyslog.conf is as follows: > (comments etc > stripped for brevity, and obviously sanitized) > $DefaultNetstreamDriver gtls > $ModLoad imuxsock > $ActionSendStreamDriverAuthMode x509/name > $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem > $DefaultNetstreamDriverCertFile /etc/tls/cert/client1.pem > $DefaultNetstreamDriverKeyFile /etc/tls/hkey/client1.pem > $ActionSendStreamDriverPermittedPeer logserver.mydomain.com > $ActionSendStreamDriverMode 1 > $RepeatedMsgReduction on > $ActionFileDefaultTemplate RSYSLOG_ForwardFormat > auth,authpriv.* /var/log/auth.log > *.* @@logserver.mydomain.com:10514 > > > The server config is nearly identical, save for the ruleset definitions > in > an include file, and the relevant cert changes. > $ModLoad imuxsock > $ModLoad imklog > $KLogPath /var/run/rsyslog/kmsg > $ModLoad imtcp > $DefaultNetstreamDriver gtls > $DefaultNetstreamDriverCAFile /etc/tls/pkey/ca.pem > $DefaultNetstreamDriverCertFile /etc/tls/cert/server.pem > $DefaultNetstreamDriverKeyFile /etc/tls/hkey/server.pem > $InputTCPServerStreamDriverAuthMode x509/name > $InputTCPServerStreamDriverPermittedPeer *.mydomain.com > $InputTCPServerStreamDriverMode 1 > $InputTCPServerRun 10514 > $RepeatedMsgReduction on > $CreateDirs on > $FileOwner syslog > $FileGroup adm > $FileCreateMode 0640 > $DirCreateMode 0755 > $Umask 0022 > $IncludeConfig /etc/rsyslog.d/*.conf > > > Regards, > > -G > > Gavin McDonald > EVI Logistic Enterprises > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From m.ehinger at ltur.de Fri Oct 29 14:20:57 2010 From: m.ehinger at ltur.de (Maik Ehinger) Date: Fri, 29 Oct 2010 14:20:57 +0200 Subject: [rsyslog] log files only created on start Message-ID: Hi, i deleted the log file while rsyslog is running by accident. If i look at the debug output (rsyslog -c5 -d -n) all seems fine. Although the log file doesn't exist! Is this expexed behaviour? No warning at all? If i restart rsyslog the file will be created and all is fine again. I'm using version 5.6.0 on FreeBSD 8.1. Thanks Maik L'TUR Tourismus AG - Europe's No. 1 for Last-Minute-Travel - with 169 Shops in six European countries L'TUR Tourismus AG, Augustaplatz 8, 76530 Baden-Baden, Vorsitzender des Aufsichtsrats: Karlheinz Koegel Vorsitzender des Vorstandes: Markus Orth - Vorstand: Kai Klitzke, Oliver Kluth, Michael Specht Registergericht: Amtsgericht Mannheim HRB 201600 *** Legal Disclaimer *** Diese E-Mail ist vertraulich. Bitte informieren Sie uns unverz?glich falls Sie nicht der bestimmungsgem??e Empf?nger sind. This e-mail is confidential. Please notify us immediately if you are not the intended recipient. From rgerhards at hq.adiscon.com Fri Oct 29 14:22:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 29 Oct 2010 14:22:54 +0200 Subject: [rsyslog] log files only created on start References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD5E6@GRFEXC.intern.adiscon.com> That's the usual Unix/Linux way of things. If you delete a file, you will no longer see it, but existing processes can still write to it. Nothing rsyslog-specific. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Maik Ehinger > Sent: Friday, October 29, 2010 2:21 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] log files only created on start > > Hi, > > i deleted the log file while rsyslog is running by accident. > > If i look at the debug output (rsyslog -c5 -d -n) all seems fine. > Although > the log file doesn't exist! > > Is this expexed behaviour? No warning at all? > > If i restart rsyslog the file will be created and all is fine again. > > I'm using version 5.6.0 on FreeBSD 8.1. > > Thanks > Maik > > > > > > > L'TUR Tourismus AG - Europe's No. 1 for Last-Minute-Travel - with 169 > Shops in six European countries > L'TUR Tourismus AG, Augustaplatz 8, 76530 Baden-Baden, Vorsitzender des > Aufsichtsrats: Karlheinz Koegel > Vorsitzender des Vorstandes: Markus Orth - Vorstand: Kai Klitzke, > Oliver Kluth, Michael Specht > Registergericht: Amtsgericht Mannheim HRB 201600 > > *** Legal Disclaimer *** > Diese E-Mail ist vertraulich. Bitte informieren Sie uns unverz?glich > falls Sie nicht der bestimmungsgem??e Empf?nger sind. > This e-mail is confidential. Please notify us immediately if you are > not the intended recipient. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From ktm at rice.edu Fri Oct 29 14:27:30 2010 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 29 Oct 2010 07:27:30 -0500 Subject: [rsyslog] log files only created on start In-Reply-To: References: Message-ID: <20101029122730.GG27429@aart.is.rice.edu> On Fri, Oct 29, 2010 at 02:20:57PM +0200, Maik Ehinger wrote: > Hi, > > i deleted the log file while rsyslog is running by accident. > > If i look at the debug output (rsyslog -c5 -d -n) all seems fine. Although > the log file doesn't exist! > > Is this expexed behaviour? No warning at all? > > If i restart rsyslog the file will be created and all is fine again. > > I'm using version 5.6.0 on FreeBSD 8.1. > > Thanks > Maik > > Yes, this is correct behavior. A filehandle to the file is held by rsyslog so it can continue to log to the file even though you have unlinked it by deleting it from another process. There is not really a need for a warning because if you did not want to delete the file, you would not have done so. Obviously, you should make the deletion require a privileged user in general, but that is simply System Admin 101. Cheers, Ken From david at lang.hm Fri Oct 29 20:38:51 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 29 Oct 2010 11:38:51 -0700 (PDT) Subject: [rsyslog] log files only created on start In-Reply-To: References: Message-ID: On Fri, 29 Oct 2010, Maik Ehinger wrote: > Hi, > > i deleted the log file while rsyslog is running by accident. > > If i look at the debug output (rsyslog -c5 -d -n) all seems fine. Although > the log file doesn't exist! > > Is this expexed behaviour? No warning at all? > > If i restart rsyslog the file will be created and all is fine again. you don't need to do a full restart, a kill -HUP should be enough to have rsyslog re-open the files. it would be very slow and inefficent to have rsyslog open and close the file for each write, but by keeping the file open, if you rm or mv a logfile, rsyslog will keep writing to that same file (in it's new location with a mv to the same filessytem, otherwise to a file that still exists and takes up disk space, but that nothing else can open) David Lang From Brianr at mylife.com Sat Oct 30 01:46:01 2010 From: Brianr at mylife.com (Brian Rogoff) Date: Fri, 29 Oct 2010 16:46:01 -0700 Subject: [rsyslog] rsyslog conf file questions Message-ID: <615F3D61E5B3E74AB3C81CF3904B922503FD56B9@maximus.REUNION.COM> Hi, I'm considering using rsyslog to replace the custom logging system of a distributed program, and I had some questions about how to do this, and whether my tentative design is reasonable. Assume all machines are running a recent rsyslog (>= 5.6) on Linux. We use logging for system monitoring and performance, not for security. I plan to use the rsyslog time stamps to calculate the time through our system. The program runs on several clusters, and consists of numerous services/processes. For concreteness, let's just look at one cluster, and give each machine on it a name like cluster_01, cluster_02, etc. I'd like to designate two machines in each cluster as being the log servers; two to provide some redundancy for failures. Let cluster_01 and cluster_02 be the logging servers. There are about 20 machines per cluster, so two logging servers will collect for about 18 machines, each of which is running about 6 (six) services that we log events from. The services get 50 - 100 requests/second, all have 1Gbps Ethernet links into the same switch and disks that are fast enough to easily deal with the write bandwidth needed. I'd like the log messages from each service on each machine to go to a log files something like this cluster_01:/var/log/cluster_03 --cluster_04 . . . --cluster_nn cluster_01:/var/log/cluster_03/service1/log --service5/log --service7/log First question, does this make sense? Both as an explanation of what I'm trying to do, and as a reasonable logging architecture? Second question. What should my rsyslog.conf look like on each machine? On the logging servers, I'd like all messages not from the server machine to be stored in the log file with path determined from hostname, service name, and pid, say. I may need more info later to assure that I can trace the path of messages more easily, but this should be sufficient for starters. I'd prefer that the conf files do not hardcode the names of the other machines on themselves. On the cluster machines running the services, I'd like the conf files to all be exactly the same, so they may refer to the logging servers by name but not themselves. I has been very slow going for me trying to figure out the syntax to do all of this. I looked at the example conf files, and I was able to use expression based filters to get some of the way there but I think everything I described should be doable with rsyslog. Third, are there any subtle issues I should be thinking about here. For example, since I'd like to use the log messages to calculate the performance of the services, do I need to introduce some extra time stamps in the message flow or are the rsyslog generated timestamps enough? -- Brian From david at lang.hm Sat Oct 30 02:21:09 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 29 Oct 2010 17:21:09 -0700 (PDT) Subject: [rsyslog] rsyslog conf file questions In-Reply-To: <615F3D61E5B3E74AB3C81CF3904B922503FD56B9@maximus.REUNION.COM> References: <615F3D61E5B3E74AB3C81CF3904B922503FD56B9@maximus.REUNION.COM> Message-ID: On Fri, 29 Oct 2010, Brian Rogoff wrote: > Hi, > I'm considering using rsyslog to replace the custom logging system of a > distributed program, and I had some questions about how to do this, and whether > my tentative design is reasonable. Assume all machines are running a recent > rsyslog (>= 5.6) on Linux. > > We use logging for system monitoring and performance, not for security. > I plan to use the rsyslog time stamps to calculate the time through our > system. > > The program runs on several clusters, and consists of numerous > services/processes. For concreteness, let's just look at one cluster, > and give each machine on it a name like cluster_01, cluster_02, etc. > I'd like to designate two machines in each cluster as being the log > servers; two to provide some redundancy for failures. Let cluster_01 > and cluster_02 be the logging servers. There are about 20 machines > per cluster, so two logging servers will collect for about 18 machines, > each of which is running about 6 (six) services that we log events from. > The services get 50 - 100 requests/second, all have 1Gbps Ethernet links > into the same switch and disks that are fast enough to easily deal with > the write bandwidth needed. > > I'd like the log messages from each service on each machine to go to a > log files something like this > > cluster_01:/var/log/cluster_03 > --cluster_04 > . > . > . > --cluster_nn > > cluster_01:/var/log/cluster_03/service1/log > --service5/log > --service7/log > > First question, does this make sense? Both as an explanation of what I'm > trying to do, and as a reasonable logging architecture? sure, I may end up doing it somewhat differently (but I don't know all the details), there's nothing fundamentally wrong with what you are doing. what is the purpose of having everything broken out into individual files like this? I see a lot of people who start off trying to do something like this and then try to run reports across everything (where it would be much simpler if everything was just in one file) throughput wise, there is nothing you havve said here that should strain rsyslog, even on pretty modest hardware. > Second question. What should my rsyslog.conf look like on each machine? > On the logging servers, I'd like all messages not from the server machine > to be stored in the log file with path determined from hostname, service name, > and pid, say. I may need more info later to assure that I can trace the path > of messages more easily, but this should be sufficient for starters. I'd > prefer that the conf files do not hardcode the names of the other machines > on themselves. On the cluster machines running the services, I'd like the > conf files to all be exactly the same, so they may refer to the logging > servers by name but not themselves. I has been very slow going for me trying > to figure out the syntax to do all of this. I looked at the example conf files, > and I was able to use expression based filters to get some of the way there > but I think everything I described should be doable with rsyslog. everything you are trying to do is possible, but getting it fully setup with everything you are trying to do is quite a bit of work. This request probably isn't inteded to sound like 'do my homework for me', but it's sounding pretty close to that. Adiscon does offer a service to work with you to do exactly this, but the level of detail you are asking for seems like it exceeds normal mailing list support (for the record, I do not work for adiscon) I think that you should look at dynafiles for hints on how to do what it sounds like you want to do. > Third, are there any subtle issues I should be thinking about here. For > example, since I'd like to use the log messages to calculate the performance > of the services, do I need to introduce some extra time stamps in the message > flow or are the rsyslog generated timestamps enough? you really haven't provided enough information for this question. when you say you want to measure the performance of a service, what does that mean? what are you measuring? rsyslog will log the time that it received the message. If you need to measure something like hits per second, this may be good enough. But if you needed to measure how long it took to service each individual request, the rsyslog timestamp is almost worthless. David Lang From Brianr at mylife.com Sat Oct 30 04:25:17 2010 From: Brianr at mylife.com (Brian Rogoff) Date: Fri, 29 Oct 2010 19:25:17 -0700 Subject: [rsyslog] rsyslog conf file questions References: <615F3D61E5B3E74AB3C81CF3904B922503FD56B9@maximus.REUNION.COM> Message-ID: <615F3D61E5B3E74AB3C81CF3904B922503FD56BA@maximus.REUNION.COM> -----Original Message----- From: rsyslog-bounces at lists.adiscon.com on behalf of david at lang.hm Sent: Fri 10/29/2010 5:21 PM To: rsyslog-users Subject: Re: [rsyslog] rsyslog conf file questions > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com on behalf of david at lang.hm > Sent: Fri 10/29/2010 5:21 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog conf file questions > > > On Fri, 29 Oct 2010, Brian Rogoff wrote: > > ... > what is the purpose of having everything broken out into individual files > like this? In the current set up, log files are broken out into individual files much like what I described, so I thought for a first pass I'd try and preserve that structure as much as possible. > I see a lot of people who start off trying to do something like > this and then try to run reports across everything (where it would be much > simpler if everything was just in one file) Do you see a disadvantage to doing it this way? > Second question. What should my rsyslog.conf look like on each machine? > On the logging servers, I'd like all messages not from the server machine > to be stored in the log file with path determined from hostname, service name, > and pid, say. I may need more info later to assure that I can trace the path > of messages more easily, but this should be sufficient for starters. I'd > prefer that the conf files do not hardcode the names of the other machines > on themselves. On the cluster machines running the services, I'd like the > conf files to all be exactly the same, so they may refer to the logging > servers by name but not themselves. I has been very slow going for me trying > to figure out the syntax to do all of this. I looked at the example conf files, > and I was able to use expression based filters to get some of the way there > but I think everything I described should be doable with rsyslog. > everything you are trying to do is possible, but getting it fully setup > with everything you are trying to do is quite a bit of work. I'm a bit surprised. What I described is really pretty simple, the skeleton of a first pass at redoing our logging with rsyslog. If it is quite a bit of work to do this in rsyuslog (and I don't doubt you, I've failed to implement it so far!) then I wonder whether it is worth replacing the existing home grown logging at all. > This request probably isn't inteded to sound like 'do my homework for me', but it's > sounding pretty close to that. No, I work for a company, I'm not a student. I looked at the examples and docs and couldn't figure out how to do what I want. I could have written a very short message just asking for hints, but I thought more context would be helpful. Sorry if it appears that I'm just lazily asking you to do my work (well, I am!); I'd be happy with a pointer to some similar setup that I could modify. > Adiscon does offer a service to work with you to do exactly this, but the level of detail > you are asking for seems like it exceeds normal mailing list support (for the record, I do > not work for adiscon) Conceptually, this looks like less that 10 lines of code in each .conf file, so it doesn't seem very detailed or complex. If it is complex to implement in rsyslog, that suggests that the tool wasn't designed with the problem I described in mind, and that I should look elsewhere, or just use our homebrew logger. > I think that you should look at dynafiles for hints on how to do what it > sounds like you want to do. Thanks. > > Third, are there any subtle issues I should be thinking about here. For > > example, since I'd like to use the log messages to calculate the performance > > of the services, do I need to introduce some extra time stamps in the message > > flow or are the rsyslog generated timestamps enough? > > you really haven't provided enough information for this question. when you > say you want to measure the performance of a service, what does that mean? > what are you measuring? Response times for each service. From your answer below, it would appear that I'll have to encode that information for each service and store it in the log message. > > rsyslog will log the time that it received the message. If you need to > measure something like hits per second, this may be good enough. But if > you needed to measure how long it took to service each individual request, > the rsyslog timestamp is almost worthless. -- Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/ms-tnef Size: 4834 bytes Desc: not available URL: From Brianr at mylife.com Sat Oct 30 05:24:07 2010 From: Brianr at mylife.com (Brian Rogoff) Date: Fri, 29 Oct 2010 20:24:07 -0700 Subject: [rsyslog] rsyslog conf file questions References: <615F3D61E5B3E74AB3C81CF3904B922503FD56B9@maximus.REUNION.COM> Message-ID: <615F3D61E5B3E74AB3C81CF3904B922503FD56BC@maximus.REUNION.COM> Dave, I may have been going about this the wrong way, thinking I should use all of rsyslog's fancy features. Since I can already use expression based filters and redirect messages to the server, I can do the rest by putting tags in my messages and have my log analysis programs be tag aware. That way rsyslog doesn't do much at all. Thanks for the prod. -- Brian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/ms-tnef Size: 2610 bytes Desc: not available URL: From john at feurix.com Sun Oct 31 14:35:34 2010 From: john at feurix.com (John Feuerstein) Date: Sun, 31 Oct 2010 14:35:34 +0100 Subject: [rsyslog] epoll() for RELP? Message-ID: <4CCD70A6.7080105@feurix.com> Greetings, I stumbled across the --enable-unlimited-select configure option [1] and an older conversation with Rainer on the mailing list [2] indicating that epoll() is only supported for imudp and the plain tcp netstream driver. What is the current status for stable version 5.6.0 and epoll() for all other inputs? I'm especially interested in epoll() support for RELP. FYI, I'm unable to compile 5.6.0 with --enable-unlimited-select: > CCLD lmnsd_ptcp.la > .libs/lmnsd_ptcp_la-nsdpoll_ptcp.o: In function `freeFdSet': > /home/john/Projects/src/rsyslog/rsyslog-5.6.0/runtime/unlimited_select.h:39: multiple definition of `freeFdSet' > .libs/lmnsd_ptcp_la-nsdsel_ptcp.o:/home/john/Projects/src/rsyslog/rsyslog-5.6.0/runtime/unlimited_select.h:39: first defined here > collect2: ld returned 1 exit status > make[2]: *** [lmnsd_ptcp.la] Error 1 Best regards, John [1] http://www.gossamer-threads.com/lists/rsyslog/users/3029 [2] http://www.mail-archive.com/rsyslog at lists.adiscon.com/msg02950.html