From champ at softwink.com Wed Dec 1 21:38:44 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 1 Dec 2010 15:38:44 -0500 Subject: [rsyslog] Fun with liblognorm. In-Reply-To: <20101020175951.GA13070@bundy.vistech.net> References: <20101020175951.GA13070@bundy.vistech.net> Message-ID: <20101201203844.GA11736@bundy.vistech.net> I had a chance to build and play a bit with liblognorm. Rainer has done some good work on it! It was fun to play with and here are some of my initial thoughts. First off, I was using a snapshots from the git repository for libestr, libee and liblognorm. Compiling/building was pretty straight forward. Untar the archives and run "autoreconf -vfi", and then run "./configure && make && make install". When running 'autoreconf' with liblognorm, it complains about the m4 directory not existing. Not a big deal, simply create it and re-run 'autoreconf'. Once everything was built, I started playing with Rainer's 'normalizer' example that ships with liblognorm. That was sort of nice to have to 'tie everything' together. It basically takes the log input via stdin and you point it to a log normalizer repository with the -r flag.. This program (well, wrapper) is in the liblognorm/src directory. You'd run it like this, assuming you're in the liblognorm/src directory... cat log-messages | ./normalizer -r ../sample-db/messages.sampdb In the 'samble-db' are sort of the 'rules' for parsing. I wrote a couple of test 'rules' just to play. Here's a quick little NTP rule I wrote: :%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: synchronized to %ip:ipv4% My syslog "input" is.. Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 Here's the normlizer output: To normalize: 'Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.1' normalized: '[cee at ... ip="192.168.0.1" tag="ntpd[3821\]" host="testbox" date="Dec 1 14:10:11"]' So, that work nicely. Nifty. I made a few more 'complex' rules, and those worked fine as well. However, if the rule is off a bit, then you've got issue. Here's what I mean.. Back on my example above.. If this: Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 changes to this: Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10, stratium 1 The 'normalizer' will call the ",stratium 1" part of the message as "unclassified". However, it doesn't appear that it'll grab the IP address, tag, etc. Also, I thing the "real work" is going to be writing rules. That's going to take some effort, in which I hope to assist with. Anyways, just wanted to share my brief bit of playing with liblognorm. It looks very promising! I'm hoping to start playing with some test Sagan + liblognorm code.. just for fun. :) -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From mark at thermeon.com Wed Dec 1 21:46:51 2010 From: mark at thermeon.com (Mark Olliver) Date: Wed, 1 Dec 2010 20:46:51 +0000 Subject: [rsyslog] Execute Alert script Message-ID: Hi, I am new to rsyslog but have found it very capable. I am using it via logger to log output of some of my scripts to local5. However I do want rsyslog to alert me via an alert script should it encounter a warning or error message sent to the log. I have tested my alert script and that works ok, i give it two arguments subject and message. However I can not seem to work out how to make this work from rsyslog. I tried the following: $template Alert,"'Rsyslog Alert' :msg" local5.info :msg, content "warning" ^/usr/bin/alert_script Alert Ideally i want the Subject to be what i set it to, the message to be everything that comes after the word warning. I would also only like one alert every 24 hours. Any ideas or help would be gratefully received. Thanks Mark From david at lang.hm Wed Dec 1 22:14:08 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 1 Dec 2010 13:14:08 -0800 (PST) Subject: [rsyslog] Execute Alert script In-Reply-To: References: Message-ID: On Wed, 1 Dec 2010, Mark Olliver wrote: > Hi, > > I am new to rsyslog but have found it very capable. I am using it via logger > to log output of some of my scripts to local5. However I do want rsyslog to > alert me via an alert script should it encounter a warning or error message > sent to the log. > > I have tested my alert script and that works ok, i give it two arguments > subject and message. However I can not seem to work out how to make this > work from rsyslog. > > I tried the following: > > $template Alert,"'Rsyslog Alert' :msg" > local5.info > :msg, content "warning" > ^/usr/bin/alert_script Alert > > > Ideally i want the Subject to be what i set it to, the message to be > everything that comes after the word warning. I would also only like one > alert every 24 hours. > > Any ideas or help would be gratefully received. rsyslog doesn't keep a history of what it's done, so it can't limit you to one alert every 24 hours, you really want another tool (something like SEC) for that. when you are creating templates, you put the properties you want between % characters (so instead of :msg, do %msg%) David Lang From rgerhards at hq.adiscon.com Thu Dec 2 08:32:17 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 08:32:17 +0100 Subject: [rsyslog] Feedback on ommysql requested Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> Hi all, I just received an interesting patch via the bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=213 What I see looks good to me, but I am not really in into MySQL. Do you think this is something I should merge? If not, why not? I'll wait until monday and will merge if I have not heard any good argument against that. Thanks, Rainer From rgerhards at hq.adiscon.com Thu Dec 2 08:45:11 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 08:45:11 +0100 Subject: [rsyslog] Execute Alert script References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD803@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, December 01, 2010 10:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] Execute Alert script > > On Wed, 1 Dec 2010, Mark Olliver wrote: > > > Hi, > > > > I am new to rsyslog but have found it very capable. I am using it via > logger > > to log output of some of my scripts to local5. However I do want > rsyslog to > > alert me via an alert script should it encounter a warning or error > message > > sent to the log. > > > > I have tested my alert script and that works ok, i give it two > arguments > > subject and message. However I can not seem to work out how to make > this > > work from rsyslog. > > > > I tried the following: > > > > $template Alert,"'Rsyslog Alert' :msg" > > local5.info > > :msg, content "warning" > > ^/usr/bin/alert_script Alert > > > > > > Ideally i want the Subject to be what i set it to, the message to be > > everything that comes after the word warning. I would also only like > one > > alert every 24 hours. > > > > Any ideas or help would be gratefully received. > > rsyslog doesn't keep a history of what it's done, so it can't limit you > to > one alert every 24 hours, you really want another tool (something like > SEC) for that. It's actually possible: $ActionExecOnlyOnceEveryInterval allows you do this kind of thing. This is the only sense of history rsyslog has, but there are many uses for it. I think I introduced it when I wrote ommail, which really needs such a capability. As such, a sample of its use can be found in the ommail doc: http://www.rsyslog.com/doc/ommail.html Rainer > > when you are creating templates, you put the properties you want > between % > characters (so instead of :msg, do %msg%) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Dec 2 09:22:46 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 09:22:46 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> Thanks for the nice review and instructions :) I have begun to work heavily on a message modfication module for rsyslog which will support liblognorm-style normalization inside rsyslog. In git there already is a branch "lognorm", which I will hopefully complete and merge into master soon. It provides some *very* interesting shortcuts of pulling specific information out of syslog messages. I'll probably promote it some more when it is available. IMHO it's the coolest and potentially most valuable feature I have added in the past three years. Once I have enabled tags in liblognorm/libee, you can even very easily classify log messages based on their content. More inline below... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Wednesday, December 01, 2010 9:39 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Fun with liblognorm. > > > I had a chance to build and play a bit with liblognorm. Rainer > has done some good work on it! It was fun to play with and here are > some of my initial thoughts. > > First off, I was using a snapshots from the git repository for > libestr, libee and liblognorm. Compiling/building was pretty straight > forward. Untar the archives and run "autoreconf -vfi", and then run > "./configure && make && make install". When running 'autoreconf' with > liblognorm, it complains about the m4 directory not existing. Not a > big deal, simply create it and re-run 'autoreconf'. I did a couple of bug fixes yesterday. Frequently pull from git ;) > > Once everything was built, I started playing with Rainer's > 'normalizer' example that ships with liblognorm. That was sort of > nice > to have to 'tie everything' together. It basically takes the log input > via stdin and you point it to a log normalizer repository with the -r > flag.. This program (well, wrapper) is in the liblognorm/src > directory. You'd run it like this, assuming you're in the > liblognorm/src directory... > > cat log-messages | ./normalizer -r ../sample-db/messages.sampdb > > In the 'samble-db' are sort of the 'rules' for parsing. I wrote > a couple of test 'rules' just to play. Here's a quick little NTP rule > I wrote: > > :%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: synchronized to > %ip:ipv4% > > My syslog "input" is.. > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > Here's the normlizer output: > > To normalize: 'Dec 1 14:10:11 testbox ntpd[3821]: synchronized to > 192.168.0.1' > normalized: '[cee at ... ip="192.168.0.1" tag="ntpd[3821\]" host="testbox" > date="Dec 1 14:10:11"]' > > So, that work nicely. Nifty. I made a few more 'complex' > rules, and those worked fine as well. I added a capability to generate graphs of the actual call tree. I think this is *very* useful. An article on how to do that will be posted soon to the web site (will make sure a notification goes to the list). > However, if the rule is off a > bit, then you've got issue. Here's what I mean.. Back on my example > above.. If this: > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > changes to this: > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10, > stratium 1 Well, that's by intention. The normalizer must know exactly which message it is dealing with. This is even more important when we use it for classification. So these two messages are definitely different, and I would consider it very dangerous to automatically merge them into a single one (which would not be a problem from a purely implementation PoV). The more fuzzy the recognition is, the higher is the chance of false recognition, something that would be really, really bad in the context of normalization. > The 'normalizer' will call the ",stratium 1" part of the message as > "unclassified". However, it doesn't appear that it'll grab the IP > address, tag, etc. > > Also, I thing the "real work" is going to be writing rules. That's > going to take some effort, in which I hope to assist with. Yes, that's definitely a lot of work -- and more than what a single person can do. In order to make the normalizer really useful, we need a community effort. If everyone contributes sample databases for their devices, we could gain good results fast. But the key is getting enough momentum, so please help spread the word! Rainer > Anyways, just wanted to share my brief bit of playing with liblognorm. > It looks very promising! I'm hoping to start playing with some test > Sagan + liblognorm code.. just for fun. :) > > > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From dave at fly.srk.fer.hr Thu Dec 2 12:30:23 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 2 Dec 2010 12:30:23 +0100 Subject: [rsyslog] Feedback on ommysql requested In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> Message-ID: <20101202113023.GA29993@fly.srk.fer.hr> Rainer Gerhards wrote: > Hi all, > > I just received an interesting patch via the bug tracker: > > http://bugzilla.adiscon.com/show_bug.cgi?id=213 > > What I see looks good to me, but I am not really in into MySQL. Do you think > this is something I should merge? If not, why not? I'll wait until monday and > will merge if I have not heard any good argument against that. It says the config file is taken from one of the following locations: ====== my.cnf .my.cnf ~/.my.cnf /etc/my.cnf ====== The first two depend on CWD. I'm not a fan of locating configuration files based on the current working directory. That leads to trouble, particularly because CWD is searched before absolute path locations. So if a file called my.cnf somehow ends in rsyslogd CWD (which is probably / by default, but I haven't checked), rsyslogd will change its behavior, which is bad. I would prefer having a directive in the rsyslog configuration file which can specify my.cnf location. If more than one instance of ommysql is used each one should have its own (I suppose that works out of the box). If that directive is missing, fall back to: 1. ~/.my.cnf 2. compile-time directive (defaults to $prefix/etc/my.cnf) There is a problem with NFS mounted programs and Solaris zones (and maybe AIX which recently got something like Solaris zones). In such settings you'd have programs (mysql, in this case) in /opt/foo/bin and the default location for the config file wil be /etc/opt/foo/prog.conf. That allows mounting /opt/foo read-only and having configuration file in the writeable part of the file system. The compile-time directive should take a list of directories, so one could specify: ./configure --with-mysql-config=/etc/opt/foo/my.cnf:/opt/foo/etc/my.cnf The configure script could autodetect such things (if $prefix is /opt/foo, then use the above default), but if that's too much work, just leave it to the person who is compiling the source. I wouldn't use CWD to locate any configuration file. You could also try locating the config file in run-time based on the location of the mysql client library. That is not portable, but it would work on systems which have dlinfo() implemented (Solaris, Linux and FreeBSD, at least). In that case you'd get an installation prefix for the mysql client library via dladdr() and dlinfo() and then try to guess the config file location. That might be somewhat problematic, though, in cases when the location is something like /usr/lib64/mysql/libmysqlclient.so.15. If the above run-time detection is implemented, than it should be in addition to the hard-coded locations, not instead of them. And the configure script should have a directive to disable that code (or to enable it, if disabled by default). -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From dave at fly.srk.fer.hr Thu Dec 2 13:47:31 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 2 Dec 2010 13:47:31 +0100 Subject: [rsyslog] Can't use PrivDrop with TCP listener Message-ID: <20101202124731.GA4132@fly.srk.fer.hr> With rsyslog 5.6.2 I tried to use UDP and TCP listeners on port 514 in combination with privilege drop. Config file contains: $PrivDropToUser bulb $PrivDropToGroup bulb $Ruleset indata $RulesetCreateMainQueue on $InputTCPServerBindRuleset indata $InputTCPServerRun 514 $InputUDPServerBindRuleset indata $UDPServerRun 514 I get this error: rsyslogd: Could not create tcp listener, ignoring port 514. [try http://www.rsyslog.com/e/2077 ] strace shows: 28240 socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 4 28240 setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 28240 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 28240 uname({sys="Linux", node="ssm2.nat.myrio.net", ...}) = 0 28240 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) 28240 fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 28240 bind(4, {sa_family=AF_INET6, sin6_port=htons(514), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 That was IPv6 UDP socket. 28240 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 5 28240 setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 28240 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) 28240 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 28240 bind(5, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 That was IPv4 UDP socket. 28241 setuid(1004) = 0 28241 futex(0x40bded1c, FUTEX_WAKE_PRIVATE, 1) = 0 28241 futex(0x7fffc9f38600, FUTEX_WAKE_PRIVATE, 1 28240 <... futex resumed> ) = 0 28240 setuid(1004) = 0 That was privilege drop. 28243 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 9 28243 setsockopt(9, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 28243 bind(9, {sa_family=AF_INET6, sin6_port=htons(514), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EACCES (Permission denied) 28243 close(9) = 0 28243 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 28243 bind(9, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied) Those were TCP sockets. Bind() fails because it happens after privilege drop. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Thu Dec 2 13:49:51 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 13:49:51 +0100 Subject: [rsyslog] Can't use PrivDrop with TCP listener References: <20101202124731.GA4132@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD80C@GRFEXC.intern.adiscon.com> Known issue, the full story is here: http://wiki.rsyslog.com/index.php/Security > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Thursday, December 02, 2010 1:48 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Can't use PrivDrop with TCP listener > > With rsyslog 5.6.2 I tried to use UDP and TCP listeners on port 514 in > combination with privilege drop. Config file contains: > > $PrivDropToUser bulb > $PrivDropToGroup bulb > > $Ruleset indata > $RulesetCreateMainQueue on > > $InputTCPServerBindRuleset indata > $InputTCPServerRun 514 > $InputUDPServerBindRuleset indata > $UDPServerRun 514 > > I get this error: > > rsyslogd: Could not create tcp listener, ignoring port 514. [try > http://www.rsyslog.com/e/2077 ] > > strace shows: > > 28240 socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 4 > 28240 setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 > 28240 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 28240 uname({sys="Linux", node="ssm2.nat.myrio.net", ...}) = 0 > 28240 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) > 28240 fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > 28240 bind(4, {sa_family=AF_INET6, sin6_port=htons(514), > inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, > sin6_scope_id=0}, 28) = 0 > > That was IPv6 UDP socket. > > 28240 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 5 > 28240 setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 28240 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) > 28240 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > 28240 bind(5, {sa_family=AF_INET, sin_port=htons(514), > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 > > That was IPv4 UDP socket. > > 28241 setuid(1004) = 0 > 28241 futex(0x40bded1c, FUTEX_WAKE_PRIVATE, 1) = 0 > 28241 futex(0x7fffc9f38600, FUTEX_WAKE_PRIVATE, 1 > 28240 <... futex resumed> ) = 0 > 28240 setuid(1004) = 0 > > That was privilege drop. > > 28243 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 9 > 28243 setsockopt(9, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 > 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) > 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > 28243 bind(9, {sa_family=AF_INET6, sin6_port=htons(514), > inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, > sin6_scope_id=0}, 28) = -1 EACCES (Permission denied) > 28243 close(9) = 0 > 28243 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 > 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) > 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > 28243 bind(9, {sa_family=AF_INET, sin_port=htons(514), > sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied) > > Those were TCP sockets. Bind() fails because it happens after privilege > drop. > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From champ at softwink.com Thu Dec 2 16:00:01 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 10:00:01 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> Message-ID: <20101202150001.GA30820@bundy.vistech.net> On Thu, Dec 02, 2010 at 09:22:46AM +0100, Rainer Gerhards wrote: > Thanks for the nice review and instructions :) > > I have begun to work heavily on a message modification module for rsyslog > which will support liblognorm-style normalization inside rsyslog. In git > there already is a branch "lognorm", which I will hopefully complete and > merge into master soon. It provides some *very* interesting shortcuts of In the rsyslog git tree? I have a specific reason for asking, which I'll get to later... > pulling specific information out of syslog messages. I'll probably promote it > some more when it is available. IMHO it's the coolest and potentially most > valuable feature I have added in the past three years. Once I have enabled > tags in liblognorm/libee, you can even very easily classify log messages > based on their content. To be honest, it's not only going to be a valuable addition to rsyslog, but my software as well... Not only that, I think once people 'understand it's function', I could see many other projects benefiting from it! > I did a couple of bug fixes yesterday. Frequently pull from git ;) Oh! I know it's a moving target! :) I'll git it frequently! > > So, that work nicely. Nifty. I made a few more 'complex' > > rules, and those worked fine as well. > > I added a capability to generate graphs of the actual call tree. I think this > is *very* useful. An article on how to do that will be posted soon to the web > site (will make sure a notification goes to the list). Yes, please do! I'd seen mention in the source about this, but didn't really dive into what you where trying to do there. > > However, if the rule is off a > > bit, then you've got issue. Here's what I mean.. Back on my example > > above.. If this: > > > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > > > changes to this: > > > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10, > > stratium 1 > > Well, that's by intention. The normalizer must know exactly which message it > is dealing with. This is even more important when we use it for > classification. So these two messages are definitely different, and I would > consider it very dangerous to automatically merge them into a single one > (which would not be a problem from a purely implementation PoV). The more > fuzzy the recognition is, the higher is the chance of false recognition, > something that would be really, really bad in the context of normalization. I understand. I merely meant that comment as a observation and wasn't criticizing it. Once I played a little bit, I actually thought, "oh, actually that make sense why it'd work like that!" :) > > > The 'normalizer' will call the ",stratium 1" part of the message as > > "unclassified". However, it doesn't appear that it'll grab the IP > > address, tag, etc. > > > > Also, I thing the "real work" is going to be writing rules. That's > > going to take some effort, in which I hope to assist with. > > Yes, that's definitely a lot of work -- and more than what a single person > can do. In order to make the normalizer really useful, we need a community > effort. If everyone contributes sample databases for their devices, we could > gain good results fast. But the key is getting enough momentum, so please > help spread the word! I already have been spreading the word. Strangely, I mentioned it to the OSSEC guys, and they didn't seem that interested in it. That might change over time. I've talked about it on the Sagan mailing list and people there seem to "get it". I'm tinkering with the idea about adding some liblognorm code into Sagan (probably today). A couple of things dawned on me. Many log lines that Sagan detects as "hostile" won't need normalization. Much of the information I need I already have. However, information from appliances like firewalls, routers, etc. will. So, I'll probably add a 'normalize' flag into my rule set. That way, I'm only attempting to normalize log lines when I know I need critical information from. This way, I don't waste CPU ticks on attempting to normalize log lines that don't need it. Another thing.... Much of the base information I'm already getting. For example, if I have this rule.... :%date:date-rfc3164% %host:word% %tag:char-to:\x3a%:synchronized to %ip:ipv4% Sagan already has the data, host, tag. Really, all I'd need is: :synchronized to %ip:ipv4% This is my dilemma. Do I make my own 'Sagan' liblognorm rules, which are a stripped down version of the liblognorm rules or do I just have Sagan rebuild the 'syslog string' so that the 'standard' liblognorm rules can be used? I like the idea of Sagan's own rules, but then that means keeping up with two liblognorm rule sets. I don't like that. Lastly, and this goes back to the beginning of this post. It's not a huge deal, but how are you going to handle dependencies with rsyslog? That is, in the end, will people have to download/compile/install libestr, libee, liblognorm, {insert other dependencies here for rsyslog}, then rsyslog? While that's not a huge deal for me, IMHO added dependencies 'turn off' users from using features. Even though the 'dependencies' in question are usually trivial to install, it adds yet another layer for end users to find issues with. Considering the libraries in question (ee/lognorm/estr) are pretty small (at least now!), would a one time package/build of something like 'liblognorm-complete' be possible. I know all this seems silly, but it's these little things IMHO that cause potential users from shifting away from software. I could be wrong, and maybe I'm over analyzing the issue. Oh.. on more thing. Do you think it's to early to start writing liblognorm rules? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From dave at fly.srk.fer.hr Thu Dec 2 16:23:35 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 2 Dec 2010 16:23:35 +0100 Subject: [rsyslog] Can't use PrivDrop with TCP listener In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD80C@GRFEXC.intern.adiscon.com> References: <20101202124731.GA4132@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD80C@GRFEXC.intern.adiscon.com> Message-ID: <20101202152335.GA11633@fly.srk.fer.hr> Is there a way to run it as an ordinary user with added CAP_NET_BIND_SERVICE capability on Linux? I tried to play with sucap and execcap, but couldn't figure out the right incantation. I tried something like: sucap bulb bulb execcap cap_net_bind_service=eip rsyslogd but I was getting: Caps: =ep cap_setpcap-ep sucaps: capsetp: Operation not permitted sucap: child did not exit cleanly. Caps: = Strace just showed that the capset syscall was failing with EPERM. I can't find any decent documentation for those commands. Rainer Gerhards wrote: > Known issue, the full story is here: > > http://wiki.rsyslog.com/index.php/Security > > > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > > Sent: Thursday, December 02, 2010 1:48 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] Can't use PrivDrop with TCP listener > > > > With rsyslog 5.6.2 I tried to use UDP and TCP listeners on port 514 in > > combination with privilege drop. Config file contains: > > > > $PrivDropToUser bulb > > $PrivDropToGroup bulb > > > > $Ruleset indata > > $RulesetCreateMainQueue on > > > > $InputTCPServerBindRuleset indata > > $InputTCPServerRun 514 > > $InputUDPServerBindRuleset indata > > $UDPServerRun 514 > > > > I get this error: > > > > rsyslogd: Could not create tcp listener, ignoring port 514. [try > > http://www.rsyslog.com/e/2077 ] > > > > strace shows: > > > > 28240 socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 4 > > 28240 setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 > > 28240 setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > > 28240 uname({sys="Linux", node="ssm2.nat.myrio.net", ...}) = 0 > > 28240 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR) > > 28240 fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > > 28240 bind(4, {sa_family=AF_INET6, sin6_port=htons(514), > > inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, > > sin6_scope_id=0}, 28) = 0 > > > > That was IPv6 UDP socket. > > > > 28240 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 5 > > 28240 setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > > 28240 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) > > 28240 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > > 28240 bind(5, {sa_family=AF_INET, sin_port=htons(514), > > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 > > > > That was IPv4 UDP socket. > > > > 28241 setuid(1004) = 0 > > 28241 futex(0x40bded1c, FUTEX_WAKE_PRIVATE, 1) = 0 > > 28241 futex(0x7fffc9f38600, FUTEX_WAKE_PRIVATE, 1 > > 28240 <... futex resumed> ) = 0 > > 28240 setuid(1004) = 0 > > > > That was privilege drop. > > > > 28243 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 9 > > 28243 setsockopt(9, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 > > 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > > 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) > > 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > > 28243 bind(9, {sa_family=AF_INET6, sin6_port=htons(514), > > inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, > > sin6_scope_id=0}, 28) = -1 EACCES (Permission denied) > > 28243 close(9) = 0 > > 28243 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 > > 28243 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 > > 28243 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) > > 28243 fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > > 28243 bind(9, {sa_family=AF_INET, sin_port=htons(514), > > sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied) > > > > Those were TCP sockets. Bind() fails because it happens after privilege > > drop. > > > > -- > > .-. .-. Yes, I am an agent of Satan, but my duties are largely > > (_ \ / _) ceremonial. > > | > > | dave at fly.srk.fer.hr > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Thu Dec 2 16:24:13 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 16:24:13 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> sorry, I need to be a bit brief as I have a conference call upcoming later today for which I need to prepare -- and I hope to fix some more thing. So the most important facts first, more later (maybe monday...). > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 4:00 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > On Thu, Dec 02, 2010 at 09:22:46AM +0100, Rainer Gerhards wrote: > > Thanks for the nice review and instructions :) > > > > I have begun to work heavily on a message modification module for > rsyslog > > which will support liblognorm-style normalization inside rsyslog. In > git > > there already is a branch "lognorm", which I will hopefully complete > and > > merge into master soon. It provides some *very* interesting shortcuts > of > > In the rsyslog git tree? I have a specific reason for asking, > which I'll get to later... http://git.adiscon.com/?p=rsyslog.git;a=shortlog;h=refs/heads/lognorm > > > pulling specific information out of syslog messages. I'll probably > promote it > > some more when it is available. IMHO it's the coolest and potentially > most > > valuable feature I have added in the past three years. Once I have > enabled > > tags in liblognorm/libee, you can even very easily classify log > messages > > based on their content. > > To be honest, it's not only going to be a valuable addition to > rsyslog, but my software as well... Not only that, I think once > people 'understand it's function', I could see many other projects > benefiting from it! > yup, that's why I made a lib and not solely a rsyslog component :) > > I did a couple of bug fixes yesterday. Frequently pull from git ;) > > Oh! I know it's a moving target! :) I'll git it frequently! > > > > So, that work nicely. Nifty. I made a few more 'complex' > > > rules, and those worked fine as well. > > > > I added a capability to generate graphs of the actual call tree. I > think this > > is *very* useful. An article on how to do that will be posted soon to > the web > > site (will make sure a notification goes to the list). > > Yes, please do! I'd seen mention in the source about this, > but didn't really dive into what you where trying to do there. http://www.liblognorm.com/help/creating-a-graph-of-the-sampledb/ > > > > However, if the rule is off a > > > bit, then you've got issue. Here's what I mean.. Back on my > example > > > above.. If this: > > > > > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > > > > > changes to this: > > > > > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10, > > > stratium 1 > > > > Well, that's by intention. The normalizer must know exactly which > message it > > is dealing with. This is even more important when we use it for > > classification. So these two messages are definitely different, and I > would > > consider it very dangerous to automatically merge them into a single > one > > (which would not be a problem from a purely implementation PoV). The > more > > fuzzy the recognition is, the higher is the chance of false > recognition, > > something that would be really, really bad in the context of > normalization. > > I understand. I merely meant that comment as a observation and > wasn't criticizing it. Once I played a little bit, I actually > thought, > "oh, actually that make sense why it'd work like that!" :) > > > > > The 'normalizer' will call the ",stratium 1" part of the message as > > > "unclassified". However, it doesn't appear that it'll grab the IP > > > address, tag, etc. > > > > > > Also, I thing the "real work" is going to be writing rules. > That's > > > going to take some effort, in which I hope to assist with. > > > > Yes, that's definitely a lot of work -- and more than what a single > person > > can do. In order to make the normalizer really useful, we need a > community > > effort. If everyone contributes sample databases for their devices, > we could > > gain good results fast. But the key is getting enough momentum, so > please > > help spread the word! > > I already have been spreading the word. Strangely, I mentioned > it to the OSSEC guys, and they didn't seem that interested in it. > That might change over time. I've talked about it on the Sagan mailing > list and people there seem to "get it". I think many people still think "we can normalize in any case, just let's use our regex approach". A very important point is speed. I will be very interested in practical rsyslog performance with that parser. But I think we can outperform most other normalizers by magnitudes. > > I'm tinkering with the idea about adding some liblognorm code > into Sagan (probably today). A couple of things dawned on me. Many > log > lines that Sagan detects as "hostile" won't need normalization. Much > of > the information I need I already have. However, information from > appliances like firewalls, routers, etc. will. So, I'll probably > add a 'normalize' flag into my rule set. That way, I'm only > attempting to normalize log lines when I know I need critical > information from. This way, I don't waste CPU ticks on attempting > to normalize log lines that don't need it. I am following a similar approach with the new mmnormalize in rsyslog. You call it when you need it. > > Another thing.... Much of the base information I'm already > getting. For example, if I have this rule.... > > :%date:date-rfc3164% %host:word% %tag:char-to:\x3a%:synchronized to > %ip:ipv4% > > Sagan already has the data, host, tag. Really, all I'd need is: > > :synchronized to %ip:ipv4% > > This is my dilemma. It is! And I am well aware of it. In rsyslog, I have the same issue. I think of something like a "common prefix" inside the sample db (maybe rulebase is a better name, btw :)). That would be common to all rules, and only the common prefix would need to be changed for different headers. It's not 100% sorted out, there is still enough work to do on the core engine (needs more parsers, parser priority, str optimizations). > Do I make my own 'Sagan' liblognorm rules, > which > are a stripped down version of the liblognorm rules or do I just have > Sagan rebuild the 'syslog string' so that the 'standard' liblognorm > rules can be used? I like the idea of Sagan's own rules, but then > that > means keeping up with two liblognorm rule sets. I don't like that. > > Lastly, and this goes back to the beginning of this post. It's > not a huge deal, but how are you going to handle dependencies with > rsyslog? That is, in the end, will people have to > download/compile/install libestr, libee, liblognorm, {insert other > dependencies here for rsyslog}, then rsyslog? While that's not a huge > deal for me, IMHO added dependencies 'turn off' users from using > features. Even though the 'dependencies' in question are usually > trivial to install, it adds yet another layer for end users to find > issues with. Considering the libraries in question (ee/lognorm/estr) > are pretty small (at least now!), would a one time package/build of > something like 'liblognorm-complete' be possible. I know all this > seems > silly, but it's these little things IMHO that cause potential users > from shifting away from software. I could be wrong, and maybe I'm > over analyzing the issue. rsyslog v6 will need libestr and libee in any case. There is also a volunteer who will package that. I try to keep dependencies as low as possible, but the alternative would be to copy the same code into different projects. I don't like that. I hope those folks who build packages will tie the right things together (and I am *very* grateful for their excellent work!). > Oh.. on more thing. Do you think it's to early to start > writing liblognorm rules? Depends... You will probably want to revisit the rules in a few weeks, when we have more capabilities. But on the other hand, I need some experience with building them, so that I know what does not work out. The current parsers are extremely limited and some (word, char-to) are very generic. But if that works, it will continue to work with new version. Wehn the classifier is there (hopefully december), you will probably want to add classification tags for easy filtering (if that matters for Sagan). Rainer > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From jason at jasonantman.com Thu Dec 2 16:44:04 2010 From: jason at jasonantman.com (Jason Antman) Date: Thu, 02 Dec 2010 10:44:04 -0500 Subject: [rsyslog] Feedback on ommysql requested In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> Message-ID: <4CF7BEC4.3020701@jasonantman.com> My thoughts: This should be optional. Personally I don't want any more complexity in ommysql, as I'm trying to use this for relatively important production purposes. Reasons: 1) I don't like configuration file dependencies. Especially since our DBA handles my.cnf and I handle rsyslog configuration, I think it's bad to have changes to an unrelated config file (my.cnf) effect another daemon (rsyslog). 2) The patch pre-configure options (TCP, cert validation, secure authentication) make no sense if MySQL is running on localhost. Honestly, I didn't even want a password for the MySQL user as needed in current ommysql. 3) Why is it assumed that something present in the global /etc/my.cnf is applicable to rsyslog? Just my two cents. -Jason Antman Rutgers University Rainer Gerhards wrote: > Hi all, > > I just received an interesting patch via the bug tracker: > > http://bugzilla.adiscon.com/show_bug.cgi?id=213 > > What I see looks good to me, but I am not really in into MySQL. Do you think > this is something I should merge? If not, why not? I'll wait until monday and > will merge if I have not heard any good argument against that. > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > From ktm at rice.edu Thu Dec 2 16:51:26 2010 From: ktm at rice.edu (Kenneth Marshall) Date: Thu, 2 Dec 2010 09:51:26 -0600 Subject: [rsyslog] Feedback on ommysql requested In-Reply-To: <4CF7BEC4.3020701@jasonantman.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD800@GRFEXC.intern.adiscon.com> <4CF7BEC4.3020701@jasonantman.com> Message-ID: <20101202155126.GZ19162@aart.is.rice.edu> Yes, optional -- please. It is a nice feature if you know it is in play. Otherwise you have mysterious problems with your DB connection that will require lsof and strace to diagnose. Cheer, Ken On Thu, Dec 02, 2010 at 10:44:04AM -0500, Jason Antman wrote: > My thoughts: > > This should be optional. Personally I don't want any more complexity in > ommysql, as I'm trying to use this for relatively important production > purposes. > > Reasons: > 1) I don't like configuration file dependencies. Especially since our > DBA handles my.cnf and I handle rsyslog configuration, I think it's bad > to have changes to an unrelated config file (my.cnf) effect another > daemon (rsyslog). > 2) The patch pre-configure options (TCP, cert validation, secure > authentication) make no sense if MySQL is running on localhost. > Honestly, I didn't even want a password for the MySQL user as needed in > current ommysql. > 3) Why is it assumed that something present in the global /etc/my.cnf is > applicable to rsyslog? > > Just my two cents. > -Jason Antman > Rutgers University > > Rainer Gerhards wrote: > > Hi all, > > > > I just received an interesting patch via the bug tracker: > > > > http://bugzilla.adiscon.com/show_bug.cgi?id=213 > > > > What I see looks good to me, but I am not really in into MySQL. Do you think > > this is something I should merge? If not, why not? I'll wait until monday and > > will merge if I have not heard any good argument against that. > > > > Thanks, > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From champ at softwink.com Thu Dec 2 17:09:31 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 11:09:31 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> Message-ID: <20101202160931.GB3620@bundy.vistech.net> On Thu, Dec 02, 2010 at 04:24:13PM +0100, Rainer Gerhards wrote: > sorry, I need to be a bit brief as I have a conference call upcoming later > today for which I need to prepare -- and I hope to fix some more thing. So > the most important facts first, more later (maybe monday...). No problem. > > I already have been spreading the word. Strangely, I mentioned > > it to the OSSEC guys, and they didn't seem that interested in it. > > That might change over time. I've talked about it on the Sagan mailing > > list and people there seem to "get it". > > I think many people still think "we can normalize in any case, just let's use > our regex approach". A very important point is speed. I will be very > interested in practical rsyslog performance with that parser. But I think we > can outperform most other normalizers by magnitudes. Agreed. Regex isn't a practical approach for these types of things. Speed is a must, and i agree. It'll probably out preform other normalizers by quite a bit. > > This is my dilemma. > > It is! And I am well aware of it. In rsyslog, I have the same issue. I think > of something like a "common prefix" inside the sample db (maybe rulebase is a > better name, btw :)). That would be common to all rules, and only the common > prefix would need to be changed for different headers. It's not 100% sorted > out, there is still enough work to do on the core engine (needs more parsers, > parser priority, str optimizations). That makes sense, if I understand correctly. Basically some way you can "tell" the library, Ie - "I only have the 'message' portion, so apply the rule base to it, but only using the 'message' portion of the rule"? That sort of thing? > rsyslog v6 will need libestr and libee in any case. There is also a volunteer > who will package that. I try to keep dependencies as low as possible, but the > alternative would be to copy the same code into different projects. I don't > like that. I hope those folks who build packages will tie the right things > together (and I am *very* grateful for their excellent work!). Okay. That'd be great. Like you, I'm not a fan of adding to many dependencies. > > > Oh.. on more thing. Do you think it's to early to start > > writing liblognorm rules? > > Depends... You will probably want to revisit the rules in a few weeks, when > we have more capabilities. But on the other hand, I need some experience with > building them, so that I know what does not work out. The current parsers are > extremely limited and some (word, char-to) are very generic. But if that > works, it will continue to work with new version. Wehn the classifier is > there (hopefully december), you will probably want to add classification tags > for easy filtering (if that matters for Sagan). Okay.. I understand. One more question, and this is more of a future support sort of thing. I'm only asking because I'm wondering if this was brought up with the CEE dictionary thing. You have things like %ip:ipv4% and %port:number% . Do you have any idea if there will eventually be something like a %ip:%ipv4:src% or %ip:ipv4:dst% type of flags (same idea applying to %port:number%)? This might be useful, for not only normalization, but XML and JSON output. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 2 17:15:02 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 17:15:02 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > It is! And I am well aware of it. In rsyslog, I have the same issue. > I think > > of something like a "common prefix" inside the sample db (maybe > rulebase is a > > better name, btw :)). That would be common to all rules, and only the > common > > prefix would need to be changed for different headers. It's not 100% > sorted > > out, there is still enough work to do on the core engine (needs more > parsers, > > parser priority, str optimizations). > > That makes sense, if I understand correctly. Basically some > way you can "tell" the library, Ie - "I only have the 'message' > portion, so apply the rule base to it, but only using the 'message' > portion of the rule"? That sort of thing? simpler: the rule base (I tend to switch to this term ;)) will have an extra entry, e.g. commonPrefix=<%PRI:PRI%>%date:date-rfc3164%... and rule=Port=%port%number%... and the process will combine the two while building the tree, like this: <%PRI:PRI%>%date:date-rfc3164%... rule=Port=%port%number%... And now that I wrote this, it's probably something to implement very soon, because it is pretty simple ;) Boils down to string concatenation. > > > Oh.. on more thing. Do you think it's to early to start > > > writing liblognorm rules? > > > > Depends... You will probably want to revisit the rules in a few > weeks, when > > we have more capabilities. But on the other hand, I need some > experience with > > building them, so that I know what does not work out. The current > parsers are > > extremely limited and some (word, char-to) are very generic. But if > that > > works, it will continue to work with new version. Wehn the classifier > is > > there (hopefully december), you will probably want to add > classification tags > > for easy filtering (if that matters for Sagan). > > Okay.. I understand. One more question, and this is more of a > future support sort of thing. I'm only asking because I'm wondering if > this was brought up with the CEE dictionary thing. You have things > like %ip:ipv4% and %port:number% . Do you have any idea if there will > eventually be something like a %ip:%ipv4:src% or %ip:ipv4:dst% type of > flags (same idea applying to %port:number%)? This might be useful, > for not only normalization, but XML and JSON output. Can you elaborate what you mean by %ip:ipv4:src%, I am not 100% sure I really understood... Rainer From champ at softwink.com Thu Dec 2 17:25:28 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 11:25:28 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> Message-ID: <20101202162528.GA5281@bundy.vistech.net> > > Okay.. I understand. One more question, and this is more of a > > future support sort of thing. I'm only asking because I'm wondering if > > this was brought up with the CEE dictionary thing. You have things > > like %ip:ipv4% and %port:number% . Do you have any idea if there will > > eventually be something like a %ip:%ipv4:src% or %ip:ipv4:dst% type of > > flags (same idea applying to %port:number%)? This might be useful, > > for not only normalization, but XML and JSON output. > > Can you elaborate what you mean by %ip:ipv4:src%, I am not 100% sure I really > understood... Here's what I mean.. From an example Cisco PIX log (message portion only). %PIX-7-710005: UDP request discarded from 192.168.20.45/53 to %outside:192.168.20.208/37989 We have a source IP, destination IP, source port, destination port. So an example rule might be: %PIX-7-710005: UDP request discarded from %ip:ipv4:src%/%port:number:src% to %outside:%ip:ipv4:dst%/%port:number:dst% Does this make sense? Or is it outside to scope of things? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 2 17:27:05 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 17:27:05 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <20101202162528.GA5281@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > Okay.. I understand. One more question, and this is more of a > > > future support sort of thing. I'm only asking because I'm > wondering if > > > this was brought up with the CEE dictionary thing. You have things > > > like %ip:ipv4% and %port:number% . Do you have any idea if there > will > > > eventually be something like a %ip:%ipv4:src% or %ip:ipv4:dst% type > of > > > flags (same idea applying to %port:number%)? This might be > useful, > > > for not only normalization, but XML and JSON output. > > > > Can you elaborate what you mean by %ip:ipv4:src%, I am not 100% sure > I really > > understood... > > Here's what I mean.. From an example Cisco PIX log (message > portion only). > > %PIX-7-710005: UDP request discarded from 192.168.20.45/53 to > %outside:192.168.20.208/37989 > > We have a source IP, destination IP, source port, destination > port. So an example rule might be: > > %PIX-7-710005: UDP request discarded from > %ip:ipv4:src%/%port:number:src% to > %outside:%ip:ipv4:dst%/%port:number:dst% That would be %src-ip:ipv4%/%src-port:number% to %outside:%dst-ip:ipv4%/%dst-port:number% The idea is not that it is an IP field, but the field (name) should be quite specific. Rainer > > Does this make sense? Or is it outside to scope of things? > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Thu Dec 2 17:32:33 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 11:32:33 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <20101202162528.GA5281@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com> Message-ID: <20101202163233.GA5475@bundy.vistech.net> > That would be > > %src-ip:ipv4%/%src-port:number% to %outside:%dst-ip:ipv4%/%dst-port:number% > > The idea is not that it is an IP field, but the field (name) should be quite > specific. Oh wait... So in the %src-ip:ipv4% , the "src-ip" portion is just what the rule base "names" the field? Ie - it could easily be: %nifty:%ipv4% .... now "nifty" is the "field name" for the "ip address"? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 2 17:33:27 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 17:33:27 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com><20101202162528.GA5281@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com> <20101202163233.GA5475@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD81D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 5:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > That would be > > > > %src-ip:ipv4%/%src-port:number% to %outside:%dst-ip:ipv4%/%dst- > port:number% > > > > The idea is not that it is an IP field, but the field (name) should > be quite > > specific. > > Oh wait... So in the %src-ip:ipv4% , the "src-ip" portion is > just what the rule base "names" the field? Ie - it could easily be: > > %nifty:%ipv4% .... now "nifty" is the "field name" for the "ip > address"? jup > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Thu Dec 2 17:36:15 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 11:36:15 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD81D@GRFEXC.intern.adiscon.com> References: <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <20101202162528.GA5281@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com> <20101202163233.GA5475@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81D@GRFEXC.intern.adiscon.com> Message-ID: <20101202163615.GB5475@bundy.vistech.net> > > Oh wait... So in the %src-ip:ipv4% , the "src-ip" portion is > > just what the rule base "names" the field? Ie - it could easily be: > > > > %nifty:%ipv4% .... now "nifty" is the "field name" for the "ip > > address"? > > jup Ah! Brilliant! In the future, you'll probably want to encourage some standards.. or you'll end up with %ip-source%ipv4% and %ip-src%ipv4% ... :) However, great job.. love it.. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 2 17:37:18 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 17:37:18 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com><20101202162528.GA5281@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com><20101202163233.GA5475@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81D@GRFEXC.intern.adiscon.com> <20101202163615.GB5475@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD81E@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 5:36 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > Oh wait... So in the %src-ip:ipv4% , the "src-ip" portion is > > > just what the rule base "names" the field? Ie - it could easily > be: > > > > > > %nifty:%ipv4% .... now "nifty" is the "field name" for the "ip > > > address"? > > > > jup > > Ah! Brilliant! In the future, you'll probably want to > encourage some standards.. or you'll end up with %ip-source%ipv4% and > %ip-src%ipv4% ... :) That will be the CEE dictionary :) Rainer > > However, great job.. love it.. > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From friedl at hq.adiscon.com Thu Dec 2 17:42:38 2010 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 2 Dec 2010 17:42:38 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com><20101202162528.GA5281@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81B@GRFEXC.intern.adiscon.com><20101202163233.GA5475@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81D@GRFEXC.intern.adiscon.com> <20101202163615.GB5475@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD81F@GRFEXC.intern.adiscon.com> > > > > Oh wait... So in the %src-ip:ipv4% , the "src-ip" portion is > just > > > what the rule base "names" the field? Ie - it could easily be: > > > > > > %nifty:%ipv4% .... now "nifty" is the "field name" for the "ip > > > address"? > > > > jup > > Ah! Brilliant! In the future, you'll probably want to > encourage some standards.. or you'll end up with %ip-source%ipv4% and > %ip-src%ipv4% ... :) > > However, great job.. love it.. > I changed the text for field names in this article a bit. I hope it now describes better how it is meant: http://www.liblognorm.com/help/creating-a-sampledb/ Florian > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If > it wasn't for C, we'd be using BASI, PASAL and OBOL. From rgerhards at hq.adiscon.com Thu Dec 2 18:27:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 18:27:29 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, December 02, 2010 5:15 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > > Sent: Thursday, December 02, 2010 5:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > > It is! And I am well aware of it. In rsyslog, I have the same > issue. > > I think > > > of something like a "common prefix" inside the sample db (maybe > > rulebase is a > > > better name, btw :)). That would be common to all rules, and only > the > > common > > > prefix would need to be changed for different headers. It's not > 100% > > sorted > > > out, there is still enough work to do on the core engine (needs > more > > parsers, > > > parser priority, str optimizations). > > > > That makes sense, if I understand correctly. Basically some > > way you can "tell" the library, Ie - "I only have the 'message' > > portion, so apply the rule base to it, but only using the 'message' > > portion of the rule"? That sort of thing? > > simpler: the rule base (I tend to switch to this term ;)) will have an > extra > entry, e.g. > > commonPrefix=<%PRI:PRI%>%date:date-rfc3164%... > and > rule=Port=%port%number%... > > and the process will combine the two while building the tree, like > this: > <%PRI:PRI%>%date:date-rfc3164%... rule=Port=%port%number%... > > And now that I wrote this, it's probably something to implement very > soon, > because it is pretty simple ;) Boils down to string concatenation. Said and done - I pushed the rsyslog work away, as this is more useful. You can now pull from git, the rule base (sample db) format has changed. You can now specify a common prefix, as I said. Sample: prefix=:%date:date-rfc3164% %host:word% %seqnum:number%: %othseq:char-to:\x3a%: %%%tag:char-to:\x3a%: rule=: Configured from console by %tty:word:% (%ip:ipv4%) rule=: Authentication failure for %proto:word% req from host %ip:ipv4% rule=: Interface %interface:char-to:,%, changed state to %state:word% rule=: Line protocol on Interface %interface:char-to:,%, changed state to %state:word% rule=: Attempted to connect to %servname:word% from %ip:ipv4% This is also much easier to read. Note that the first space is part of the sample. I did this to keep consistent with how rsyslog treats things in regard to RFC3164. But you could also move it to the common prefix. I'll probably add also an "prefixextend" command so that a single ruleset could also handle that. Rainer From bgs at bgs.hu Thu Dec 2 18:22:24 2010 From: bgs at bgs.hu (Bgs) Date: Thu, 02 Dec 2010 18:22:24 +0100 Subject: [rsyslog] Disappearing buffered remote logs on reboot/boot Message-ID: <4CF7D5D0.3090600@bgs.hu> Hi all, I ran into a strange problem. My setup: Ubuntu, rsyslog with remote tcp logging to central syslog server. Connection is periodic, so I need the buffer to file ability of rsyslog. Ubuntu 10.04LTS, rsyslog 4.2.0-2ubuntu8, using imuxsock module for remote logging. Config: queue type = linkedlist workdirectory/filename/maxdiskpace set saveonshutdown on retrycount -1 What I would expect when there is no connection at reboot is: save in memory buffer to file (create if there was nothing there yet), at boot open it and send it in at the first opportunity. What actually happens: A portion of the buffer is saved to file (checked by booting from outside and looking at the rsyslog spool) and when the system boots up, the file disappears without any content showing up on the remote side. After the last message before connection break, the next one that gets through are the boot logs. During normal operation, connection breaks are handled without any problem. What might the problem be? Configuration problem? Ubuntu upstart related? rsyslog bug? Regards Bgs From rgerhards at hq.adiscon.com Thu Dec 2 18:56:08 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 18:56:08 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com> it looks like I screwed up in this commit... I must have "cleaned up" something that was useful ;) Will check... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, December 02, 2010 6:27 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Thursday, December 02, 2010 5:15 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > > > Sent: Thursday, December 02, 2010 5:10 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > > > > It is! And I am well aware of it. In rsyslog, I have the same > > issue. > > > I think > > > > of something like a "common prefix" inside the sample db (maybe > > > rulebase is a > > > > better name, btw :)). That would be common to all rules, and only > > the > > > common > > > > prefix would need to be changed for different headers. It's not > > 100% > > > sorted > > > > out, there is still enough work to do on the core engine (needs > > more > > > parsers, > > > > parser priority, str optimizations). > > > > > > That makes sense, if I understand correctly. Basically some > > > way you can "tell" the library, Ie - "I only have the 'message' > > > portion, so apply the rule base to it, but only using the > 'message' > > > portion of the rule"? That sort of thing? > > > > simpler: the rule base (I tend to switch to this term ;)) will have > an > > extra > > entry, e.g. > > > > commonPrefix=<%PRI:PRI%>%date:date-rfc3164%... > > and > > rule=Port=%port%number%... > > > > and the process will combine the two while building the tree, like > > this: > > <%PRI:PRI%>%date:date-rfc3164%... rule=Port=%port%number%... > > > > And now that I wrote this, it's probably something to implement very > > soon, > > because it is pretty simple ;) Boils down to string concatenation. > > Said and done - I pushed the rsyslog work away, as this is more useful. > You > can now pull from git, the rule base (sample db) format has changed. > You can > now specify a common prefix, as I said. Sample: > > prefix=:%date:date-rfc3164% %host:word% %seqnum:number%: > %othseq:char-to:\x3a%: %%%tag:char-to:\x3a%: > rule=: Configured from console by %tty:word:% (%ip:ipv4%) > rule=: Authentication failure for %proto:word% req from host %ip:ipv4% > rule=: Interface %interface:char-to:,%, changed state to %state:word% > rule=: Line protocol on Interface %interface:char-to:,%, changed state > to > %state:word% > rule=: Attempted to connect to %servname:word% from %ip:ipv4% > > This is also much easier to read. Note that the first space is part of > the > sample. I did this to keep consistent with how rsyslog treats things in > regard to RFC3164. But you could also move it to the common prefix. > I'll > probably add also an "prefixextend" command so that a single ruleset > could > also handle that. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From champ at softwink.com Thu Dec 2 19:02:12 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 13:02:12 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com> References: <20101020175951.GA13070@bundy.vistech.net> <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com> Message-ID: <20101202180212.GA8868@bundy.vistech.net> > it looks like I screwed up in this commit... I must have "cleaned up" > something that was useful ;) Will check... Did you update the normalizer (example) as well? Or is that part of this 'cleaned up' processes? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From david at lang.hm Thu Dec 2 19:03:41 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 2 Dec 2010 10:03:41 -0800 (PST) Subject: [rsyslog] Execute Alert script In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD803@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD803@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 2 Dec 2010, Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, December 01, 2010 10:14 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Execute Alert script >> >> On Wed, 1 Dec 2010, Mark Olliver wrote: >> >>> Hi, >>> >>> I am new to rsyslog but have found it very capable. I am using it via >> logger >>> to log output of some of my scripts to local5. However I do want >> rsyslog to >>> alert me via an alert script should it encounter a warning or error >> message >>> sent to the log. >>> >>> I have tested my alert script and that works ok, i give it two >> arguments >>> subject and message. However I can not seem to work out how to make >> this >>> work from rsyslog. >>> >>> I tried the following: >>> >>> $template Alert,"'Rsyslog Alert' :msg" >>> local5.info >>> :msg, content "warning" >>> ^/usr/bin/alert_script Alert >>> >>> >>> Ideally i want the Subject to be what i set it to, the message to be >>> everything that comes after the word warning. I would also only like >> one >>> alert every 24 hours. >>> >>> Any ideas or help would be gratefully received. >> >> rsyslog doesn't keep a history of what it's done, so it can't limit you >> to >> one alert every 24 hours, you really want another tool (something like >> SEC) for that. > > It's actually possible: $ActionExecOnlyOnceEveryInterval allows you do this > kind of thing. This is the only sense of history rsyslog has, but there are > many uses for it. I think I introduced it when I wrote ommail, which really > needs such a capability. As such, a sample of its use can be found in the > ommail doc: > > http://www.rsyslog.com/doc/ommail.html I forgot about that, although I'll point out that that limits your reports to one per filter rule, while useing something like SEC can do the same type of thing, but can also do things like doing the rate limiting on the message that is being sent. David Lang > Rainer > >> >> when you are creating templates, you put the properties you want >> between % >> characters (so instead of :msg, do %msg%) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Dec 2 19:08:25 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 19:08:25 +0100 Subject: [rsyslog] Execute Alert script References: <9B6E2A8877C38245BFB15CC491A11DA71DD803@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD823@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, December 02, 2010 7:04 PM > To: rsyslog-users > Subject: Re: [rsyslog] Execute Alert script > > On Thu, 2 Dec 2010, Rainer Gerhards wrote: > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of david at lang.hm > >> Sent: Wednesday, December 01, 2010 10:14 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Execute Alert script > >> > >> On Wed, 1 Dec 2010, Mark Olliver wrote: > >> > >>> Hi, > >>> > >>> I am new to rsyslog but have found it very capable. I am using it > via > >> logger > >>> to log output of some of my scripts to local5. However I do want > >> rsyslog to > >>> alert me via an alert script should it encounter a warning or error > >> message > >>> sent to the log. > >>> > >>> I have tested my alert script and that works ok, i give it two > >> arguments > >>> subject and message. However I can not seem to work out how to make > >> this > >>> work from rsyslog. > >>> > >>> I tried the following: > >>> > >>> $template Alert,"'Rsyslog Alert' :msg" > >>> local5.info > >>> :msg, content "warning" > >>> ^/usr/bin/alert_script Alert > >>> > >>> > >>> Ideally i want the Subject to be what i set it to, the message to > be > >>> everything that comes after the word warning. I would also only > like > >> one > >>> alert every 24 hours. > >>> > >>> Any ideas or help would be gratefully received. > >> > >> rsyslog doesn't keep a history of what it's done, so it can't limit > you > >> to > >> one alert every 24 hours, you really want another tool (something > like > >> SEC) for that. > > > > It's actually possible: $ActionExecOnlyOnceEveryInterval allows you > do this > > kind of thing. This is the only sense of history rsyslog has, but > there are > > many uses for it. I think I introduced it when I wrote ommail, which > really > > needs such a capability. As such, a sample of its use can be found in > the > > ommail doc: > > > > http://www.rsyslog.com/doc/ommail.html > > I forgot about that, although I'll point out that that limits your > reports > to one per filter rule, while useing something like SEC can do the same > type of thing, but can also do things like doing the rate limiting on > the > message that is being sent. sure, it's pretty limited. Rainer From rgerhards at hq.adiscon.com Thu Dec 2 19:10:23 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 19:10:23 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101020175951.GA13070@bundy.vistech.net><20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com> <20101202180212.GA8868@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD824@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 7:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > it looks like I screwed up in this commit... I must have "cleaned up" > > something that was useful ;) Will check... > > Did you update the normalizer (example) as well? Or is that > part of this 'cleaned up' processes? I just committed a fixed version, which also include the "extendprefix" functionality. All samples are also updated (both of them ;)). Rainer From champ at softwink.com Thu Dec 2 19:22:34 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 2 Dec 2010 13:22:34 -0500 Subject: [rsyslog] Fun with liblognorm / rsyslog In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD824@GRFEXC.intern.adiscon.com> References: <20101201203844.GA11736@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com> <20101202150001.GA30820@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com> <20101202160931.GB3620@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com> <20101202180212.GA8868@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD824@GRFEXC.intern.adiscon.com> Message-ID: <20101202182234.GA9751@bundy.vistech.net> > > Did you update the normalizer (example) as well? Or is that > > part of this 'cleaned up' processes? > > I just committed a fixed version, which also include the "extendprefix" > functionality. All samples are also updated (both of them ;)). Two things.. autoreconf -vfi complains about the m4 directory not being there. No biggie, mkdir m4. Also (for the list) "sample-db" is now "rulebase" :) I ran a small test with "normalize" against the cisco.rulebase. Didn't work. I'm assuming the example "normalizer" isn't up to date with the rule change? Sorry for the constant e-mails. Just somewhat excited to play with this! -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 2 19:33:39 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 2 Dec 2010 19:33:39 +0100 Subject: [rsyslog] Fun with liblognorm / rsyslog References: <20101201203844.GA11736@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD807@GRFEXC.intern.adiscon.com><20101202150001.GA30820@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD815@GRFEXC.intern.adiscon.com><20101202160931.GB3620@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD81A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD820@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA71DD821@GRFEXC.intern.adiscon.com><20101202180212.GA8868@bundy.vistech.net><9B6E2A8877C38245BFB15CC491A11DA71DD824@GRFEXC.intern.adiscon.com> <20101202182234.GA9751@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD825@GRFEXC.intern.adiscon.com> yet another typo, now fixed (it's getting late) ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Thursday, December 02, 2010 7:23 PM > To: rsyslog-users > Subject: Re: [rsyslog] Fun with liblognorm / rsyslog > > > > Did you update the normalizer (example) as well? Or is that > > > part of this 'cleaned up' processes? > > > > I just committed a fixed version, which also include the > "extendprefix" > > functionality. All samples are also updated (both of them ;)). > > Two things.. autoreconf -vfi complains about the m4 directory > not being there. No biggie, mkdir m4. Also (for the list) > "sample-db" is now "rulebase" :) > > I ran a small test with "normalize" against the cisco.rulebase. > Didn't work. I'm assuming the example "normalizer" isn't up to date > with the rule change? > > Sorry for the constant e-mails. Just somewhat excited to play > with this! > > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From bgs at bgs.hu Fri Dec 3 21:21:44 2010 From: bgs at bgs.hu (Bgs) Date: Fri, 03 Dec 2010 21:21:44 +0100 Subject: [rsyslog] Disappearing buffered remote logs on reboot/boot In-Reply-To: <4CF7D5D0.3090600@bgs.hu> References: <4CF7D5D0.3090600@bgs.hu> Message-ID: <4CF95158.8070804@bgs.hu> New info from tests: I switched to disk from linked list. This solved the logs lost on rsyslogd shutdown. These are normal clients so the performance impact is negligible. I'd spare unneeded IO, but I can live with it. I also set the retry timer to 10 seconds which seems to bring efficiency up a lot. After reboot, logs are not all lost. But there still are some. It looks like the first portion of the logs (written into the spool file but probably also still in memory) waiting to be sent is handled as ok, while other logs in the spool file are handled correctly as unsent and sent in after boot. For example I did the following test: I pulled the plug to simulate a network outage. Made 1500 log entries using logger and numbered them. All entries appeared in the local 'messages' and also in to spool file. I rebooted the system which came up with network connection. I got a bunch of logs on the server, starting with number 756 and all test entries up to number 1500. The first 755 entries were lost. So the questions that remain: 1) Are the lost entries related to tcp socket buffers or log entry tagging? 2) If the later, is there a way to keep the unsent messages with the appropriate status? 3) If it's socket related, can anything be done about it? Is this the described loss possibility for tcp syslog? 4) Would RELP solve this problem? (Reliably handling the entries). 5) Is there a way in linkedlist mode to tell rsyslogd to write out all buffered data to the spool? With file logs this can beg done with a HUP, but that does not force rsyslogd to write out the queue too... Regards Bgs On 12/02/2010 06:22 PM, Bgs wrote: > > Hi all, > > I ran into a strange problem. My setup: > > Ubuntu, rsyslog with remote tcp logging to central syslog server. > Connection is periodic, so I need the buffer to file ability of rsyslog. > Ubuntu 10.04LTS, rsyslog 4.2.0-2ubuntu8, using imuxsock module for > remote logging. > Config: > queue type = linkedlist > workdirectory/filename/maxdiskpace set > saveonshutdown on > retrycount -1 > > What I would expect when there is no connection at reboot is: save in > memory buffer to file (create if there was nothing there yet), at boot > open it and send it in at the first opportunity. > > What actually happens: A portion of the buffer is saved to file > (checked by booting from outside and looking at the rsyslog spool) and > when the system boots up, the file disappears without any content > showing up on the remote side. After the last message before > connection break, the next one that gets through are the boot logs. > > During normal operation, connection breaks are handled without any > problem. > > What might the problem be? Configuration problem? Ubuntu upstart > related? rsyslog bug? > > Regards > Bgs > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Dec 3 22:00:39 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 3 Dec 2010 13:00:39 -0800 (PST) Subject: [rsyslog] Disappearing buffered remote logs on reboot/boot In-Reply-To: <4CF95158.8070804@bgs.hu> References: <4CF7D5D0.3090600@bgs.hu> <4CF95158.8070804@bgs.hu> Message-ID: My understanding is that rsyslog attempts to save all the data to disk when it shuts down, but it also has a timeout to keep it from sitting forever if it is having trouble doing so. you may want to try setting this timeout longer. you may also need to do the disk-assist version of the queue rather than one of the pure memory queues. this is an aread I haven't worked with much. RELP would help get the logs to the queue, but doesn't have any effect on the shutdown behavior of rsyslog. David Lang On Fri, 3 Dec 2010, Bgs wrote: > New info from tests: > > I switched to disk from linked list. This solved the logs lost on rsyslogd > shutdown. These are normal clients so the performance impact is negligible. > I'd spare unneeded IO, but I can live with it. I also set the retry timer to > 10 seconds which seems to bring efficiency up a lot. After reboot, logs are > not all lost. But there still are some. It looks like the first portion of > the logs (written into the spool file but probably also still in memory) > waiting to be sent is handled as ok, while other logs in the spool file are > handled correctly as unsent and sent in after boot. > > For example I did the following test: I pulled the plug to simulate a network > outage. Made 1500 log entries using logger and numbered them. All entries > appeared in the local 'messages' and also in to spool file. I rebooted the > system which came up with network connection. I got a bunch of logs on the > server, starting with number 756 and all test entries up to number 1500. The > first 755 entries were lost. > > So the questions that remain: > > 1) Are the lost entries related to tcp socket buffers or log entry tagging? > 2) If the later, is there a way to keep the unsent messages with the > appropriate status? > 3) If it's socket related, can anything be done about it? Is this the > described loss possibility for tcp syslog? > 4) Would RELP solve this problem? (Reliably handling the entries). > 5) Is there a way in linkedlist mode to tell rsyslogd to write out all > buffered data to the spool? With file logs this can beg done with a HUP, but > that does not force rsyslogd to write out the queue too... > > Regards > Bgs > > > On 12/02/2010 06:22 PM, Bgs wrote: >> >> Hi all, >> >> I ran into a strange problem. My setup: >> >> Ubuntu, rsyslog with remote tcp logging to central syslog server. >> Connection is periodic, so I need the buffer to file ability of rsyslog. >> Ubuntu 10.04LTS, rsyslog 4.2.0-2ubuntu8, using imuxsock module for remote >> logging. >> Config: >> queue type = linkedlist >> workdirectory/filename/maxdiskpace set >> saveonshutdown on >> retrycount -1 >> >> What I would expect when there is no connection at reboot is: save in >> memory buffer to file (create if there was nothing there yet), at boot open >> it and send it in at the first opportunity. >> >> What actually happens: A portion of the buffer is saved to file (checked by >> booting from outside and looking at the rsyslog spool) and when the system >> boots up, the file disappears without any content showing up on the remote >> side. After the last message before connection break, the next one that >> gets through are the boot logs. >> >> During normal operation, connection breaks are handled without any problem. >> >> What might the problem be? Configuration problem? Ubuntu upstart related? >> rsyslog bug? >> >> Regards >> Bgs >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From friedl at hq.adiscon.com Mon Dec 6 12:49:37 2010 From: friedl at hq.adiscon.com (Florian Riedl) Date: Mon, 6 Dec 2010 12:49:37 +0100 Subject: [rsyslog] CERN uses rsyslog Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD852@GRFEXC.intern.adiscon.com> Hi all, we just found out, that the CERN in Suisse uses rsyslog in their log environment for the LHCb. Quite some good news one may think. Of course it is. http://www.rsyslog.com/cern-uses-rsyslog/ Best regards, Florian Riedl From pierre.blondeau at info.unicaen.fr Wed Dec 8 10:31:19 2010 From: pierre.blondeau at info.unicaen.fr (Pierre BLONDEAU) Date: Wed, 08 Dec 2010 10:31:19 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: <4CEC2D32.9070802@feurix.com> References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com><9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com><9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> Message-ID: <4CFF5067.8070002@info.unicaen.fr> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I have a similar problem in the practice but not the same architecture. I use the debian backport version of rsyslog ( 4.6.4 ). I use one instance of rsyslog who listen only whit gtls on x509/certvalid mode. At most, I have about 200 machines connected. I don't use relp but my rsyslog server send au informations in one mysql server. My server is not very powerful, but it does not seem to be saturated : one Intel(R) Xeon(TM) CPU 2.40GHz 2 cors 2 GB of memory I have rsyslog and mysql on the same server. After several hours of running (2 or 3), the process rsyslog is 100% CPU. The machines are connected, but they no longer seem to send the log because they create temporary queues that I configured. When I make a 'strace -fp ', I see many : [pid 6320] accept(1, 0xb6a53114, [128]) = -1 EAGAIN (Resource temporarily unavailable) [pid 6320] accept(2, 0xb6a53114, [128]) = -1 EAGAIN (Resource temporarily unavailable) [pid 6320] accept(1, 0xb6a53114, [128]) = -1 EAGAIN (Resource temporarily unavailable) [pid 6320] accept(2, 0xb6a53114, [128]) = -1 EAGAIN (Resource temporarily unavailable) Note that the call system is not the same as in the first post I not sure it's the same problem, but symptoms appear identical. Thanks, Pierre - -- - ---------------------------------------------- Pierre BLONDEAU Administrateur Syst?mes & r?seaux Universit? de Caen Laboratoire GREYC, D?partement d'informatique tel : 02 31 56 73 38 bureau : Campus 2, Science 3, 406 - ---------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz/UGcACgkQru2OSmFgnO2VYgCbBVsaJ/5jmCQ5KgKjiaabbbRB 5n4AoMw3AOht7z3jU3yW/c+J/ycgsc02 =QVDB -----END PGP SIGNATURE----- From rgerhards at hq.adiscon.com Wed Dec 8 12:15:31 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 8 Dec 2010 12:15:31 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com><9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com><9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com><4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD878@GRFEXC.intern.adiscon.com> Known - and fixed - issue: http://bugzilla.adiscon.com/show_bug.cgi?id=194 Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Pierre BLONDEAU > Sent: Wednesday, December 08, 2010 10:31 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] One thread eating up infinite CPU time > (spinlock?) > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hello, > > I have a similar problem in the practice but not the same architecture. > > I use the debian backport version of rsyslog ( 4.6.4 ). > > I use one instance of rsyslog who listen only whit gtls on > x509/certvalid mode. At most, I have about 200 machines connected. I > don't use relp but my rsyslog server send au informations in one mysql > server. > > My server is not very powerful, but it does not seem to be saturated : > one Intel(R) Xeon(TM) CPU 2.40GHz 2 cors > 2 GB of memory > > I have rsyslog and mysql on the same server. > > After several hours of running (2 or 3), the process rsyslog is 100% > CPU. The machines are connected, but they no longer seem to send the > log > because they create temporary queues that I configured. > > When I make a 'strace -fp ', I see many : > [pid 6320] accept(1, 0xb6a53114, [128]) = -1 EAGAIN (Resource > temporarily unavailable) > [pid 6320] accept(2, 0xb6a53114, [128]) = -1 EAGAIN (Resource > temporarily unavailable) > [pid 6320] accept(1, 0xb6a53114, [128]) = -1 EAGAIN (Resource > temporarily unavailable) > [pid 6320] accept(2, 0xb6a53114, [128]) = -1 EAGAIN (Resource > temporarily unavailable) > > Note that the call system is not the same as in the first post > > I not sure it's the same problem, but symptoms appear identical. > > Thanks, > Pierre > > > - -- > - ---------------------------------------------- > Pierre BLONDEAU > Administrateur Syst?mes & r?seaux > Universit? de Caen > Laboratoire GREYC, D?partement d'informatique > > tel : 02 31 56 73 38 > bureau : Campus 2, Science 3, 406 > - ---------------------------------------------- > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkz/UGcACgkQru2OSmFgnO2VYgCbBVsaJ/5jmCQ5KgKjiaabbbRB > 5n4AoMw3AOht7z3jU3yW/c+J/ycgsc02 > =QVDB > -----END PGP SIGNATURE----- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbiebl at gmail.com Wed Dec 8 16:35:04 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 8 Dec 2010 16:35:04 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: <4CFF5067.8070002@info.unicaen.fr> References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> Message-ID: 2010/12/8 Pierre BLONDEAU : > I use the debian backport version of rsyslog ( 4.6.4 ). The fix has been applied to 4.6.4-2, which currently is in unstable and will transition to squeeze/testing in a few days. I'll then prepare updated backports for lenny. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From pierre.blondeau at info.unicaen.fr Wed Dec 8 21:38:56 2010 From: pierre.blondeau at info.unicaen.fr (Pierre BLONDEAU) Date: Wed, 08 Dec 2010 21:38:56 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> Message-ID: <4CFFECE0.20001@info.unicaen.fr> Hi, Excuse me, I did not find this bug. Thank you for updating lenny backport. Tell me when the backport will to date. Thank you again. Regards, Pierre -- ---------------------------------------- Pierre BLONDEAU Administrateur Syst?mes & r?seaux D?partement d'informatique Universit? de Caen tel : 02 31 56 73 38 bureau : Campus 2, Science 3, 406 ---------------------------------------- From mbiebl at gmail.com Wed Dec 8 22:06:14 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 8 Dec 2010 22:06:14 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: <4CFFECE0.20001@info.unicaen.fr> References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> <4CFFECE0.20001@info.unicaen.fr> Message-ID: 2010/12/8 Pierre BLONDEAU : > Hi, > > Excuse me, I did not find this bug. > Thank you for updating lenny backport. > Tell me when the backport will to date. If you have installed the rsyslog backport following the instructions at [1], i.e. adding === deb http://backports.debian.org/debian-backports lenny-backports main === to /etc/apt/sources.list and ==== Package: * Pin: release a=lenny-backports Pin-Priority: 200 ==== to /etc/apt/preferences (you will probably need to create that file), then rsyslog will automatically be updated as soon as it is available from the (backports) archive. Cheers, Michael [1] http://backports.debian.org/ -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From pierre.blondeau at info.unicaen.fr Wed Dec 8 22:15:51 2010 From: pierre.blondeau at info.unicaen.fr (Pierre BLONDEAU) Date: Wed, 08 Dec 2010 22:15:51 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> <4CFFECE0.20001@info.unicaen.fr> Message-ID: <4CFFF587.6030708@info.unicaen.fr> Le 08/12/2010 22:06, Michael Biebl a ?crit : > 2010/12/8 Pierre BLONDEAU : >> Hi, >> >> Excuse me, I did not find this bug. >> Thank you for updating lenny backport. >> Tell me when the backport will to date. > > If you have installed the rsyslog backport following the instructions > at [1], i.e. > adding > === > deb http://backports.debian.org/debian-backports lenny-backports main > === > to /etc/apt/sources.list > and > > ==== > Package: * > Pin: release a=lenny-backports > Pin-Priority: 200 > ==== > to /etc/apt/preferences (you will probably need to create that file), > > then rsyslog will automatically be updated as soon as it is available > from the (backports) archive. > > Cheers, > Michael > > > > [1] http://backports.debian.org/ Hi, Yes, i know, but actualy the version of rsyslog in backport is : 4.6.4-1 and i have a debian lenny. I'm just being misspoken. My request was more: you have an idea of the date of update rsyslog in backport ? Regards, Pierre -- ---------------------------------------- Pierre BLONDEAU Administrateur Syst?mes & r?seaux D?partement d'informatique Universit? de Caen tel : 02 31 56 73 38 bureau : Campus 2, Science 3, 406 ---------------------------------------- From mbiebl at gmail.com Wed Dec 8 22:19:29 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 8 Dec 2010 22:19:29 +0100 Subject: [rsyslog] One thread eating up infinite CPU time (spinlock?) In-Reply-To: <4CFFF587.6030708@info.unicaen.fr> References: <4CD8063A.40004@feurix.com> <4CD88856.6000705@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6BF@GRFEXC.intern.adiscon.com> <4CD98DBB.1020808@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C3@GRFEXC.intern.adiscon.com> <4CD9CC79.9070000@feurix.com> <9B6E2A8877C38245BFB15CC491A11DA71DD6C4@GRFEXC.intern.adiscon.com> <4CEC2A58.6030000@feurix.com> <4CEC2D32.9070802@feurix.com> <4CFF5067.8070002@info.unicaen.fr> <4CFFECE0.20001@info.unicaen.fr> <4CFFF587.6030708@info.unicaen.fr> Message-ID: 2010/12/8 Pierre BLONDEAU : > > I'm just being misspoken. > My request was more: you have an idea of the date of update rsyslog in > backport ? As already said, the current version 4.6.4-2 from unstable will need to migrate to testing first. This will happen automatically in 3 days. After that I'll prepare the backport (versioned 4.6.4-2~bpo50+1). So expect this to happen in about 3-4 days. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Thu Dec 9 11:38:58 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 9 Dec 2010 02:38:58 -0800 (PST) Subject: [rsyslog] UDP packet loss Message-ID: quote from Rainer on twitter: worked all day on performance test lab for rsyslog -- setting the lab up always is real pain... but already got some interesting results... Ok, it's been almost two days since you sent out this teaser, what are you seeing? David Lang From bernard.fay at gmail.com Thu Dec 9 14:43:16 2010 From: bernard.fay at gmail.com (Bernard Fay) Date: Thu, 9 Dec 2010 08:43:16 -0500 Subject: [rsyslog] variables in filters Message-ID: Hi, Is-it possible to use variable in rsyslog? We have a rsyslog server that collects events from windows servers. Some events from windows are about tmp files that we don't need in our rsyslog server. For one tmp file there is many events but the name of the file is not shown in all events. The only way I know to related all events to one tmp file is by the handle ID. I can filter the event for tmp file base on a regex for the file name. In rsyslog, does it have a way to extract a string from a rawmsg and assign it to a variable to be used by another filter to filter the related events to this tmp file? I hope I am clear enough. By the way, my version of rsyslog is 4.2.0. Thanks, Bernard From pgollucci at p6m7g8.com Fri Dec 10 07:46:54 2010 From: pgollucci at p6m7g8.com (Philip M. Gollucci) Date: Fri, 10 Dec 2010 06:46:54 +0000 Subject: [rsyslog] Fwd: cvs commit: ports/sysutils/rsyslog3 Makefile bsd.rsyslog.mk distinfo ports/sysutils/rsyslog4 Makefile distinfo pkg-plist ports/sysutils/rsyslog4-devel Makefile distinfo pkg-plist ports/sysutils/rsyslog5 Makefile distinfo ... Message-ID: <4D01CCDE.1060806@p6m7g8.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 FYI - -------- Original Message -------- Subject: cvs commit: ports/sysutils/rsyslog3 Makefile bsd.rsyslog.mk distinfo ports/sysutils/rsyslog4 Makefile distinfo pkg-plist ports/sysutils/rsyslog4-devel Makefile distinfo pkg-plist ports/sysutils/rsyslog5 Makefile distinfo ... Date: Fri, 10 Dec 2010 06:44:20 +0000 (UTC) From: Philip M. Gollucci To: ports-committers at FreeBSD.org, cvs-ports at FreeBSD.org, cvs-all at FreeBSD.org pgollucci 2010-12-10 06:44:20 UTC FreeBSD ports repository Modified files: sysutils/rsyslog3 Makefile bsd.rsyslog.mk distinfo sysutils/rsyslog4 Makefile distinfo pkg-plist sysutils/rsyslog4-devel Makefile distinfo pkg-plist sysutils/rsyslog5 Makefile distinfo sysutils/rsyslog5-devel Makefile distinfo sysutils/rsyslog6-devel Makefile distinfo pkg-plist sysutils/rsyslog6-devel/files patch-Makefile.in Log: - Update: rsyslog3 -> 3.22.3, unbreak rsyslog4 -> 4.6.5 rsyslog4-devel -> 4.7.3 rsyslog5 -> 5.6.2 rsyslog5-devel -> 5.7.2 rsyslog6-devel -> 6.1.1 Sponsored by: RideCharge Inc. / TaxiMagic Revision Changes Path 1.54 +0 -2 ports/sysutils/rsyslog3/Makefile 1.18 +1 -1 ports/sysutils/rsyslog3/bsd.rsyslog.mk 1.39 +2 -3 ports/sysutils/rsyslog3/distinfo 1.18 +2 -2 ports/sysutils/rsyslog4-devel/Makefile 1.12 +2 -3 ports/sysutils/rsyslog4-devel/distinfo 1.8 +2 -1 ports/sysutils/rsyslog4-devel/pkg-plist 1.17 +2 -2 ports/sysutils/rsyslog4/Makefile 1.11 +2 -3 ports/sysutils/rsyslog4/distinfo 1.6 +0 -1 ports/sysutils/rsyslog4/pkg-plist 1.23 +1 -1 ports/sysutils/rsyslog5-devel/Makefile 1.16 +2 -3 ports/sysutils/rsyslog5-devel/distinfo 1.23 +1 -1 ports/sysutils/rsyslog5/Makefile 1.16 +2 -3 ports/sysutils/rsyslog5/distinfo 1.26 +1 -1 ports/sysutils/rsyslog6-devel/Makefile 1.19 +2 -3 ports/sysutils/rsyslog6-devel/distinfo 1.2 +16 -16 ports/sysutils/rsyslog6-devel/files/patch-Makefile.in 1.14 +1 -0 ports/sysutils/rsyslog6-devel/pkg-plist http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog3/Makefile.diff?r1=1.53&r2=1.54&f=h | --- ports/sysutils/rsyslog3/Makefile 2010/10/19 21:31:53 1.53 | +++ ports/sysutils/rsyslog3/Makefile 2010/12/10 06:44:19 1.54 | @@ -2,7 +2,7 @@ | # Date created: 9 July 2007 | # Whom: Andrew Pantyukhin | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog3/Makefile,v 1.53 2010/10/19 21:31:53 pav Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog3/Makefile,v 1.54 2010/12/10 06:44:19 pgollucci Exp $ | # | | PORTNAME= rsyslog | @@ -11,8 +11,6 @@ COMMENT= Enhanced syslog daemon | | LIB_DEPENDS= gnutls.40:${PORTSDIR}/security/gnutls | | -BROKEN= does not compile | - | USE_RC_SUBR= ${PORTNAME}d | SUB_FILES= pkg-message | MAN8= rsyslogd.8 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog3/bsd.rsyslog.mk.diff?r1=1.17&r2=1.18&f=h | --- ports/sysutils/rsyslog3/bsd.rsyslog.mk 2010/10/28 21:00:17 1.17 | +++ ports/sysutils/rsyslog3/bsd.rsyslog.mk 2010/12/10 06:44:19 1.18 | @@ -1,5 +1,5 @@ | PORTNAME?= rsyslog | -PORTVERSION?= 3.22.2 | +PORTVERSION?= 3.22.3 | CATEGORIES?= sysutils | MASTER_SITES?= http://www.rsyslog.com/files/download/rsyslog/ | MAINTAINER?= miwi at FreeBSD.org http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog3/distinfo.diff?r1=1.38&r2=1.39&f=h | --- ports/sysutils/rsyslog3/distinfo 2010/10/06 00:01:22 1.38 | +++ ports/sysutils/rsyslog3/distinfo 2010/12/10 06:44:19 1.39 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-3.22.2.tar.gz) = b6a3b591764b20679dfb142219c4a8b8 | -SHA256 (rsyslog-3.22.2.tar.gz) = d7afc6cb0fead582a20a7aa1f1fed9ee0f6ed41c4fa930d4d16095edb8183db3 | -SIZE (rsyslog-3.22.2.tar.gz) = 1315816 | +SHA256 (rsyslog-3.22.3.tar.gz) = d234c607ec8a93c579c1ebb5af26cc26d4616362547eaac98529574a84b23f5e | +SIZE (rsyslog-3.22.3.tar.gz) = 1315930 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4-devel/Makefile.diff?r1=1.17&r2=1.18&f=h | --- ports/sysutils/rsyslog4-devel/Makefile 2010/10/28 21:00:18 1.17 | +++ ports/sysutils/rsyslog4-devel/Makefile 2010/12/10 06:44:20 1.18 | @@ -2,13 +2,13 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4-devel/Makefile,v 1.17 2010/10/28 21:00:18 erwin Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4-devel/Makefile,v 1.18 2010/12/10 06:44:20 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 4.7.2 | +PORTVERSION= 4.7.3 | CATEGORIES= sysutils | -MASTER_SITES= http://download.rsyslog.com/rsyslog/ | +MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME | PKGNAMESUFFIX?= -${MNAME} | .endif http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4-devel/distinfo.diff?r1=1.11&r2=1.12&f=h | --- ports/sysutils/rsyslog4-devel/distinfo 2010/10/06 00:07:04 1.11 | +++ ports/sysutils/rsyslog4-devel/distinfo 2010/12/10 06:44:20 1.12 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-4.7.2.tar.gz) = 15b5dd2330b1ec4943f32032d388d2a7 | -SHA256 (rsyslog-4.7.2.tar.gz) = 007e3777291d679fec0fec3de1cb65e2aa5ce90ae2424812bcaaae302843e93e | -SIZE (rsyslog-4.7.2.tar.gz) = 2108575 | +SHA256 (rsyslog-4.7.3.tar.gz) = a4063b6027e01dc380a97833ceb98ca0b0a060b9e843ed794829f60bd7af93b9 | +SIZE (rsyslog-4.7.3.tar.gz) = 2129255 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4-devel/pkg-plist.diff?r1=1.7&r2=1.8&f=h | --- ports/sysutils/rsyslog4-devel/pkg-plist 2010/10/06 00:07:04 1.7 | +++ ports/sysutils/rsyslog4-devel/pkg-plist 2010/12/10 06:44:20 1.8 | @@ -49,6 +49,7 @@ sbin/rsyslogd | %%PORTDOCS%%%%DOCSDIR%%/imfile.html | %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html | %%PORTDOCS%%%%DOCSDIR%%/imklog.html | +%%PORTDOCS%%%%DOCSDIR%%/imptcp.html | %%PORTDOCS%%%%DOCSDIR%%/imrelp.html | %%PORTDOCS%%%%DOCSDIR%%/imtcp.html | %%PORTDOCS%%%%DOCSDIR%%/imuxsock.html | @@ -70,7 +71,7 @@ sbin/rsyslogd | %%PORTDOCS%%%%DOCSDIR%%/omoracle.html | %%PORTDOCS%%%%DOCSDIR%%/omrelp.html | %%PORTDOCS%%%%DOCSDIR%%/omsnmp.html | -%%PORTDOCS%%%%DOCSDIR%%/professional_support.html | +%%PORTDOCS%%%%DOCSDIR%%/omuxsock.html | %%PORTDOCS%%%%DOCSDIR%%/property_replacer.html | %%PORTDOCS%%%%DOCSDIR%%/queueWorkerLogic.jpg | %%PORTDOCS%%%%DOCSDIR%%/queueWorkerLogic_small.jpg http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4/Makefile.diff?r1=1.16&r2=1.17&f=h | --- ports/sysutils/rsyslog4/Makefile 2010/10/28 21:00:18 1.16 | +++ ports/sysutils/rsyslog4/Makefile 2010/12/10 06:44:19 1.17 | @@ -2,13 +2,13 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4/Makefile,v 1.16 2010/10/28 21:00:18 erwin Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog4/Makefile,v 1.17 2010/12/10 06:44:19 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 4.6.3 | +PORTVERSION= 4.6.5 | CATEGORIES= sysutils | -MASTER_SITES= http://download.rsyslog.com/rsyslog/ | +MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME | PKGNAMESUFFIX?= -${MNAME} | .endif http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4/distinfo.diff?r1=1.10&r2=1.11&f=h | --- ports/sysutils/rsyslog4/distinfo 2010/10/06 00:02:54 1.10 | +++ ports/sysutils/rsyslog4/distinfo 2010/12/10 06:44:19 1.11 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-4.6.3.tar.gz) = 41ff79a72a931a612fda251cfd400f4e | -SHA256 (rsyslog-4.6.3.tar.gz) = b9c430527a5bd95da9ce1b43b7f64804e19d173c8b420418f40a4485eff9bd20 | -SIZE (rsyslog-4.6.3.tar.gz) = 2085422 | +SHA256 (rsyslog-4.6.5.tar.gz) = 577b35340d4d0ba95c5d444d90282dfc938e2db8cf7c9ac6996d5ac111961195 | +SIZE (rsyslog-4.6.5.tar.gz) = 2080355 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog4/pkg-plist.diff?r1=1.5&r2=1.6&f=h | --- ports/sysutils/rsyslog4/pkg-plist 2010/02/25 05:27:42 1.5 | +++ ports/sysutils/rsyslog4/pkg-plist 2010/12/10 06:44:19 1.6 | @@ -69,7 +69,6 @@ sbin/rsyslogd | %%PORTDOCS%%%%DOCSDIR%%/omoracle.html | %%PORTDOCS%%%%DOCSDIR%%/omrelp.html | %%PORTDOCS%%%%DOCSDIR%%/omsnmp.html | -%%PORTDOCS%%%%DOCSDIR%%/professional_support.html | %%PORTDOCS%%%%DOCSDIR%%/property_replacer.html | %%PORTDOCS%%%%DOCSDIR%%/queueWorkerLogic.jpg | %%PORTDOCS%%%%DOCSDIR%%/queueWorkerLogic_small.jpg http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5-devel/Makefile.diff?r1=1.22&r2=1.23&f=h | --- ports/sysutils/rsyslog5-devel/Makefile 2010/10/28 21:00:18 1.22 | +++ ports/sysutils/rsyslog5-devel/Makefile 2010/12/10 06:44:20 1.23 | @@ -2,11 +2,11 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5-devel/Makefile,v 1.22 2010/10/28 21:00:18 erwin Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5-devel/Makefile,v 1.23 2010/12/10 06:44:20 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 5.7.0 | +PORTVERSION= 5.7.2 | CATEGORIES= sysutils | MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5-devel/distinfo.diff?r1=1.15&r2=1.16&f=h | --- ports/sysutils/rsyslog5-devel/distinfo 2010/10/06 00:14:37 1.15 | +++ ports/sysutils/rsyslog5-devel/distinfo 2010/12/10 06:44:20 1.16 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-5.7.0.tar.gz) = 6f73e1dc8be7c3702283137ef7b88671 | -SHA256 (rsyslog-5.7.0.tar.gz) = aec838f402dc74d8d087468556e36d1c7e67e179b6247431aeeee8ee815898f7 | -SIZE (rsyslog-5.7.0.tar.gz) = 2271241 | +SHA256 (rsyslog-5.7.2.tar.gz) = d0a73a7aa0e2ec06b56af68efea3c3d899bc55bc2539022f8d2316d18a9c6408 | +SIZE (rsyslog-5.7.2.tar.gz) = 2301370 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/Makefile.diff?r1=1.22&r2=1.23&f=h | --- ports/sysutils/rsyslog5/Makefile 2010/10/28 21:00:18 1.22 | +++ ports/sysutils/rsyslog5/Makefile 2010/12/10 06:44:20 1.23 | @@ -2,11 +2,11 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.22 2010/10/28 21:00:18 erwin Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog5/Makefile,v 1.23 2010/12/10 06:44:20 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 5.6.0 | +PORTVERSION= 5.6.2 | CATEGORIES= sysutils | MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog5/distinfo.diff?r1=1.15&r2=1.16&f=h | --- ports/sysutils/rsyslog5/distinfo 2010/10/21 21:46:49 1.15 | +++ ports/sysutils/rsyslog5/distinfo 2010/12/10 06:44:20 1.16 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-5.6.0.tar.gz) = 62c8edad559a39ca8c124ca87908075b | -SHA256 (rsyslog-5.6.0.tar.gz) = 54dde8f250aa641d810cafc6745403a753bc2822b6aeef0df094cb7103a3c505 | -SIZE (rsyslog-5.6.0.tar.gz) = 2253026 | +SHA256 (rsyslog-5.6.2.tar.gz) = 09c86194f48d2c8ba3dc27b78b19a4175e90c77f9fe13ea97ba226584b69598c | +SIZE (rsyslog-5.6.2.tar.gz) = 2255892 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/Makefile.diff?r1=1.25&r2=1.26&f=h | --- ports/sysutils/rsyslog6-devel/Makefile 2010/10/28 21:00:18 1.25 | +++ ports/sysutils/rsyslog6-devel/Makefile 2010/12/10 06:44:20 1.26 | @@ -2,11 +2,11 @@ | # Date created: 29 December 2008 | # Whom: Cristiano Rolim Pereira | # | -# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog6-devel/Makefile,v 1.25 2010/10/28 21:00:18 erwin Exp $ | +# $FreeBSD: /usr/local/www/cvsroot/FreeBSD/ports/sysutils/rsyslog6-devel/Makefile,v 1.26 2010/12/10 06:44:20 pgollucci Exp $ | # | | PORTNAME= rsyslog | -PORTVERSION= 6.1.0 | +PORTVERSION= 6.1.1 | CATEGORIES= sysutils | MASTER_SITES= http://www.rsyslog.com/files/download/rsyslog/ | .ifdef MNAME http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/distinfo.diff?r1=1.18&r2=1.19&f=h | --- ports/sysutils/rsyslog6-devel/distinfo 2010/10/06 00:16:57 1.18 | +++ ports/sysutils/rsyslog6-devel/distinfo 2010/12/10 06:44:20 1.19 | @@ -1,3 +1,2 @@ | -MD5 (rsyslog-6.1.0.tar.gz) = 9fefe1e486084446ffeff728e53d5001 | -SHA256 (rsyslog-6.1.0.tar.gz) = 8511b1a0e2429b44a5450a3b0346595703834ca3169e2dcd16948e2d145c0df3 | -SIZE (rsyslog-6.1.0.tar.gz) = 2263513 | +SHA256 (rsyslog-6.1.1.tar.gz) = 970d1ae88ea544ff916a61c572afbe7a4617ef675b6a017812bb67a78265f00f | +SIZE (rsyslog-6.1.1.tar.gz) = 2312617 http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/files/patch-Makefile.in.diff?r1=1.1&r2=1.2&f=h | --- ports/sysutils/rsyslog6-devel/files/patch-Makefile.in 2010/10/06 00:20:08 1.1 | +++ ports/sysutils/rsyslog6-devel/files/patch-Makefile.in 2010/12/10 06:44:20 1.2 | @@ -1,19 +1,19 @@ | ---- ./Makefile.in.orig 2010-10-05 05:14:22.724781915 +0000 | -+++ ./Makefile.in 2010-10-05 05:14:41.908897529 +0000 | -@@ -52,7 +52,6 @@ | - @ENABLE_PGSQL_TRUE at am__append_10 = plugins/ompgsql | - @ENABLE_SNMP_TRUE at am__append_11 = plugins/omsnmp | - @ENABLE_CUST1_TRUE at am__append_12 = plugins/cust1 | -- at ENABLE_IMTEMPLATE_TRUE@am__append_13 = plugins/imtemplate | - @ENABLE_OMSTDOUT_TRUE at am__append_14 = plugins/omstdout | - @ENABLE_PMLASTMSG_TRUE at am__append_15 = plugins/pmlastmsg | - @ENABLE_PMRFC3164SD_TRUE at am__append_16 = plugins/pmrfc3164sd | -@@ -194,7 +193,7 @@ | - plugins/imklog plugins/imsolaris plugins/omgssapi \ | - plugins/imgssapi plugins/omrelp plugins/imrelp plugins/ommysql \ | - plugins/omlibdbi plugins/ompgsql plugins/omsnmp plugins/cust1 \ | +--- ./Makefile.in.orig 2010-11-30 10:45:25.000000000 +0000 | ++++ ./Makefile.in 2010-12-10 06:41:03.768306480 +0000 | +@@ -54,7 +54,6 @@ | + @ENABLE_PGSQL_TRUE at am__append_11 = plugins/ompgsql | + @ENABLE_SNMP_TRUE at am__append_12 = plugins/omsnmp | + @ENABLE_CUST1_TRUE at am__append_13 = plugins/cust1 | +- at ENABLE_IMTEMPLATE_TRUE@am__append_14 = plugins/imtemplate | + @ENABLE_OMSTDOUT_TRUE at am__append_15 = plugins/omstdout | + @ENABLE_PMLASTMSG_TRUE at am__append_16 = plugins/pmlastmsg | + @ENABLE_PMRFC3164SD_TRUE at am__append_17 = plugins/pmrfc3164sd | +@@ -203,7 +202,7 @@ | + plugins/omgssapi plugins/imgssapi plugins/omrelp \ | + plugins/imrelp plugins/ommysql plugins/omlibdbi \ | + plugins/ompgsql plugins/omsnmp plugins/cust1 \ | - plugins/imtemplate plugins/omstdout plugins/pmlastmsg \ | + plugins/omstdout plugins/pmlastmsg \ | plugins/pmrfc3164sd plugins/omruleset plugins/omdbalerting \ | - plugins/omudpspoof plugins/omuxsock plugins/omtemplate \ | - plugins/imfile plugins/imptcp plugins/imdiag plugins/ommail \ | + plugins/omudpspoof plugins/omuxsock plugins/omhdfs \ | + plugins/omtemplate plugins/imfile plugins/imptcp \ http://cvsweb.FreeBSD.org/ports/sysutils/rsyslog6-devel/pkg-plist.diff?r1=1.13&r2=1.14&f=h | --- ports/sysutils/rsyslog6-devel/pkg-plist 2010/10/07 22:42:31 1.13 | +++ ports/sysutils/rsyslog6-devel/pkg-plist 2010/12/10 06:44:20 1.14 | @@ -50,6 +50,7 @@ sbin/rsyslogd | %%PORTDOCS%%%%DOCSDIR%%/imgssapi.html | %%PORTDOCS%%%%DOCSDIR%%/imklog.html | %%PORTDOCS%%%%DOCSDIR%%/imptcp.html | +%%PORTDOCS%%%%DOCSDIR%%/impstats.html | %%PORTDOCS%%%%DOCSDIR%%/imrelp.html | %%PORTDOCS%%%%DOCSDIR%%/imsolaris.html | %%PORTDOCS%%%%DOCSDIR%%/imtcp.html -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iD8DBQFNAczddbiP+9ubjBwRAoHuAJ9kssRDXvraByt0YKCpz1vanXrprACfY96/ q7EeZhBOnbjtg5T0e64/FXI= =H+DT -----END PGP SIGNATURE----- From david at lang.hm Fri Dec 10 09:12:13 2010 From: david at lang.hm (david at lang.hm) Date: Fri, 10 Dec 2010 00:12:13 -0800 (PST) Subject: [rsyslog] variables in filters In-Reply-To: References: Message-ID: no,it not possible to set a variable that then gets used in the config file. David Lang On Thu, 9 Dec 2010, Bernard Fay wrote: > Hi, > > Is-it possible to use variable in rsyslog? > > We have a rsyslog server that collects events from windows servers. Some > events from windows are about tmp files that we don't need in our rsyslog > server. For one tmp file there is many events but the name of the file is > not shown in all events. The only way I know to related all events to one > tmp file is by the handle ID. I can filter the event for tmp file base on a > regex for the file name. In rsyslog, does it have a way to extract a string > from a rawmsg and assign it to a variable to be used by another filter to > filter the related events to this tmp file? > > I hope I am clear enough. > > By the way, my version of rsyslog is 4.2.0. > > Thanks, > Bernard > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From tbergfeld at hq.adiscon.com Fri Dec 10 13:16:45 2010 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 10 Dec 2010 13:16:45 +0100 Subject: [rsyslog] liblognorm, libee and libestr has been released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> Hi all, we have just released liblognorm, libee and libestr. liblognorm 0.1.0 has been released Thursday, December 9th, 2010 Liblognorm is a event and log normalization library that is capable of real-time processing. It provides the capability to normalize events to a set of standard formats. It is most efficient when used together with almost unstructured data, as for example found in typical syslog messages. While liblognorm provides a service similar to other projects, it is unique in the way it works: a) As a library, it's services can be used by any third-party application with ease. As such, other projects can benefit from liblognorm functionality without the user even knowing. b) Liblognorm is very fast, and especially much faster than regular-expression based solutions. This is possible because it uses a special data structure which (kind of) combines many regular expressions into a single, and thus faster, big one. This is the initial public release, targeted at early adopters. We will continue to enhance considerably, but the 0.1.0 version offers decent stability and features and so can actually be used. You can download libee here. libee 0.1.0 has been released Thursday, December 9th, 2010 Libee is an event expression library which is inspired by the upcoming CEE standard. Right now, it provides capabilities to generate and emit messages in a set of standard format and read a set of different input formats. Libee also comes with a handy conversion tool that provide format transformation without the need to program. This is the initial public release. You can download libee here. libestr 0.1.0 Thursday, December 9th, 2010 This is the initial public release of libestr, a library for some string essentials. You can download it here. -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From dave at fly.srk.fer.hr Fri Dec 10 15:48:44 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Fri, 10 Dec 2010 15:48:44 +0100 Subject: [rsyslog] Closing all file descriptors in a process Message-ID: <20101210144844.GB5541@fly.srk.fer.hr> There is a comment in omprog.c: /* we close all file handles as we fork soon * Is there a better way to do this? - mail me! * rgerhards at adiscon.com */ for(i = 3 ; i <= 65535 ; ++i) close(i); The better, not-portable, way is to open /proc/self/fd directory, read all the entries and close exactly those file descriptors. Configure script should check if /proc/self/fd exists and has the right contents. If it doesn't exist, then check for /dev/fd directory. Perhaps the check needs to be done in run-time, as well. IIRC, FreeBSD doesn't eneble procfs by default, but it can be mounted by administrators who wish to do so. I don't know if FreeBSD has /dev/fd. If the above directory is not available, use getrlimit(RLIMIT_NOFILE) to get the upper bound for the file descriptor numbers (it will be in the soft limit field) and close them all in a loop. The limit will usually be much lower than 65536. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From dave at fly.srk.fer.hr Fri Dec 10 18:06:43 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Fri, 10 Dec 2010 18:06:43 +0100 Subject: [rsyslog] Huge pauses when writing to FIFO In-Reply-To: <20101116143404.GA1167@fly.srk.fer.hr> References: <20101116125645.GB26159@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD726@GRFEXC.intern.adiscon.com> <20101116143404.GA1167@fly.srk.fer.hr> Message-ID: <20101210170643.GC5541@fly.srk.fer.hr> Dra?en Ka?ar wrote: > Then I negated the condition in preparePipe() to: > > if(!pData->bBlocking) > mode |= O_NONBLOCK; > > and now strace doesn't show O_NONBLOCK and there is no sleep in select(). > I'll run more comprehensive tests just to be sure, but this looks ok. I've tried using the blocking ompipe in real world and it's not functioning too well. The open function uses O_RDWR on a FIFO. That happens to yield undefined behavior under POSIX and also some real implementations (Solaris, for example). But I'm on Linux, so it's documented to work always. The problem with that is that rsyslog has that FIFO opened for reading as well, so it can write to it regardless of whether the intended reader has opened the FIFO or not. And so we can lose messages if rsyslog reopens the FIFO before the reader manages to open it. So I changed the preparePipe() code with this: rdfd = open((char*) pData->f_fname, O_RDONLY|O_NONBLOCK|O_CLOEXEC); pData->fd = open((char*) pData->f_fname, O_WRONLY|O_NONBLOCK|O_CLOEXEC); if(pData->bBlocking) { long flags; flags = fcntl(pData->fd, F_GETFL); fcntl(pData->fd, F_SETFL, flags & ~O_NONBLOCK); } close(rdfd); That will always open FIFO and it's portable. But then, when there is no reader, ompipe is constantly trying to write and all those writes fail with EPIPE. The writing might get suspended after a certain number of attempts, but then there'd be a long pause before the next attempt, which is the kind of behavior I wanted to avoid. So I added a blocking open()[1] to the writePipe() function in case when the write() fails with EPIPE on a blocking FIFO. The thread will block, but the blocking FIFO is supposed to be used on its own action queue, so I hope everything will work fine in that case. The patch for rsyslog 5.6.2 is attached. I didn't test it on non-blocking FIFOs, though. Any comments? [1] poll() would return immediately with POLLERR, so can't be used for this purpose. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr -------------- next part -------------- A non-text attachment was scrubbed... Name: rsyslog-ompipe-block-mode.diff Type: text/x-diff Size: 2450 bytes Desc: not available URL: From champ at softwink.com Fri Dec 10 19:29:14 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Fri, 10 Dec 2010 13:29:14 -0500 Subject: [rsyslog] liblognorm, libee and libestr has been released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> Message-ID: <20101210182914.GA7734@bundy.vistech.net> On Fri, Dec 10, 2010 at 01:16:45PM +0100, Tom Bergfeld wrote: > Hi all, > > we have just released liblognorm, libee and libestr. > > liblognorm 0.1.0 has been released > Thursday, December 9th, 2010 First off, congrat's on the release. I know it's still far from done, but it's a great stepping stone. I had the chance to play with liblognorm a bit today. It's pretty dang straight forward. I have one question (code snip it): es_emptyStr(str); ee_fmtEventToRFC5424(event, &str); cstr = es_str2cstr(str, NULL); printf("Normalize: '%s'\n", cstr); .. And everything normalizes as expected .. [cee at 115 port="48973" ip="192.168.0.1" tag="sshd[4416\]" host="myhost" date="Dec 10 12:13:50"] How do I pull just the "ip" field? That is, how to I pull seperated fields (ip, tag, host, date, etc)? This is probably a simple question, but figured I might ask. I'll continue looking around. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From john at feurix.com Sat Dec 11 11:35:50 2010 From: john at feurix.com (John Feuerstein) Date: Sat, 11 Dec 2010 11:35:50 +0100 Subject: [rsyslog] Closing all file descriptors in a process In-Reply-To: <20101210144844.GB5541@fly.srk.fer.hr> References: <20101210144844.GB5541@fly.srk.fer.hr> Message-ID: <4D035406.8070508@feurix.com> This stackoverflow answer[1] by mark4o might also be of interest: > While portable, closing all file descriptors up to > sysconf(_SC_OPEN_MAX) is not reliable, because on most systems this > call returns the current file descriptor soft limit, which could have > been lowered below the highest used file descriptor. Another issue is > that on many systems sysconf(_SC_OPEN_MAX) may return INT_MAX, which > can cause this approach to be unacceptably slow. Unfortunately, there > is no reliable, portable alternative that does not involve iterating > over every possible non-negative int file descriptor. > > Although not portable, most operating systems in common use today > provide one or more of the following solutions to this problem: > 1. A library function to close all file descriptors >= fd. This is the > simplest solution for the common case of closing all file > descriptors, although it cannot be used for much else. To close all > file descriptors except for a certain set, dup2 can be used to move > them to the low end beforehand, and to move them back afterward if > necessary. > + closefrom(fd) (Solaris 9 or later, NetBSD 3.0 or later, > OpenBSD 3.5 or later.) > + fcntl(fd, F_CLOSEM, 0) (AIX, IRIX, NetBSD) > 2. A library function to provide the maximum file descriptor currently > in use by the process. To close all file descriptors above a > certain number, either close all of them up to this maximum, or > continually get and close the highest file descriptor in a loop > until the low bound is reached. Which is more efficient depends on > the file descriptor density. > + fcntl(0, F_MAXFD) (NetBSD) > + pstat_getproc(&ps, sizeof(struct pst_status), (size_t)0, > (int)getpid()) > Returns information about the process, including the highest > file descriptor currently open in ps.pst_highestfd. (HP-UX) > 3. A directory containing an entry for each open file descriptor. This > is the most flexible approach as it allows for closing all file > descriptors, finding the highest file descriptor, or doing just > about anything else on every open file descriptor, even those of > another process (on most systems). However this can be more > complicated than the other approaches for the common uses. Also, it > can fail for a variety of reasons such as proc/fdescfs not mounted, > a chroot environment, or no file descriptors available to open the > directory (process or system limit). Therefore use of this approach > is often combined with a fallback mechanism. [2]Example (OpenSSH), > [3]another example (glib). > + /proc/pid/fd/ or /proc/self/fd/ (Linux, Solaris, AIX, Cygwin, > NetBSD) (AIX does not support "self".) > + /dev/fd/ (FreeBSD, Darwin) > It can be difficult to handle all corner cases reliably with this > approach. For example consider the situation where all file > descriptors >= fd are to be closed, but all file descriptors < fd > are used, the current process resource limit is fd, and there are > file descriptors >= fd in use. Because the process resource limit > has been reached the directory cannot be opened. If closing every > file descriptor from fd through the resource limit or > sysconf(_SC_OPEN_MAX) is used as a fallback, nothing will be > closed. [1] http://stackoverflow.com/questions/899038/getting-the-highest-allocated-file-descriptor/918469#918469 [2] http://anoncvs.mindrot.org/index.cgi/openssh/openbsd-compat/bsd-closefrom.c?view=markup [3] http://git.gnome.org/cgit/glib/tree/gio/libasyncns/asyncns.c?h=2.21.0#n205 From rgerhards at hq.adiscon.com Mon Dec 13 11:14:09 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Dec 2010 11:14:09 +0100 Subject: [rsyslog] liblognorm, libee and libestr has been released References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> <20101210182914.GA7734@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8B3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Friday, December 10, 2010 7:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] liblognorm, libee and libestr has been released > > On Fri, Dec 10, 2010 at 01:16:45PM +0100, Tom Bergfeld wrote: > > Hi all, > > > > we have just released liblognorm, libee and libestr. > > > > liblognorm 0.1.0 has been released > > Thursday, December 9th, 2010 > > First off, congrat's on the release. I know it's still far > from done, but it's a great stepping stone. > > I had the chance to play with liblognorm a bit today. It's > pretty dang straight forward. I have one question (code snip it): > > es_emptyStr(str); > ee_fmtEventToRFC5424(event, &str); > cstr = es_str2cstr(str, NULL); > printf("Normalize: '%s'\n", cstr); > > .. And everything normalizes as expected .. > > [cee at 115 port="48973" ip="192.168.0.1" tag="sshd[4416\]" host="myhost" > date="Dec 10 12:13:50"] > > How do I pull just the "ip" field? That is, how to I pull > seperated > fields (ip, tag, host, date, etc)? This is probably a simple > question, > but figured I might ask. I'll continue looking around. I updated the online doc. It is this function: http://doc.libee.org/structee__event.html#a4e3ae5e5ba094572a9d7b5368b63b279 A practical use sample of this function can be found in rsyslog's git: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/msg.c;h=65ea101f3a95c3 cd1852d8a790c04246ff0840c4;hb=master#l2240 HTH Rainer PS: I plan to release an experimental rsyslog version with normalization support today or tomorrow (a bit more probable) From marcin at mejor.pl Mon Dec 13 16:57:26 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 13 Dec 2010 16:57:26 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error Message-ID: <4D064266.6050604@mejor.pl> Hello. Libee doesn't compile with make -jN, i get error: libtool: link: cannot find the library `../src/libee.la' or unhandled argument `../src/libee.la' (I can't bugzilla for libee, so i'm writing on mailinglist) Regards, Marcin From marcin at mejor.pl Mon Dec 13 17:21:03 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Mon, 13 Dec 2010 17:21:03 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error In-Reply-To: <4D064266.6050604@mejor.pl> References: <4D064266.6050604@mejor.pl> Message-ID: <4D0647EF.4060904@mejor.pl> W dniu 13.12.2010 16:57, Marcin Miros?aw pisze: > Hello. > Libee doesn't compile with make -jN, i get error: The same with liblognorm. From mbiebl at gmail.com Mon Dec 13 18:21:25 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 13 Dec 2010 18:21:25 +0100 Subject: [rsyslog] liblognorm, libee and libestr has been released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> Message-ID: Hi 2010/12/10 Tom Bergfeld : > Hi all, > > we have just released liblognorm, libee and libestr. > The COPYING file in libee and libestr is a copy of Doxyfile. Looks like a c&p error. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From marcin at mejor.pl Mon Dec 13 18:45:30 2010 From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=) Date: Mon, 13 Dec 2010 18:45:30 +0100 Subject: [rsyslog] liblognorm, libee and libestr has been released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> Message-ID: <4D065BBA.2060608@mejor.pl> W dniu 2010-12-10 13:16, Tom Bergfeld pisze: > Hi all, > > we have just released liblognorm, libee and libestr. Hello, ebuilds for Gentoo are here: http://repoz.mejor.pl/svn/gentoo/portage/dev-libs/ (address for overlay is: http://repoz.mejor.pl/svn/gentoo/portage/). I'll fill bug in gentoo bugzilla in one,two days. From mbiebl at gmail.com Mon Dec 13 19:17:29 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 13 Dec 2010 19:17:29 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error In-Reply-To: <4D064266.6050604@mejor.pl> References: <4D064266.6050604@mejor.pl> Message-ID: 2010/12/13 Marcin Miros?aw : > Hello. > Libee doesn't compile with make -jN, i get error: > libtool: link: cannot find the library `../src/libee.la' or unhandled > argument `../src/libee.la' could you paste the complete build log, please -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From marcin at mejor.pl Mon Dec 13 19:24:21 2010 From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=) Date: Mon, 13 Dec 2010 19:24:21 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error In-Reply-To: References: <4D064266.6050604@mejor.pl> Message-ID: <4D0664D5.50305@mejor.pl> W dniu 2010-12-13 19:17, Michael Biebl pisze: > 2010/12/13 Marcin Miros?aw : >> Hello. >> Libee doesn't compile with make -jN, i get error: >> libtool: link: cannot find the library `../src/libee.la' or unhandled >> argument `../src/libee.la' > > could you paste the complete build log, please I run: make -j . Buil log is attached. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: libee-build.log URL: From joe at joetify.com Mon Dec 13 21:03:39 2010 From: joe at joetify.com (Joe Williams) Date: Mon, 13 Dec 2010 12:03:39 -0800 Subject: [rsyslog] setting the hostname Message-ID: List, I would like to statically set the hostname (don't use the actual hostname of the system) that rsyslogd uses for a node in the configuration or as an argument to the rsyslogd command on that node. I didn't immediately see a way to do this in the docs, is this possible? Thanks. -Joe Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From rgerhards at hq.adiscon.com Mon Dec 13 22:02:41 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 13 Dec 2010 22:02:41 +0100 Subject: [rsyslog] setting the hostname References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> Unfortunately, this is not possible... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Joe Williams > Sent: Monday, December 13, 2010 9:04 PM > To: rsyslog-users > Subject: [rsyslog] setting the hostname > > > List, > > I would like to statically set the hostname (don't use the actual > hostname of the system) that rsyslogd uses for a node in the > configuration or as an argument to the rsyslogd command on that node. I > didn't immediately see a way to do this in the docs, is this possible? > > Thanks. > > -Joe > > > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Mon Dec 13 22:20:07 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 13:20:07 -0800 (PST) Subject: [rsyslog] setting the hostname In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> Message-ID: On Mon, 13 Dec 2010, Rainer Gerhards wrote: > Unfortunately, this is not possible... however, what you can do is to change the template to put any string you want instead of %hostname%, that will have the same effect. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Joe Williams >> Sent: Monday, December 13, 2010 9:04 PM >> To: rsyslog-users >> Subject: [rsyslog] setting the hostname >> >> >> List, >> >> I would like to statically set the hostname (don't use the actual >> hostname of the system) that rsyslogd uses for a node in the >> configuration or as an argument to the rsyslogd command on that node. I >> didn't immediately see a way to do this in the docs, is this possible? >> >> Thanks. >> >> -Joe >> >> >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbiebl at gmail.com Mon Dec 13 22:50:58 2010 From: mbiebl at gmail.com (Michael Biebl) Date: Mon, 13 Dec 2010 22:50:58 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error In-Reply-To: <4D0664D5.50305@mejor.pl> References: <4D064266.6050604@mejor.pl> <4D0664D5.50305@mejor.pl> Message-ID: 2010/12/13 Marcin Miros?aw : > W dniu 2010-12-13 19:17, Michael Biebl pisze: >> 2010/12/13 Marcin Miros?aw : >>> Hello. >>> Libee doesn't compile with make -jN, i get error: >>> libtool: link: cannot find the library `../src/libee.la' or unhandled >>> argument `../src/libee.la' >> >> could you paste the complete build log, please > > > I run: make -j . Buil log is attached. A simple patch like the attached one should suffice. Problem is, that automake does not generate a dependency if you add it to _LIBADD via variable substitution. Rainer, I also noticed that you defined LIBEE_CFLAGS in configure.ac *and* src/Makefile.am? Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -------------- next part -------------- A non-text attachment was scrubbed... Name: libee-build-fix.patch Type: text/x-patch Size: 440 bytes Desc: not available URL: From marcin at mejor.pl Tue Dec 14 00:04:48 2010 From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=) Date: Tue, 14 Dec 2010 00:04:48 +0100 Subject: [rsyslog] libee-0.1.0 parallel make error In-Reply-To: References: <4D064266.6050604@mejor.pl> <4D0664D5.50305@mejor.pl> Message-ID: <4D06A690.7000903@mejor.pl> W dniu 2010-12-13 22:50, Michael Biebl pisze: > A simple patch like the attached one should suffice. > > Problem is, that automake does not generate a dependency if you add it > to _LIBADD via variable substitution. Works for me, thanks! > Rainer, I also noticed that you defined LIBEE_CFLAGS in configure.ac > *and* src/Makefile.am? About CFLAGS, libee, libestr, liblognorm, even if they are conifgured without debug, has "-g" added to CFLAGS. Regards. From joe at joetify.com Tue Dec 14 01:44:33 2010 From: joe at joetify.com (Joe Williams) Date: Mon, 13 Dec 2010 16:44:33 -0800 Subject: [rsyslog] setting the hostname In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> Message-ID: <457CB9A9-D985-42D9-B463-026105D7BFBE@joetify.com> David, Thanks but that won't work. I want to enforce the hostname that rsyslog is started with in case the hostname changes at some point. -Joe On Dec 13, 2010, at 1:20 PM, david at lang.hm wrote: > On Mon, 13 Dec 2010, Rainer Gerhards wrote: > >> Unfortunately, this is not possible... > > however, what you can do is to change the template to put any string you want instead of %hostname%, that will have the same effect. > > David Lang > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Joe Williams >>> Sent: Monday, December 13, 2010 9:04 PM >>> To: rsyslog-users >>> Subject: [rsyslog] setting the hostname >>> >>> >>> List, >>> >>> I would like to statically set the hostname (don't use the actual >>> hostname of the system) that rsyslogd uses for a node in the >>> configuration or as an argument to the rsyslogd command on that node. I >>> didn't immediately see a way to do this in the docs, is this possible? >>> >>> Thanks. >>> >>> -Joe >>> >>> >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From joe at joetify.com Tue Dec 14 01:45:03 2010 From: joe at joetify.com (Joe Williams) Date: Mon, 13 Dec 2010 16:45:03 -0800 Subject: [rsyslog] setting the hostname In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> Message-ID: <944DF08F-4032-49BA-8E30-0A63739C3C2F@joetify.com> Rainer, Is this something that could be patched? If it's not too hard I would be willing to give it a shot. -Joe On Dec 13, 2010, at 1:02 PM, Rainer Gerhards wrote: > Unfortunately, this is not possible... > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Joe Williams >> Sent: Monday, December 13, 2010 9:04 PM >> To: rsyslog-users >> Subject: [rsyslog] setting the hostname >> >> >> List, >> >> I would like to statically set the hostname (don't use the actual >> hostname of the system) that rsyslogd uses for a node in the >> configuration or as an argument to the rsyslogd command on that node. I >> didn't immediately see a way to do this in the docs, is this possible? >> >> Thanks. >> >> -Joe >> >> >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From david at lang.hm Tue Dec 14 01:49:27 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 16:49:27 -0800 (PST) Subject: [rsyslog] setting the hostname In-Reply-To: <457CB9A9-D985-42D9-B463-026105D7BFBE@joetify.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> <457CB9A9-D985-42D9-B463-026105D7BFBE@joetify.com> Message-ID: I'm pretty sure that rsyslog only looks up the hostname at startup time. are you seeing something in testing that indicates that it's looking up the hostname repeatedly? If this is happening, it should be changed to only look it up once, just from a performance point of view, but I sure don't remember seeing anything like that in my performance tests. David Lang On Mon, 13 Dec 2010, Joe Williams wrote: > David, > > Thanks but that won't work. I want to enforce the hostname that rsyslog is started with in case the hostname changes at some point. > > -Joe > > > > On Dec 13, 2010, at 1:20 PM, david at lang.hm wrote: > >> On Mon, 13 Dec 2010, Rainer Gerhards wrote: >> >>> Unfortunately, this is not possible... >> >> however, what you can do is to change the template to put any string you want instead of %hostname%, that will have the same effect. >> >> David Lang >> >>> Rainer >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Joe Williams >>>> Sent: Monday, December 13, 2010 9:04 PM >>>> To: rsyslog-users >>>> Subject: [rsyslog] setting the hostname >>>> >>>> >>>> List, >>>> >>>> I would like to statically set the hostname (don't use the actual >>>> hostname of the system) that rsyslogd uses for a node in the >>>> configuration or as an argument to the rsyslogd command on that node. I >>>> didn't immediately see a way to do this in the docs, is this possible? >>>> >>>> Thanks. >>>> >>>> -Joe >>>> >>>> >>>> >>>> Name: Joseph A. Williams >>>> Email: joe at joetify.com >>>> Blog: http://www.joeandmotorboat.com/ >>>> Twitter: http://twitter.com/williamsjoe >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Tue Dec 14 04:40:55 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 19:40:55 -0800 (PST) Subject: [rsyslog] imfile paragraph patch Message-ID: This is a first cut of a modification to imfile to let it read multi-line files. As-is, this should have no effect on a system as it hard-codes the mode to reading single lines (I really don't understand how to set a config variable, but for someone who does, it should be simple to replace the '0' in imfile.c with the value of the config file) With this config option change, it should be possible to real logfiles that have blank lines between multi-line log entries and have those log entries treated as a single line. I also have code in place (but disabled) to try and deal with the more complicated layout where all lines after the first one are indented if they are part of the same log entry. The problem I have is that when I discover that I have finished reading a log entry I have already read the first character of the next log entry. This extra character needs to be put pack into the input buffer, but I don't know if that is possible or not. If this isn't the case, I need a function that will let me peek at the next character in the input buffer and make my decision based on that. This compiles, but I have not tested it anywhere yet. with the hardcoded mode 0 for ('LF termination), there should be no change other than an extra test against a constant for each character read from a file. David Lang -------------- next part -------------- A non-text attachment was scrubbed... Name: imfile_paragraph.patch Type: text/x-diff Size: 4417 bytes Desc: URL: From david at lang.hm Tue Dec 14 04:57:54 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 19:57:54 -0800 (PST) Subject: [rsyslog] imfile paragraph patch In-Reply-To: References: Message-ID: I discovered UnreadChar and so now mode 2 (indented follow-up lines) has a chance of working. again compile tested (and visually code-reviewed by someone else), but not executed. David Lang On Mon, 13 Dec 2010, david at lang.hm wrote: > This is a first cut of a modification to imfile to let it read multi-line > files. > > As-is, this should have no effect on a system as it hard-codes the mode to > reading single lines (I really don't understand how to set a config variable, > but for someone who does, it should be simple to replace the '0' in imfile.c > with the value of the config file) > > With this config option change, it should be possible to real logfiles that > have blank lines between multi-line log entries and have those log entries > treated as a single line. > > I also have code in place (but disabled) to try and deal with the more > complicated layout where all lines after the first one are indented if they > are part of the same log entry. The problem I have is that when I discover > that I have finished reading a log entry I have already read the first > character of the next log entry. This extra character needs to be put pack > into the input buffer, but I don't know if that is possible or not. If this > isn't the case, I need a function that will let me peek at the next character > in the input buffer and make my decision based on that. > > This compiles, but I have not tested it anywhere yet. with the hardcoded mode > 0 for ('LF termination), there should be no change other than an extra test > against a constant for each character read from a file. > > David Lang -------------- next part -------------- A non-text attachment was scrubbed... Name: imfile_paragraph.patch Type: text/x-diff Size: 4411 bytes Desc: URL: From david at lang.hm Tue Dec 14 05:26:35 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 20:26:35 -0800 (PST) Subject: [rsyslog] question about the parser interface. Message-ID: Is it possible for a parser to just modify the input string and then let it fall through for another parser to handle the modified string? I have two rather simple parsers I want to write that fall in this category 1. Cisco with name resolution A cisco without name resolution turned on logs timestamp IPaddr %tag msg a cisco with name resolution turned on logs timestamp name : %tag msg I want to detect the bare : in the syslog field followed by the % at the start of the next tag, and if I find them, just memmove everything up (so that the % ends up where the : was, shortening the string by two characters), then let if fall through for normal processing. 2. AIX forwarding messages AIX defaults to messages in the format timestamp Message Forwarded From hostname syslogtag msg I want to look for 'Message Forwarded From' starting in the hostname field, and if I find them, memmove everything up so that the hostname is in the right place, and again let everything fall through to the normal parser for handling. I really don't want to have to duplicate the normal parser in each of these parsers as they are just (almost) trivial cleanups of the log message before it's handled normally. David Lang From david at lang.hm Tue Dec 14 05:46:28 2010 From: david at lang.hm (david at lang.hm) Date: Mon, 13 Dec 2010 20:46:28 -0800 (PST) Subject: [rsyslog] question about the parser interface. In-Reply-To: References: Message-ID: If the message can be modified by the parser, I think this is appriximatly what is needed for the cisco problem. I'm not sure exactly what to do to tie it in to the build system, so this one isn't even compile tested. David Lang On Mon, 13 Dec 2010, david at lang.hm wrote: > Date: Mon, 13 Dec 2010 20:26:35 -0800 (PST) > From: david at lang.hm > Reply-To: rsyslog-users > To: rsyslog-users > Subject: [rsyslog] question about the parser interface. > > Is it possible for a parser to just modify the input string and then let it > fall through for another parser to handle the modified string? > > I have two rather simple parsers I want to write that fall in this category > > 1. Cisco with name resolution > > A cisco without name resolution turned on logs > timestamp IPaddr %tag msg > > a cisco with name resolution turned on logs > timestamp name : %tag msg > > I want to detect the bare : in the syslog field followed by the % at the > start of the next tag, and if I find them, just memmove everything up (so > that the % ends up where the : was, shortening the string by two characters), > then let if fall through for normal processing. > > 2. AIX forwarding messages > > AIX defaults to messages in the format > > timestamp Message Forwarded From hostname syslogtag msg > > I want to look for 'Message Forwarded From' starting in the hostname field, > and if I find them, memmove everything up so that the hostname is in the > right place, and again let everything fall through to the normal parser for > handling. > > I really don't want to have to duplicate the normal parser in each of these > parsers as they are just (almost) trivial cleanups of the log message before > it's handled normally. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -------------- next part -------------- A non-text attachment was scrubbed... Name: pmcisconames.patch Type: text/x-diff Size: 5784 bytes Desc: URL: From joe at joetify.com Tue Dec 14 06:10:45 2010 From: joe at joetify.com (Joe Williams) Date: Mon, 13 Dec 2010 21:10:45 -0800 Subject: [rsyslog] setting the hostname In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> <457CB9A9-D985-42D9-B463-026105D7BFBE@joetify.com> Message-ID: <00BE0D2E-A04E-4739-B5A5-868B72E7E91E@joetify.com> On Dec 13, 2010, at 4:49 PM, david at lang.hm wrote: > I'm pretty sure that rsyslog only looks up the hostname at startup time. are you seeing something in testing that indicates that it's looking up the hostname repeatedly? > > If this is happening, it should be changed to only look it up once, just from a performance point of view, but I sure don't remember seeing anything like that in my performance tests. No, it's a funny edge case on EC2 (and likely elsewhere) where the hostname is set by DHCP. If one reboots and rsyslog starts before the hostname has been set it caches the wrong one. My solution in the past (for other applications) has been to set the hostname statically in the configuration or as a command argument of the application. Ideally I would be able to override the hostname lookup in rsyslog with my own value, if it only gets set once I can't imagine the patch being too difficult. I'll take a look in the morning but if anyone has any pointers/suggestions please chime in. -Joe > On Mon, 13 Dec 2010, Joe Williams wrote: > >> David, >> >> Thanks but that won't work. I want to enforce the hostname that rsyslog is started with in case the hostname changes at some point. >> >> -Joe >> >> >> >> On Dec 13, 2010, at 1:20 PM, david at lang.hm wrote: >> >>> On Mon, 13 Dec 2010, Rainer Gerhards wrote: >>> >>>> Unfortunately, this is not possible... >>> >>> however, what you can do is to change the template to put any string you want instead of %hostname%, that will have the same effect. >>> >>> David Lang >>> >>>> Rainer >>>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Joe Williams >>>>> Sent: Monday, December 13, 2010 9:04 PM >>>>> To: rsyslog-users >>>>> Subject: [rsyslog] setting the hostname >>>>> >>>>> >>>>> List, >>>>> >>>>> I would like to statically set the hostname (don't use the actual >>>>> hostname of the system) that rsyslogd uses for a node in the >>>>> configuration or as an argument to the rsyslogd command on that node. I >>>>> didn't immediately see a way to do this in the docs, is this possible? >>>>> >>>>> Thanks. >>>>> >>>>> -Joe >>>>> >>>>> >>>>> >>>>> Name: Joseph A. Williams >>>>> Email: joe at joetify.com >>>>> Blog: http://www.joeandmotorboat.com/ >>>>> Twitter: http://twitter.com/williamsjoe >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From david at lang.hm Tue Dec 14 11:42:22 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 14 Dec 2010 02:42:22 -0800 (PST) Subject: [rsyslog] setting the hostname In-Reply-To: <00BE0D2E-A04E-4739-B5A5-868B72E7E91E@joetify.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> <457CB9A9-D985-42D9-B463-026105D7BFBE@joetify.com> <00BE0D2E-A04E-4739-B5A5-868B72E7E91E@joetify.com> Message-ID: Ok, that sounds like rsyslog is looking up the hostnameas it starts up, and then when DHCP changes the name later it keeps using the old one. if you are happy having it in a config file, then you can hard set it in the template in the config file. This seems like it would be effectivly the same thing as the config option you are talking about. or you could configure your DHCP client to restart rsyslog when it gets a new hostname. (or just not have rsyslog started before the network is up, kernel messages would still be logged) David Lang On Mon, 13 Dec 2010, Joe Williams wrote: > On Dec 13, 2010, at 4:49 PM, david at lang.hm wrote: > >> I'm pretty sure that rsyslog only looks up the hostname at startup time. are you seeing something in testing that indicates that it's looking up the hostname repeatedly? >> >> If this is happening, it should be changed to only look it up once, just from a performance point of view, but I sure don't remember seeing anything like that in my performance tests. > > > No, it's a funny edge case on EC2 (and likely elsewhere) where the hostname is set by DHCP. If one reboots and rsyslog starts before the hostname has been set it caches the wrong one. My solution in the past (for other applications) has been to set the hostname statically in the configuration or as a command argument of the application. Ideally I would be able to override the hostname lookup in rsyslog with my own value, if it only gets set once I can't imagine the patch being too difficult. I'll take a look in the morning but if anyone has any pointers/suggestions please chime in. > > -Joe > > > >> On Mon, 13 Dec 2010, Joe Williams wrote: >> >>> David, >>> >>> Thanks but that won't work. I want to enforce the hostname that rsyslog is started with in case the hostname changes at some point. >>> >>> -Joe >>> >>> >>> >>> On Dec 13, 2010, at 1:20 PM, david at lang.hm wrote: >>> >>>> On Mon, 13 Dec 2010, Rainer Gerhards wrote: >>>> >>>>> Unfortunately, this is not possible... >>>> >>>> however, what you can do is to change the template to put any string you want instead of %hostname%, that will have the same effect. >>>> >>>> David Lang >>>> >>>>> Rainer >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Joe Williams >>>>>> Sent: Monday, December 13, 2010 9:04 PM >>>>>> To: rsyslog-users >>>>>> Subject: [rsyslog] setting the hostname >>>>>> >>>>>> >>>>>> List, >>>>>> >>>>>> I would like to statically set the hostname (don't use the actual >>>>>> hostname of the system) that rsyslogd uses for a node in the >>>>>> configuration or as an argument to the rsyslogd command on that node. I >>>>>> didn't immediately see a way to do this in the docs, is this possible? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> -Joe >>>>>> >>>>>> >>>>>> >>>>>> Name: Joseph A. Williams >>>>>> Email: joe at joetify.com >>>>>> Blog: http://www.joeandmotorboat.com/ >>>>>> Twitter: http://twitter.com/williamsjoe >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From sivan at omniqueue.com Tue Dec 14 11:48:54 2010 From: sivan at omniqueue.com (Sivan Greenberg) Date: Tue, 14 Dec 2010 12:48:54 +0200 Subject: [rsyslog] setting the hostname In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA71DD8BE@GRFEXC.intern.adiscon.com> Message-ID: This works well for me, I am using a custom forwarding template that sets up the hostname per each host. -Sivan On Mon, Dec 13, 2010 at 11:20 PM, wrote: > On Mon, 13 Dec 2010, Rainer Gerhards wrote: > >> Unfortunately, this is not possible... > > however, what you can do is to change the template to put any string you > want instead of %hostname%, that will have the same effect. > > David Lang > >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Joe Williams >>> Sent: Monday, December 13, 2010 9:04 PM >>> To: rsyslog-users >>> Subject: [rsyslog] setting the hostname >>> >>> >>> List, >>> >>> I would like to statically set the hostname (don't use the actual >>> hostname of the system) that rsyslogd uses for a node in the >>> configuration or as an argument to the rsyslogd command on that node. I >>> didn't immediately see a way to do this in the docs, is this possible? >>> >>> Thanks. >>> >>> -Joe >>> >>> >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From marcin at mejor.pl Tue Dec 14 12:04:03 2010 From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=) Date: Tue, 14 Dec 2010 12:04:03 +0100 Subject: [rsyslog] liblognorm, libee and libestr has been released In-Reply-To: <4D065BBA.2060608@mejor.pl> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> <4D065BBA.2060608@mejor.pl> Message-ID: <4D074F23.2040202@mejor.pl> W dniu 13.12.2010 18:45, Marcin Miros?aw pisze: > I'll fill bug in gentoo bugzilla in one,two days. http://bugs.gentoo.org/show_bug.cgi?id=348694 http://bugs.gentoo.org/show_bug.cgi?id=348696 http://bugs.gentoo.org/show_bug.cgi?id=348698 -- www: http://blog.mejor.pl/ From dave at fly.srk.fer.hr Wed Dec 15 15:13:40 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 15 Dec 2010 15:13:40 +0100 Subject: [rsyslog] Losing UDP packages Message-ID: <20101215141340.GA6512@fly.srk.fer.hr> I have rsyslog 5.6.2 which is a destination for a lot of traffic which is currently using UDP as a transport protocol. We intend to switch some sources to TCP, but UDP will still be used for a large chunk of data and I can't do anything about that. Current tests show that we lose 100-400 lines in every 100000 sent. That isn't much and we could live with it, but I'd like to do something about it, nevertheless. My guess is that the kernel buffer fills up, rsyslog doesn't manage to make room fast enough and then the kernel drops packages. I'm on CentOS 5.5. Kernel log shows lines like: Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service Curiously, the times do not match the ones for which I'm certain that the package loss occured. But even so, the log shows that packet drop is occuring on the system. As far as I know, two things can be done. 1. Increase UDP receive buffer size. That should allow the system to handle the bigger spike load than it's handling now, but that's about it. There will be a spike load which is too large for any given buffer size. I didn't manage to find a configuration option for this. I can increase the system default, but I dislike doing that because it affects all sockets on the system. If rsyslog really doesn't have this configuration option, I'm willing to write a patch. 2. Put a thread that handles UDP socket in real-time class (or the closest equivalent that a given OS has). The POSIX way for this is to use sched_setscheduler() with SCHED_FIFO. There could be other non-portable interfaces of interest (like priocntl() on Solaris), but I won't discuss them right now. The patch should add two configuration parameters: * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's call it $QueueThreadSchedulingPolicy (although I'd like a shorter name) * and another to define sched_priority, let's call it $QueueThreadSchedulingPriority In case of SCHED_SPORADIC (and maybe other non-portable policies on some operating systems) there are additional parameters that can be set, so maybe the second parameter would turn out to be the string with several numbers that needs to be parsed. Or there could be additional configuration options introduced. I don't have a position about that and would like to follow the existing rsyslog policy regarding configuration options. Except that I don't know what that policy is, so I'd appreciate if someone can enlighten me. Anyway, the idea is to have a high priority thread (or a small number of threads) sitting on a UDP socket whose sole purpose in life is to transfer the messages from UDP socket to another queue(s), as fast as possible. By default (if $QueueThreadSchedulingPolicy is defined), the number of threads put in that scheduling policy can be $QueueWorkerThreads, ie. the minimum number of worker threads. There could be additional parameter to define the number of such threads, but I'm not sure how to go about it. There could be no imposed limit, so if an administator puts 128 in the config file, 128 threads it is. But this is potentialy dangerous. Or there could be rsyslog imposed restriction that the number of such threads cannot exceed the number defined in $QueueWorkerThreads, which is less dangerous. I'm personally leaning to the second option. Now, this whole thing is dangerous and must be implemented and used with care and understanding. Currently I don't understand rsyslog queues well enough, which is why I'm writing this mail. The UDP thread in FIFO class should be used just to pick the packet from the socket and place it on another action queue. Let's say we have the following configuration: $Ruleset indata $RulesetCreateMainQueue on $UDPServerRun 514 $InputUDPServerBindRuleset indata $MainMsgQueueThreadSchedulingPolicy fifo $MainMsgQueueThreadSchedulingPriority 10 $ActionName local1 $ActionQueueType LinkedList $ActionQueueFileName local1 $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 8 local1.* |/path/to/fifo & /path/to/file If I understand things correctly, there are two queues here. The first one is ruleset indata queue with the UDP input module and the second one is the action queue for the destinations (FIFO and file). I'm not sure about the following: 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", are $MainMsgQueue* directives applied to that ruleset queue? It's not very intuitive. 2. Say we add a TCP listener (or any other input module) to the above configuration and we want UDP processing thread(s) to have SCHED_FIFO priority, but other input handling threads to stay at the default priority. I'm not sure if that can be specified with one ruleset. If two rulesets must be specified, then I suppose the action directives must be duplicated for each ruleset, which isn't very readable and maintenance-friendly. (This is without taking omruleset plugin into consideration). 3. The action queue is disk assisted. Which thread writes to disk when the need arises? The one from the ruleset indata queue or the one from the action queue? -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Wed Dec 15 15:21:54 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Dec 2010 15:21:54 +0100 Subject: [rsyslog] Losing UDP packages References: <20101215141340.GA6512@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8D6@GRFEXC.intern.adiscon.com> Sorry fort he sluggish replies -- I will leave for xmas break in about a week and have a couple of things to finish... Some quicke notes below: > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Wednesday, December 15, 2010 3:14 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Losing UDP packages > > I have rsyslog 5.6.2 which is a destination for a lot of traffic which > is > currently using UDP as a transport protocol. We intend to switch some > sources to TCP, but UDP will still be used for a large chunk of data > and I > can't do anything about that. > > Current tests show that we lose 100-400 lines in every 100000 sent. > That > isn't much and we could live with it, but I'd like to do something > about > it, nevertheless. > > My guess is that the kernel buffer fills up, rsyslog doesn't manage to > make room fast enough and then the kernel drops packages. I'm on CentOS > 5.5. Kernel log shows lines like: > > Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service > Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service > > Curiously, the times do not match the ones for which I'm certain that > the > package loss occured. But even so, the log shows that packet drop is > occuring on the system. > > As far as I know, two things can be done. > > 1. Increase UDP receive buffer size. That should allow the system to > handle the bigger spike load than it's handling now, but that's about > it. > There will be a spike load which is too large for any given buffer > size. > > I didn't manage to find a configuration option for this. I can increase > the system default, but I dislike doing that because it affects all > sockets on the system. If rsyslog really doesn't have this > configuration > option, I'm willing to write a patch. I doesn't do that right now. You need to look at imudp. > > 2. Put a thread that handles UDP socket in real-time class (or the > closest > equivalent that a given OS has). The POSIX way for this is to use > sched_setscheduler() with SCHED_FIFO. There could be other non-portable > interfaces of interest (like priocntl() on Solaris), but I won't > discuss > them right now. > > The patch should add two configuration parameters: > * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's > call it $QueueThreadSchedulingPolicy (although I'd like a > shorter name) > * and another to define sched_priority, let's call it > $QueueThreadSchedulingPriority > > In case of SCHED_SPORADIC (and maybe other non-portable policies on > some > operating systems) there are additional parameters that can be set, so > maybe the second parameter would turn out to be the string with several > numbers that needs to be parsed. Or there could be additional > configuration options introduced. I don't have a position about that > and > would like to follow the existing rsyslog policy regarding > configuration > options. Except that I don't know what that policy is, so I'd > appreciate > if someone can enlighten me. UDP is not handled at the queue level. It is soley the thread that imudp runs on. Each input has its own thread, so you can focus on imudp. > > Anyway, the idea is to have a high priority thread (or a small number > of > threads) sitting on a UDP socket whose sole purpose in life is to > transfer the messages from UDP socket to another queue(s), as fast as > possible. That's what imudp does. I suggest to also have a look at: http://www.gerhards.net/download/LinuxKongress2010rsyslog.pdf RAiner > default (if $QueueThreadSchedulingPolicy is > defined), > the number of threads put in that scheduling policy can be > $QueueWorkerThreads, ie. the minimum number of worker threads. > There could be additional parameter to define the number of such > threads, > but I'm not sure how to go about it. > > There could be no imposed limit, so if an administator puts 128 in the > config file, 128 threads it is. But this is potentialy dangerous. Or > there > could be rsyslog imposed restriction that the number of such threads > cannot > exceed the number defined in $QueueWorkerThreads, which is less > dangerous. I'm personally leaning to the second option. > > > Now, this whole thing is dangerous and must be implemented and used > with > care and understanding. Currently I don't understand rsyslog queues > well > enough, which is why I'm writing this mail. The UDP thread in FIFO > class > should be used just to pick the packet from the socket and place it on > another action queue. Let's say we have the following configuration: > > $Ruleset indata > $RulesetCreateMainQueue on > > $UDPServerRun 514 > $InputUDPServerBindRuleset indata > > $MainMsgQueueThreadSchedulingPolicy fifo > $MainMsgQueueThreadSchedulingPriority 10 > > $ActionName local1 > $ActionQueueType LinkedList > $ActionQueueFileName local1 > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 8 > > local1.* |/path/to/fifo > & /path/to/file > > > If I understand things correctly, there are two queues here. The first > one > is ruleset indata queue with the UDP input module and the second one is > the action queue for the destinations (FIFO and file). I'm not sure > about > the following: > > 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", > are $MainMsgQueue* directives applied to that ruleset queue? It's not > very > intuitive. > > 2. Say we add a TCP listener (or any other input module) to the above > configuration and we want UDP processing thread(s) to have SCHED_FIFO > priority, but other input handling threads to stay at the default > priority. I'm not sure if that can be specified with one ruleset. If > two > rulesets must be specified, then I suppose the action directives must > be > duplicated for each ruleset, which isn't very readable and > maintenance-friendly. (This is without taking omruleset plugin into > consideration). > > 3. The action queue is disk assisted. Which thread writes to disk when > the > need arises? The one from the ruleset indata queue or the one from the > action queue? > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From fabien.bagard at parrot.com Wed Dec 15 15:44:22 2010 From: fabien.bagard at parrot.com (Fabien Bagard) Date: Wed, 15 Dec 2010 15:44:22 +0100 Subject: [rsyslog] Certificate not trusted problem Message-ID: <4D08D446.6080205@parrot.com> Hi ! I tried to set up a centralized rsyslog server, with cyphered connections. Both server & client certificates were signed by the same CA. The problem is my certificates are not trusted : ==> logger/rsyslogd.log <== 2010-12-15T15:17:01.685026+01:00 logging-server rsyslogd: not permitted to talk to peer, certificate invalid: signer is not a CA 2010-12-15T15:17:01.685895+01:00 logging-server rsyslogd: invalid cert info: peer provided 1 certificate(s). Certificate 1 info: certificate valid from DATE to DATE; Certificate public key: RSA; DN: C=FR,ST=France,L=Paris,O=Parrot S.A.,OU=IT Department,CN=fr-b-701-006.parrot.biz,EMAIL=fabien.bagard_AT_parrot.com; Issuer DN: C=FR,ST=France,L=Paris,O=Parrot S.A. Root CA,OU=IT Department,CN=Parrot S.A. Root CA,EMAIL=root_AT_parrot.com; ==> logger/rsyslogd-2089.log <== 2010-12-15T15:17:01.725557+01:00 logger rsyslogd-2089: netstream session 0x90b55b8 will be closed due to error [try http://www.rsyslog.com/e/2089 ] When checking my certificates with gnutls-cli, I get : ... Other Information: MD5 fingerprint: 9dbbdeea9cee323e06553e632250d669 SHA-1 fingerprint: 008247f9dcc86bbfe958ef7a1b12ed9917320ad2 Public Key Id: 29dcaddda6746140e556804d7ea4af8fd47e62a5 - The hostname in the certificate matches 'logging-server.parrot.biz'. - Peer's certificate issuer is unknown - Peer's certificate is NOT trusted - Version: TLS1.1 - Key Exchange: DHE-RSA - Cipher: AES-128-CBC - MAC: SHA1 - Compression: NULL - Session ID: 61:2D:87:98:63:9F:14:EB:11:BB:7A:9F:1B:53:FC:DE:80:E2:4C:7C:C7:3F:15:9B:8A:18:25:6D:35:40:B1:4E *** Verifying server certificate failed... Any hints on what's going wrong ? By the way, I'm pretty sure my certificates are good enough because certificates were already signed with this CA. Thanks a lot, -- Fabien Bagard IT Department tel + 33 (0)1 48 03 60 40 -------------------------------------------------------------------------------- Parrot SA 174, Quai de Jemmapes | 75010 Paris - France tel + 33 (0)1 48 03 60 60 | fax + 33 (0)1 48 03 70 08 http://www.parrot.com -------------------------------------------------------------------------------- This e-mail message and any attached document(s) are for the sole use of the intended recipient(s)and may contain confidential and legally privileged information. Any unauthorized review, copy, use and/or disclosure is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original. From raubvogel at gmail.com Wed Dec 15 18:14:52 2010 From: raubvogel at gmail.com (Mauricio Tavares) Date: Wed, 15 Dec 2010 12:14:52 -0500 Subject: [rsyslog] no local0/local1 in syslog.log Message-ID: <4D08F78C.8010609@gmail.com> If I do not want to have the facilities local0 and local1 written in syslog.log, would *.*;auth,authpriv.none;\ local0.*;local1.* -/var/log/syslog/syslog.log do the trick? From rgerhards at hq.adiscon.com Wed Dec 15 18:16:16 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Dec 2010 18:16:16 +0100 Subject: [rsyslog] no local0/local1 in syslog.log References: <4D08F78C.8010609@gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8DB@GRFEXC.intern.adiscon.com> Local0.none;local1.none should work IIRC > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mauricio Tavares > Sent: Wednesday, December 15, 2010 6:15 PM > To: rsyslog-users > Subject: [rsyslog] no local0/local1 in syslog.log > > If I do not want to have the facilities local0 and local1 written > in > syslog.log, would > > *.*;auth,authpriv.none;\ > local0.*;local1.* -/var/log/syslog/syslog.log > > do the trick? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Dec 15 19:05:45 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 15 Dec 2010 10:05:45 -0800 (PST) Subject: [rsyslog] Losing UDP packages In-Reply-To: <20101215141340.GA6512@fly.srk.fer.hr> References: <20101215141340.GA6512@fly.srk.fer.hr> Message-ID: I haven't done tests on a recent version, butabout ayearago I didextensive testing with UDP (early 5.x days) make sure you disable name lookups. I found that I was able to keep up with ~gig-E wire speed (380K logs/sec with ~250 byte log messages) as long as the peak did not exceed the memory queue size. At the time, rsyslog would process messages from the queue at between 30K and 80K messages/sec (depending on the configuration). I was testing this by using tcpreplay to send the packets, 1,000,000 packets per test. In about 100 such tests I did not experiance _any_ lost messages. I do have network receive buffers set to larger than normal values. So I don't think that the problem is on the receiver side. Rainer has been doing some tests recently on the sending side that may indicate some issues. I haven't had time to get my test systems setup again to try more tests. David Lang On Wed, 15 Dec 2010, Dra?en Ka?ar wrote: > I have rsyslog 5.6.2 which is a destination for a lot of traffic which is > currently using UDP as a transport protocol. We intend to switch some > sources to TCP, but UDP will still be used for a large chunk of data and I > can't do anything about that. > > Current tests show that we lose 100-400 lines in every 100000 sent. That > isn't much and we could live with it, but I'd like to do something about > it, nevertheless. > > My guess is that the kernel buffer fills up, rsyslog doesn't manage to > make room fast enough and then the kernel drops packages. I'm on CentOS > 5.5. Kernel log shows lines like: > > Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service > Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service > > Curiously, the times do not match the ones for which I'm certain that the > package loss occured. But even so, the log shows that packet drop is > occuring on the system. > > As far as I know, two things can be done. > > 1. Increase UDP receive buffer size. That should allow the system to > handle the bigger spike load than it's handling now, but that's about it. > There will be a spike load which is too large for any given buffer size. > > I didn't manage to find a configuration option for this. I can increase > the system default, but I dislike doing that because it affects all > sockets on the system. If rsyslog really doesn't have this configuration > option, I'm willing to write a patch. > > 2. Put a thread that handles UDP socket in real-time class (or the closest > equivalent that a given OS has). The POSIX way for this is to use > sched_setscheduler() with SCHED_FIFO. There could be other non-portable > interfaces of interest (like priocntl() on Solaris), but I won't discuss > them right now. > > The patch should add two configuration parameters: > * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's > call it $QueueThreadSchedulingPolicy (although I'd like a > shorter name) > * and another to define sched_priority, let's call it > $QueueThreadSchedulingPriority > > In case of SCHED_SPORADIC (and maybe other non-portable policies on some > operating systems) there are additional parameters that can be set, so > maybe the second parameter would turn out to be the string with several > numbers that needs to be parsed. Or there could be additional > configuration options introduced. I don't have a position about that and > would like to follow the existing rsyslog policy regarding configuration > options. Except that I don't know what that policy is, so I'd appreciate > if someone can enlighten me. > > Anyway, the idea is to have a high priority thread (or a small number of > threads) sitting on a UDP socket whose sole purpose in life is to > transfer the messages from UDP socket to another queue(s), as fast as > possible. By default (if $QueueThreadSchedulingPolicy is defined), > the number of threads put in that scheduling policy can be > $QueueWorkerThreads, ie. the minimum number of worker threads. > There could be additional parameter to define the number of such threads, > but I'm not sure how to go about it. > > There could be no imposed limit, so if an administator puts 128 in the > config file, 128 threads it is. But this is potentialy dangerous. Or there > could be rsyslog imposed restriction that the number of such threads cannot > exceed the number defined in $QueueWorkerThreads, which is less > dangerous. I'm personally leaning to the second option. > > > Now, this whole thing is dangerous and must be implemented and used with > care and understanding. Currently I don't understand rsyslog queues well > enough, which is why I'm writing this mail. The UDP thread in FIFO class > should be used just to pick the packet from the socket and place it on > another action queue. Let's say we have the following configuration: > > $Ruleset indata > $RulesetCreateMainQueue on > > $UDPServerRun 514 > $InputUDPServerBindRuleset indata > > $MainMsgQueueThreadSchedulingPolicy fifo > $MainMsgQueueThreadSchedulingPriority 10 > > $ActionName local1 > $ActionQueueType LinkedList > $ActionQueueFileName local1 > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 8 > > local1.* |/path/to/fifo > & /path/to/file > > > If I understand things correctly, there are two queues here. The first one > is ruleset indata queue with the UDP input module and the second one is > the action queue for the destinations (FIFO and file). I'm not sure about > the following: > > 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", > are $MainMsgQueue* directives applied to that ruleset queue? It's not very > intuitive. > > 2. Say we add a TCP listener (or any other input module) to the above > configuration and we want UDP processing thread(s) to have SCHED_FIFO > priority, but other input handling threads to stay at the default > priority. I'm not sure if that can be specified with one ruleset. If two > rulesets must be specified, then I suppose the action directives must be > duplicated for each ruleset, which isn't very readable and > maintenance-friendly. (This is without taking omruleset plugin into > consideration). > > 3. The action queue is disk assisted. Which thread writes to disk when the > need arises? The one from the ruleset indata queue or the one from the > action queue? > > From champ at softwink.com Wed Dec 15 19:59:08 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 15 Dec 2010 13:59:08 -0500 Subject: [rsyslog] liblognorm problems... In-Reply-To: <4D074F23.2040202@mejor.pl> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> <4D065BBA.2060608@mejor.pl> <4D074F23.2040202@mejor.pl> Message-ID: <20101215185908.GA20151@bundy.vistech.net> I'm obviously missing something.. Here's my brief code (if you need more info, I'll be happy to post). This is simply my PoC code. Anyways.... ----- struct ee_field *field = NULL; es_str_t *str2; ----- str = es_newStrFromCStr(buf, strlen(buf)); ln_normalize(ctx, str, &event); es_emptyStr(str); ee_fmtEventToRFC5424(event, &str); cstr = es_str2cstr(str,NULL); printf("normalized: '%s'\n", cstr); /* Yay */ /* We start breaking stuff below here */ field = ee_getEventField(event, str); str2 = ee_getFieldValueAsStr(field, 0); /* <-- Breaks here */ cstr = es_str2cstr(str2, NULL); printf("Field: %s\n", cstr); ----- Here's the output from when the PoC code is executed. This is with --enable-debug on the libraries.... ----- normalized: '[cee at 115 port="48973" ip="192.168.0.1" tag="sshd[4416\]" host="bundy" date="Dec 10 12:13:50"]' normalizer: fieldbucket.c:136: ee_getFieldValueAsStr: Assertion `field != ((void *)0)' failed. Aborted ----- I've been reading over the docs, but I'm obviously missing something really simple. It'd be my guess that ee_getEventField() is actually where my problem is. Sorry for the silly question. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Wed Dec 15 20:18:30 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 15 Dec 2010 20:18:30 +0100 Subject: [rsyslog] liblognorm problems... Message-ID: <005201cb9c8c$d9ff17ed$100013ac@intern.adiscon.com> >From phone: i think null is returned if field is not present. Must check this b4 using. ----- Urspr?ngliche Nachricht ----- Von: Champ Clark III [Softwink] Gesendet: Mittwoch, 15. Dezember 2010 19:59 An: rsyslog-users Betreff: Re: [rsyslog] liblognorm problems... I'm obviously missing something.. Here's my brief code (if you need more info, I'll be happy to post). This is simply my PoC code. Anyways.... ----- struct ee_field *field = NULL; es_str_t *str2; ----- str = es_newStrFromCStr(buf, strlen(buf)); ln_normalize(ctx, str, &event); es_emptyStr(str); ee_fmtEventToRFC5424(event, &str); cstr = es_str2cstr(str,NULL); printf("normalized: '%s'\n", cstr); /* Yay */ /* We start breaking stuff below here */ field = ee_getEventField(event, str); str2 = ee_getFieldValueAsStr(field, 0); /* <-- Breaks here */ cstr = es_str2cstr(str2, NULL); printf("Field: %s\n", cstr); ----- Here's the output from when the PoC code is executed. This is with --enable-debug on the libraries.... ----- normalized: '[cee at 115 port="48973" ip="192.168.0.1" tag="sshd[4416\]" host="bundy" date="Dec 10 12:13:50"]' normalizer: fieldbucket.c:136: ee_getFieldValueAsStr: Assertion `field != ((void *)0)' failed. Aborted ----- I've been reading over the docs, but I'm obviously missing something really simple. It'd be my guess that ee_getEventField() is actually where my problem is. Sorry for the silly question. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. From champ at softwink.com Wed Dec 15 20:23:42 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 15 Dec 2010 14:23:42 -0500 Subject: [rsyslog] liblognorm rulebase questions.. In-Reply-To: <20101215185908.GA20151@bundy.vistech.net> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> <4D065BBA.2060608@mejor.pl> <4D074F23.2040202@mejor.pl> <20101215185908.GA20151@bundy.vistech.net> Message-ID: <20101215192342.GB20151@bundy.vistech.net> This might be a bit pre-mature, but I thought I'd bring up. First off, I have liblognorm working in Sagan with a kludgy strtok_r() hack until I understand what I'm doing wrong with ee_getFieldValueAsStr(), etc. The way I'm doing it right now, in my little test tree of code, is to have a line like thus in the Sagan config file: normalize: cisco, /usr/local/etc/cisco.rulesbase normalize: openssh, /usr/local/etc/openssh.rulebase In my (Sagan) rule sets, at load time, if the rule has a 'normalize: cisco', then I'll call ln_loadSamples(). If we never see a 'normalize: openssh' option, then there's not much of a reason to load that liblognorm rule base. My first question is, has there been any more thought on the rulebase design? My thinking is that it'll be something along the lines described above. That is, a separate liblognorm rule base per "what your looking for"? Is this a bad assumption? My last question, is there any way to "bypass" the "prefix=" option in a liblognorm rule? In my particular case, I've already separate much of the inbound syslog message. Really, what I want is the "rule=" part of the liblognorm rule. If not, then that really leaves me a couple of options: 1. "My" rules, with a blank "prefix=" (don't like this, but works). 2. To "rebuild" the inbound message to a RFC5424 compatible type so liblognorm can deal with it as normal (not bad, just an "extra" step). Any thoughts? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From champ at softwink.com Wed Dec 15 21:07:22 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Wed, 15 Dec 2010 15:07:22 -0500 Subject: [rsyslog] liblognorm problems... In-Reply-To: <005201cb9c8c$d9ff17ed$100013ac@intern.adiscon.com> References: <005201cb9c8c$d9ff17ed$100013ac@intern.adiscon.com> Message-ID: <20101215200722.GA21847@bundy.vistech.net> On Wed, Dec 15, 2010 at 08:18:30PM +0100, Rainer Gerhards wrote: > >From phone: i think null is returned if field is not present. Must check this b4 using. Understood. This is really rough PoC code. I know I have a "ip=" field, but I'm unclear on how to "call" that from ee_getEventField(event, ??) ... I'm assuming I inform ee_getEventField to obtain the "ip" field, then use ee_getFieldValueAsStr to get the value. Correct? -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 16 08:35:27 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Dec 2010 08:35:27 +0100 Subject: [rsyslog] liblognorm rulebase questions.. References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com><4D065BBA.2060608@mejor.pl> <4D074F23.2040202@mejor.pl><20101215185908.GA20151@bundy.vistech.net> <20101215192342.GB20151@bundy.vistech.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8E1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Softwink] > Sent: Wednesday, December 15, 2010 8:24 PM > To: rsyslog-users > Subject: [rsyslog] liblognorm rulebase questions.. > > > This might be a bit pre-mature, but I thought I'd bring up. It probably is. I had hoped to have a final extensible format available now, but then we had some issues finding consensus on how tags look in CEE, and this held me. And now some expected and unexpected big bunch of work got into my way :( > > First off, I have liblognorm working in Sagan with a kludgy > strtok_r() hack until I understand what I'm doing wrong with > ee_getFieldValueAsStr(), etc. The way I'm doing it right now, in > my little test tree of code, is to have a line like thus in the > Sagan config file: > > normalize: cisco, /usr/local/etc/cisco.rulesbase > normalize: openssh, /usr/local/etc/openssh.rulebase > > In my (Sagan) rule sets, at load time, if the rule has a > 'normalize: cisco', then I'll call ln_loadSamples(). If we never see > a 'normalize: openssh' option, then there's not much of a reason to > load that liblognorm rule base. > > My first question is, has there been any more thought on > the rulebase design? My thinking is that it'll be something along the > lines described above. That is, a separate liblognorm rule base > per "what your looking for"? Is this a bad assumption? In my original thoughts, a *single* rulebase for everything makes most sense. Because this has the quickest parsing time. However, there are some practical limits on that, and it sounds natural to have separate sets for devices you can clearly differentiate. So I'd say you are not on a wrong route. > > My last question, is there any way to "bypass" the "prefix=" > option in a liblognorm rule? In my particular case, I've already > separate much of the inbound syslog message. Really, what I want > is the "rule=" part of the liblognorm rule. If not, then that really > leaves me a couple of options: "prefix" is optional. If you don't need it, just don't specify it. I introduced prefix to support using the same rule base with different, well, prefixes ;) For example, in rsyslog I have the raw message and the parsed MSG part of it. If you run the normalizer on the raw message, you need a common prefix to cover, tag, date, ... If you parse the MSG part that prefix is not present. So the core idea is to write the rules for the common format and add different prefixes as required (or not provide a prefix at all). Rainer > > 1. "My" rules, with a blank "prefix=" (don't like this, but works). > 2. To "rebuild" the inbound message to a RFC5424 compatible type so > liblognorm can deal with it as normal (not bad, just an "extra" > step). > > Any thoughts? > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. From dave at fly.srk.fer.hr Thu Dec 16 11:15:17 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 16 Dec 2010 11:15:17 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: References: <20101215141340.GA6512@fly.srk.fer.hr> Message-ID: <20101216101516.GA29280@fly.srk.fer.hr> david at lang.hm wrote: > I haven't done tests on a recent version, butabout ayearago I > didextensive testing with UDP (early 5.x days) > > make sure you disable name lookups. I did. > I found that I was able to keep up with ~gig-E wire speed (380K logs/sec > with ~250 byte log messages) as long as the peak did not exceed the > memory queue size. At the time, rsyslog would process messages from the > queue at between 30K and 80K messages/sec (depending on the > configuration). I was testing this by using tcpreplay to send the > packets, 1,000,000 packets per test. In about 100 such tests I did not > experiance _any_ lost messages. I believe it can be done without message loss. However, in my setup there are other programs on the machine which can use a lot of CPU time, so rsyslog might not be scheduled on a CPU fast enough. There is one CPU with four cores, one of them will be taken by a SCHED_FIFO process belonging to clusterware whenever it needs it and I'd like to do the same arrangement for rsyslog's UDP receiver thread. That leaves two cores for everything else, which should be perfectly fine. (There is a potential issue with memory in case the machine starts swapping, but I'm not trying to solve that right now.) I also don't have any control over the sending side, meaning it's very hard for me to figure out what the peak is (or will be, once all senders start sending). Is there any way to get some statistics from rsyslog? It would be nice if it could write some counters in a file every hour or so. For example: the number of messages received by each of the input plugins and the number of messages written by each of the output plugins. Full SNMP support would be even nicer, but that's reasonably easy to implement if all the counters exist. (There might be some issues with Net-SNMP and multithreaded programs, though.) > I do have network receive buffers set to larger than normal values. Yes, that would seem necessary. > So I don't think that the problem is on the receiver side. Well, my receiver side has other processes competing for the CPU. I think that can make a difference. > Rainer has been doing some tests recently on the sending side that may > indicate some issues. I haven't had time to get my test systems setup > again to try more tests. > > David Lang > > On Wed, 15 Dec 2010, Dra?en Ka?ar wrote: > >> I have rsyslog 5.6.2 which is a destination for a lot of traffic which is >> currently using UDP as a transport protocol. We intend to switch some >> sources to TCP, but UDP will still be used for a large chunk of data and I >> can't do anything about that. >> >> Current tests show that we lose 100-400 lines in every 100000 sent. That >> isn't much and we could live with it, but I'd like to do something about >> it, nevertheless. >> >> My guess is that the kernel buffer fills up, rsyslog doesn't manage to >> make room fast enough and then the kernel drops packages. I'm on CentOS >> 5.5. Kernel log shows lines like: >> >> Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service >> Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service >> >> Curiously, the times do not match the ones for which I'm certain that the >> package loss occured. But even so, the log shows that packet drop is >> occuring on the system. >> >> As far as I know, two things can be done. >> >> 1. Increase UDP receive buffer size. That should allow the system to >> handle the bigger spike load than it's handling now, but that's about it. >> There will be a spike load which is too large for any given buffer size. >> >> I didn't manage to find a configuration option for this. I can increase >> the system default, but I dislike doing that because it affects all >> sockets on the system. If rsyslog really doesn't have this configuration >> option, I'm willing to write a patch. >> >> 2. Put a thread that handles UDP socket in real-time class (or the closest >> equivalent that a given OS has). The POSIX way for this is to use >> sched_setscheduler() with SCHED_FIFO. There could be other non-portable >> interfaces of interest (like priocntl() on Solaris), but I won't discuss >> them right now. >> >> The patch should add two configuration parameters: >> * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's >> call it $QueueThreadSchedulingPolicy (although I'd like a >> shorter name) >> * and another to define sched_priority, let's call it >> $QueueThreadSchedulingPriority >> >> In case of SCHED_SPORADIC (and maybe other non-portable policies on some >> operating systems) there are additional parameters that can be set, so >> maybe the second parameter would turn out to be the string with several >> numbers that needs to be parsed. Or there could be additional >> configuration options introduced. I don't have a position about that and >> would like to follow the existing rsyslog policy regarding configuration >> options. Except that I don't know what that policy is, so I'd appreciate >> if someone can enlighten me. >> >> Anyway, the idea is to have a high priority thread (or a small number of >> threads) sitting on a UDP socket whose sole purpose in life is to >> transfer the messages from UDP socket to another queue(s), as fast as >> possible. By default (if $QueueThreadSchedulingPolicy is defined), >> the number of threads put in that scheduling policy can be >> $QueueWorkerThreads, ie. the minimum number of worker threads. >> There could be additional parameter to define the number of such threads, >> but I'm not sure how to go about it. >> >> There could be no imposed limit, so if an administator puts 128 in the >> config file, 128 threads it is. But this is potentialy dangerous. Or there >> could be rsyslog imposed restriction that the number of such threads cannot >> exceed the number defined in $QueueWorkerThreads, which is less >> dangerous. I'm personally leaning to the second option. >> >> >> Now, this whole thing is dangerous and must be implemented and used with >> care and understanding. Currently I don't understand rsyslog queues well >> enough, which is why I'm writing this mail. The UDP thread in FIFO class >> should be used just to pick the packet from the socket and place it on >> another action queue. Let's say we have the following configuration: >> >> $Ruleset indata >> $RulesetCreateMainQueue on >> >> $UDPServerRun 514 >> $InputUDPServerBindRuleset indata >> >> $MainMsgQueueThreadSchedulingPolicy fifo >> $MainMsgQueueThreadSchedulingPriority 10 >> >> $ActionName local1 >> $ActionQueueType LinkedList >> $ActionQueueFileName local1 >> $ActionResumeRetryCount -1 >> $ActionQueueSaveOnShutdown on >> $ActionQueueDiscardSeverity 8 >> >> local1.* |/path/to/fifo >> & /path/to/file >> >> >> If I understand things correctly, there are two queues here. The first one >> is ruleset indata queue with the UDP input module and the second one is >> the action queue for the destinations (FIFO and file). I'm not sure about >> the following: >> >> 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", >> are $MainMsgQueue* directives applied to that ruleset queue? It's not very >> intuitive. >> >> 2. Say we add a TCP listener (or any other input module) to the above >> configuration and we want UDP processing thread(s) to have SCHED_FIFO >> priority, but other input handling threads to stay at the default >> priority. I'm not sure if that can be specified with one ruleset. If two >> rulesets must be specified, then I suppose the action directives must be >> duplicated for each ruleset, which isn't very readable and >> maintenance-friendly. (This is without taking omruleset plugin into >> consideration). >> >> 3. The action queue is disk assisted. Which thread writes to disk when the >> need arises? The one from the ruleset indata queue or the one from the >> action queue? >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From mark at thermeon.com Thu Dec 16 14:03:06 2010 From: mark at thermeon.com (Mark Olliver) Date: Thu, 16 Dec 2010 13:03:06 +0000 Subject: [rsyslog] Watch for message conditions Message-ID: Hi All, I am trying to get rsyslog to alert me on certain patterns in the log. At the moment i have built a small config with the following sections However, nothing is the Alert script is not being triggered as far as i can tell. I have tested the alert script and it works ok. #/etc/rsyslog.d/55-alert.conf $template Alert, "'Rsyslog Alert' %msg%" local5.info :msg, content "warning" ^/var/c+i/bin/mail_alert Alert I am testing the above by running the following command on the cli. echo "warning: test message" | logger -t puppet -p local5.info This does correctly log into my local.5 log in /var/logs . Any ideas would be welcome. Thanks Mark From david at lang.hm Thu Dec 16 15:42:02 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 16 Dec 2010 06:42:02 -0800 (PST) Subject: [rsyslog] Losing UDP packages In-Reply-To: <20101216101516.GA29280@fly.srk.fer.hr> References: <20101215141340.GA6512@fly.srk.fer.hr> <20101216101516.GA29280@fly.srk.fer.hr> Message-ID: On Thu, 16 Dec 2010, Dra?en Ka?ar wrote: > david at lang.hm wrote: >> I haven't done tests on a recent version, butabout ayearago I >> didextensive testing with UDP (early 5.x days) >> >> make sure you disable name lookups. > > I did. > >> I found that I was able to keep up with ~gig-E wire speed (380K logs/sec >> with ~250 byte log messages) as long as the peak did not exceed the >> memory queue size. At the time, rsyslog would process messages from the >> queue at between 30K and 80K messages/sec (depending on the >> configuration). I was testing this by using tcpreplay to send the >> packets, 1,000,000 packets per test. In about 100 such tests I did not >> experiance _any_ lost messages. > > I believe it can be done without message loss. However, in my setup there > are other programs on the machine which can use a lot of CPU time, so > rsyslog might not be scheduled on a CPU fast enough. There is one CPU with > four cores, one of them will be taken by a SCHED_FIFO process belonging to > clusterware whenever it needs it and I'd like to do the same arrangement > for rsyslog's UDP receiver thread. That leaves two cores for everything > else, which should be perfectly fine. (There is a potential issue with > memory in case the machine starts swapping, but I'm not trying to solve > that right now.) this is where increasing the kernel buffers can save you by giving you more leeway in when rsyslog _needs_ cpu to avoid loss. the other thing that you can do is to boost the priority of rsyslog with nice. ideally you would do this only to the thread that is handling the UDP reception, but I don't know a good way to find the pid of that thread. on my dedicated rsyslog boxes I boost the priority by 5 or 10 to make sure that housekeeping (and reporting) processes don't block reception. > I also don't have any control over the sending side, meaning it's very > hard for me to figure out what the peak is (or will be, once all senders > start sending). make sure that you write the log with the sender's timestamp in the log message, then you can findthe peak by just using cut or awk to parse out the timestamp and run it through sort |uniq -c |sort -n |tail to find the most common 10 seconds. even if it takes rsyslog a long time to output the logs, this will still tell you the number sent (and therefor received) in a second. > Is there any way to get some statistics from rsyslog? It would be nice if > it could write some counters in a file every hour or so. For example: the > number of messages received by each of the input plugins and the number of > messages written by each of the output plugins. there is a stats module that was created fairly recently. > Full SNMP support would be even nicer, but that's reasonably easy to > implement if all the counters exist. (There might be some issues with > Net-SNMP and multithreaded programs, though.) > >> I do have network receive buffers set to larger than normal values. > > Yes, that would seem necessary. > >> So I don't think that the problem is on the receiver side. > > Well, my receiver side has other processes competing for the CPU. I think > that can make a difference. yes, if you don't get enough cpu time to process incoming logs before overflowing the kernel buffers, the logs will be lost, boosting priority and increasing buffer size will let you do this even when other things are running. one other performance tweak, there is an option to tell rsyslog not to lookup the time for every message, but only do it every N messages when there are alog of messages to process. setting this can help (the biggest win is at the beginning, 2 saves 1/2 the lookups of 1, 4 saves 3/4 the lookups, 10 saves 9/10 the lookups, etc) David Lang >> Rainer has been doing some tests recently on the sending side that may >> indicate some issues. I haven't had time to get my test systems setup >> again to try more tests. >> >> David Lang >> >> On Wed, 15 Dec 2010, Dra?en Ka?ar wrote: >> >>> I have rsyslog 5.6.2 which is a destination for a lot of traffic which is >>> currently using UDP as a transport protocol. We intend to switch some >>> sources to TCP, but UDP will still be used for a large chunk of data and I >>> can't do anything about that. >>> >>> Current tests show that we lose 100-400 lines in every 100000 sent. That >>> isn't much and we could live with it, but I'd like to do something about >>> it, nevertheless. >>> >>> My guess is that the kernel buffer fills up, rsyslog doesn't manage to >>> make room fast enough and then the kernel drops packages. I'm on CentOS >>> 5.5. Kernel log shows lines like: >>> >>> Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service >>> Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service >>> >>> Curiously, the times do not match the ones for which I'm certain that the >>> package loss occured. But even so, the log shows that packet drop is >>> occuring on the system. >>> >>> As far as I know, two things can be done. >>> >>> 1. Increase UDP receive buffer size. That should allow the system to >>> handle the bigger spike load than it's handling now, but that's about it. >>> There will be a spike load which is too large for any given buffer size. >>> >>> I didn't manage to find a configuration option for this. I can increase >>> the system default, but I dislike doing that because it affects all >>> sockets on the system. If rsyslog really doesn't have this configuration >>> option, I'm willing to write a patch. >>> >>> 2. Put a thread that handles UDP socket in real-time class (or the closest >>> equivalent that a given OS has). The POSIX way for this is to use >>> sched_setscheduler() with SCHED_FIFO. There could be other non-portable >>> interfaces of interest (like priocntl() on Solaris), but I won't discuss >>> them right now. >>> >>> The patch should add two configuration parameters: >>> * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's >>> call it $QueueThreadSchedulingPolicy (although I'd like a >>> shorter name) >>> * and another to define sched_priority, let's call it >>> $QueueThreadSchedulingPriority >>> >>> In case of SCHED_SPORADIC (and maybe other non-portable policies on some >>> operating systems) there are additional parameters that can be set, so >>> maybe the second parameter would turn out to be the string with several >>> numbers that needs to be parsed. Or there could be additional >>> configuration options introduced. I don't have a position about that and >>> would like to follow the existing rsyslog policy regarding configuration >>> options. Except that I don't know what that policy is, so I'd appreciate >>> if someone can enlighten me. >>> >>> Anyway, the idea is to have a high priority thread (or a small number of >>> threads) sitting on a UDP socket whose sole purpose in life is to >>> transfer the messages from UDP socket to another queue(s), as fast as >>> possible. By default (if $QueueThreadSchedulingPolicy is defined), >>> the number of threads put in that scheduling policy can be >>> $QueueWorkerThreads, ie. the minimum number of worker threads. >>> There could be additional parameter to define the number of such threads, >>> but I'm not sure how to go about it. >>> >>> There could be no imposed limit, so if an administator puts 128 in the >>> config file, 128 threads it is. But this is potentialy dangerous. Or there >>> could be rsyslog imposed restriction that the number of such threads cannot >>> exceed the number defined in $QueueWorkerThreads, which is less >>> dangerous. I'm personally leaning to the second option. >>> >>> >>> Now, this whole thing is dangerous and must be implemented and used with >>> care and understanding. Currently I don't understand rsyslog queues well >>> enough, which is why I'm writing this mail. The UDP thread in FIFO class >>> should be used just to pick the packet from the socket and place it on >>> another action queue. Let's say we have the following configuration: >>> >>> $Ruleset indata >>> $RulesetCreateMainQueue on >>> >>> $UDPServerRun 514 >>> $InputUDPServerBindRuleset indata >>> >>> $MainMsgQueueThreadSchedulingPolicy fifo >>> $MainMsgQueueThreadSchedulingPriority 10 >>> >>> $ActionName local1 >>> $ActionQueueType LinkedList >>> $ActionQueueFileName local1 >>> $ActionResumeRetryCount -1 >>> $ActionQueueSaveOnShutdown on >>> $ActionQueueDiscardSeverity 8 >>> >>> local1.* |/path/to/fifo >>> & /path/to/file >>> >>> >>> If I understand things correctly, there are two queues here. The first one >>> is ruleset indata queue with the UDP input module and the second one is >>> the action queue for the destinations (FIFO and file). I'm not sure about >>> the following: >>> >>> 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", >>> are $MainMsgQueue* directives applied to that ruleset queue? It's not very >>> intuitive. >>> >>> 2. Say we add a TCP listener (or any other input module) to the above >>> configuration and we want UDP processing thread(s) to have SCHED_FIFO >>> priority, but other input handling threads to stay at the default >>> priority. I'm not sure if that can be specified with one ruleset. If two >>> rulesets must be specified, then I suppose the action directives must be >>> duplicated for each ruleset, which isn't very readable and >>> maintenance-friendly. (This is without taking omruleset plugin into >>> consideration). >>> >>> 3. The action queue is disk assisted. Which thread writes to disk when the >>> need arises? The one from the ruleset indata queue or the one from the >>> action queue? >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > From david at lang.hm Thu Dec 16 15:46:17 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 16 Dec 2010 06:46:17 -0800 (PST) Subject: [rsyslog] Watch for message conditions In-Reply-To: References: Message-ID: I'm not familiar with sending things to scripts like this, but in general when something isn't working the way you think it should, the way to start the debugging is to fire up rsyslog with debug enabled. usually the startup messages tell you what it's not able to parse. looking at this, shouldn't it be :msg, contains, "warning" instead of :msg, content "warning" David Lang On Thu, 16 Dec 2010, Mark Olliver wrote: > Hi All, > > I am trying to get rsyslog to alert me on certain patterns in the log. At > the moment i have built a small config with the following sections However, > nothing is the Alert script is not being triggered as far as i can tell. I > have tested the alert script and it works ok. > > > #/etc/rsyslog.d/55-alert.conf > $template Alert, "'Rsyslog Alert' %msg%" > local5.info > :msg, content "warning" > ^/var/c+i/bin/mail_alert Alert > > > I am testing the above by running the following command on the cli. > > echo "warning: test message" | logger -t puppet -p local5.info > > > This does correctly log into my local.5 log in /var/logs . > > Any ideas would be welcome. > > Thanks > > Mark > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mark at thermeon.com Thu Dec 16 16:28:24 2010 From: mark at thermeon.com (Mark Olliver) Date: Thu, 16 Dec 2010 15:28:24 +0000 Subject: [rsyslog] Watch for message conditions In-Reply-To: References: Message-ID: Ok, I think i have got a bit further, no i get an alert correctly however I am not getting the contents of my message set correctly. As far as i understand by creating a template I can then call omprog and give it the template name. It should then execute the program and ARGV0 will be the contents of the template. Is that correct? If so then can you tell my error from the following? #55-alert.conf $ModLoad omprog $Template, AlertMSG, "Test Alert\r\n%HOSTNAME%\r\n%timegenerated%\r\n%msg%" $ActionOMProgBinary /usr/bin/alert_mail :msg,contains,"warning" :omprog:;AlertMSG Also I take it %msg% contains the whole message as sent to syslog? The last question i have it how do i restrict this to just alert me for queue local5.* Thanks Mark On 16 December 2010 14:46, wrote: > I'm not familiar with sending things to scripts like this, but in general > when something isn't working the way you think it should, the way to start > the debugging is to fire up rsyslog with debug enabled. usually the startup > messages tell you what it's not able to parse. > > looking at this, shouldn't it be > > :msg, contains, "warning" > > instead of > > :msg, content "warning" > > David Lang > > > > > On Thu, 16 Dec 2010, Mark Olliver wrote: > > Hi All, >> >> I am trying to get rsyslog to alert me on certain patterns in the log. At >> the moment i have built a small config with the following sections >> However, >> nothing is the Alert script is not being triggered as far as i can tell. I >> have tested the alert script and it works ok. >> >> >> #/etc/rsyslog.d/55-alert.conf >> $template Alert, "'Rsyslog Alert' %msg%" >> local5.info >> :msg, content "warning" >> ^/var/c+i/bin/mail_alert Alert >> >> >> I am testing the above by running the following command on the cli. >> >> echo "warning: test message" | logger -t puppet -p local5.info >> >> >> This does correctly log into my local.5 log in /var/logs . >> >> Any ideas would be welcome. >> >> Thanks >> >> Mark >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From champ at softwink.com Thu Dec 16 16:34:00 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 16 Dec 2010 10:34:00 -0500 Subject: [rsyslog] liblognorm rulebase questions.. In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD8E1@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD89E@GRFEXC.intern.adiscon.com> <4D065BBA.2060608@mejor.pl> <4D074F23.2040202@mejor.pl> <20101215185908.GA20151@bundy.vistech.net> <20101215192342.GB20151@bundy.vistech.net> <9B6E2A8877C38245BFB15CC491A11DA71DD8E1@GRFEXC.intern.adiscon.com> Message-ID: <20101216153400.GA9957@bundy.vistech.net> > It probably is. I had hoped to have a final extensible format available now, > but then we had some issues finding consensus on how tags look in CEE, and > this held me. And now some expected and unexpected big bunch of work got into > my way :( Yep, I understand how that can happen. I guess, from a practical stand point, is to start using it and see how things evolve from there. I was just curious, but it sounds like a lot is on your plate already. > "prefix" is optional. If you don't need it, just don't specify it. I > introduced prefix to support using the same rule base with different, well, > prefixes ;) For example, in rsyslog I have the raw message and the parsed MSG > part of it. If you run the normalizer on the raw message, you need a common > prefix to cover, tag, date, ... If you parse the MSG part that prefix is not > present. So the core idea is to write the rules for the common format and add > different prefixes as required (or not provide a prefix at all). Okay.. Got it. Thanks for the information. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From david at lang.hm Thu Dec 16 16:43:28 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 16 Dec 2010 07:43:28 -0800 (PST) Subject: [rsyslog] Watch for message conditions In-Reply-To: References: Message-ID: On Thu, 16 Dec 2010, Mark Olliver wrote: > Ok, > > I think i have got a bit further, no i get an alert correctly however I am > not getting the contents of my message set correctly. As far as i understand > by creating a template I can then call omprog and give it the template name. > It should then execute the program and ARGV0 will be the contents of the > template. > Is that correct? I don't know, I haven't used omprog yet. > If so then can you tell my error from the following? > #55-alert.conf > $ModLoad omprog > $Template, AlertMSG, "Test Alert\r\n%HOSTNAME%\r\n%timegenerated%\r\n%msg%" > $ActionOMProgBinary /usr/bin/alert_mail > :msg,contains,"warning" :omprog:;AlertMSG > > > Also I take it %msg% contains the whole message as sent to syslog? it contains the message, but the message is everything after the syslog tag, not the timestamp, hostname, etc. you may want to use %rawmsg% instead which is _everything_ that was received. > The last question i have it how do i restrict this to just alert me for > queue local5.* multiple conditions gets uglier. there are three possible approaches 1. go to the less efficient if (( condition) and (condition)) decisioning approach 2. carefully order your rules and have the rules prior to this throw away all messages that are not local5.* 3. use multiple rulesets and have a condition that sends only local5.* rules to a ruleset that contains this rule. David Lang > Thanks > > Mark > > On 16 December 2010 14:46, wrote: > >> I'm not familiar with sending things to scripts like this, but in general >> when something isn't working the way you think it should, the way to start >> the debugging is to fire up rsyslog with debug enabled. usually the startup >> messages tell you what it's not able to parse. >> >> looking at this, shouldn't it be >> >> :msg, contains, "warning" >> >> instead of >> >> :msg, content "warning" >> >> David Lang >> >> >> >> >> On Thu, 16 Dec 2010, Mark Olliver wrote: >> >> Hi All, >>> >>> I am trying to get rsyslog to alert me on certain patterns in the log. At >>> the moment i have built a small config with the following sections >>> However, >>> nothing is the Alert script is not being triggered as far as i can tell. I >>> have tested the alert script and it works ok. >>> >>> >>> #/etc/rsyslog.d/55-alert.conf >>> $template Alert, "'Rsyslog Alert' %msg%" >>> local5.info >>> :msg, content "warning" >>> ^/var/c+i/bin/mail_alert Alert >>> >>> >>> I am testing the above by running the following command on the cli. >>> >>> echo "warning: test message" | logger -t puppet -p local5.info >>> >>> >>> This does correctly log into my local.5 log in /var/logs . >>> >>> Any ideas would be welcome. >>> >>> Thanks >>> >>> Mark >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From friedl at hq.adiscon.com Thu Dec 16 17:08:44 2010 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 16 Dec 2010 17:08:44 +0100 Subject: [rsyslog] rsyslog 6.1.2 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8F1@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 6.1.2, the new v6-devel. This release has as first release the experimental support for log normalization (via liblognorm). Support for normalizing log messages has been added in the form of mmnormalize. The core engine (property replacer, filter engine) has been enhanced to support properties from normalized events. Note: this is EXPERIMENTAL code. It is currently known that there are issues if the functionality is used with * disk-based queues * asynchronous action queues You can not use the new functionality together with these features. This limitation will be removed in later releases. However, we preferred to release early, so that one can experiment with the new feature set and accepted the price that this means the full set of functionality is not yet available. If not used together with these features, log normalizing should be pretty stable. We have written a small guide that should help you with setting up the new version and gives a introduction to the mmnormalize feature. You can find it here: http://www.rsyslog.com/normalizer-first-steps-for-mmnormalize/ Please note: To install this version you need libestr and libee installed on you system. Again, this new feature is EXPERIMENTAL! Additionaly, this release brings some enhancements for the testing tool tcpflood. It now supports sending via UDP, has the capability to run multiple iterations and generate statistic data records. ChangeLog: http://www.rsyslog.com/changelog-for-6-1-2-devel/ Download: http://www.rsyslog.com/rsyslog-6-1-2-devel/ As always, feedback is appreciated. Best regards, Florian Riedl -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From dave at fly.srk.fer.hr Thu Dec 16 17:48:38 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 16 Dec 2010 17:48:38 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: References: <20101215141340.GA6512@fly.srk.fer.hr> <20101216101516.GA29280@fly.srk.fer.hr> Message-ID: <20101216164838.GA17175@fly.srk.fer.hr> david at lang.hm wrote: > On Thu, 16 Dec 2010, Dra?en Ka?ar wrote: >> I believe it can be done without message loss. However, in my setup there >> are other programs on the machine which can use a lot of CPU time, so >> rsyslog might not be scheduled on a CPU fast enough. There is one CPU with >> four cores, one of them will be taken by a SCHED_FIFO process belonging to >> clusterware whenever it needs it and I'd like to do the same arrangement >> for rsyslog's UDP receiver thread. That leaves two cores for everything >> else, which should be perfectly fine. (There is a potential issue with >> memory in case the machine starts swapping, but I'm not trying to solve >> that right now.) > > this is where increasing the kernel buffers can save you by giving you > more leeway in when rsyslog _needs_ cpu to avoid loss. > > the other thing that you can do is to boost the priority of rsyslog with > nice. Using SCHED_FIFO scheduling class should be better than boosting nice value. SCHED_FIFO is basically a real-time scheduling class. If the thread is runnable, it gets the CPU immediately (unless there are other RT threads with higher priority) and stays on it as long as it wants. I'm not sure if Linux kernel handles priority inversion properly, but even if it doesn't, I hope that won't be a problem. > ideally you would do this only to the thread that is handling the > UDP reception, but I don't know a good way to find the pid of that > thread. I suppose it can be done, but pthread_setschedparam(pthread_self(), SCHED_FIFO, ...) is portable, simpler and better for my purposes. Also slightly dangerous, if one doesn't have enough processors for all RT threads. :-) >> Is there any way to get some statistics from rsyslog? It would be nice if >> it could write some counters in a file every hour or so. For example: the >> number of messages received by each of the input plugins and the number of >> messages written by each of the output plugins. > > there is a stats module that was created fairly recently. Thanks, I'll take a look. >> Well, my receiver side has other processes competing for the CPU. I think >> that can make a difference. > > one other performance tweak, there is an option to tell rsyslog not to > lookup the time for every message, but only do it every N messages when > there are alog of messages to process. setting this can help (the biggest > win is at the beginning, 2 saves 1/2 the lookups of 1, 4 saves 3/4 the > lookups, 10 saves 9/10 the lookups, etc) I know, I've done that already. > > David Lang > >>> Rainer has been doing some tests recently on the sending side that may >>> indicate some issues. I haven't had time to get my test systems setup >>> again to try more tests. >>> >>> David Lang >>> >>> On Wed, 15 Dec 2010, Dra?en Ka?ar wrote: >>> >>>> I have rsyslog 5.6.2 which is a destination for a lot of traffic which is >>>> currently using UDP as a transport protocol. We intend to switch some >>>> sources to TCP, but UDP will still be used for a large chunk of data and I >>>> can't do anything about that. >>>> >>>> Current tests show that we lose 100-400 lines in every 100000 sent. That >>>> isn't much and we could live with it, but I'd like to do something about >>>> it, nevertheless. >>>> >>>> My guess is that the kernel buffer fills up, rsyslog doesn't manage to >>>> make room fast enough and then the kernel drops packages. I'm on CentOS >>>> 5.5. Kernel log shows lines like: >>>> >>>> Dec 8 14:44:22 ssm1 kernel: Initalizing network drop monitor service >>>> Dec 13 15:09:26 ssm1 kernel: Initalizing network drop monitor service >>>> >>>> Curiously, the times do not match the ones for which I'm certain that the >>>> package loss occured. But even so, the log shows that packet drop is >>>> occuring on the system. >>>> >>>> As far as I know, two things can be done. >>>> >>>> 1. Increase UDP receive buffer size. That should allow the system to >>>> handle the bigger spike load than it's handling now, but that's about it. >>>> There will be a spike load which is too large for any given buffer size. >>>> >>>> I didn't manage to find a configuration option for this. I can increase >>>> the system default, but I dislike doing that because it affects all >>>> sockets on the system. If rsyslog really doesn't have this configuration >>>> option, I'm willing to write a patch. >>>> >>>> 2. Put a thread that handles UDP socket in real-time class (or the closest >>>> equivalent that a given OS has). The POSIX way for this is to use >>>> sched_setscheduler() with SCHED_FIFO. There could be other non-portable >>>> interfaces of interest (like priocntl() on Solaris), but I won't discuss >>>> them right now. >>>> >>>> The patch should add two configuration parameters: >>>> * one to define scheduling policy (SCHED_FIFO, SCHED_RR, etc.), let's >>>> call it $QueueThreadSchedulingPolicy (although I'd like a >>>> shorter name) >>>> * and another to define sched_priority, let's call it >>>> $QueueThreadSchedulingPriority >>>> >>>> In case of SCHED_SPORADIC (and maybe other non-portable policies on some >>>> operating systems) there are additional parameters that can be set, so >>>> maybe the second parameter would turn out to be the string with several >>>> numbers that needs to be parsed. Or there could be additional >>>> configuration options introduced. I don't have a position about that and >>>> would like to follow the existing rsyslog policy regarding configuration >>>> options. Except that I don't know what that policy is, so I'd appreciate >>>> if someone can enlighten me. >>>> >>>> Anyway, the idea is to have a high priority thread (or a small number of >>>> threads) sitting on a UDP socket whose sole purpose in life is to >>>> transfer the messages from UDP socket to another queue(s), as fast as >>>> possible. By default (if $QueueThreadSchedulingPolicy is defined), >>>> the number of threads put in that scheduling policy can be >>>> $QueueWorkerThreads, ie. the minimum number of worker threads. >>>> There could be additional parameter to define the number of such threads, >>>> but I'm not sure how to go about it. >>>> >>>> There could be no imposed limit, so if an administator puts 128 in the >>>> config file, 128 threads it is. But this is potentialy dangerous. Or there >>>> could be rsyslog imposed restriction that the number of such threads cannot >>>> exceed the number defined in $QueueWorkerThreads, which is less >>>> dangerous. I'm personally leaning to the second option. >>>> >>>> >>>> Now, this whole thing is dangerous and must be implemented and used with >>>> care and understanding. Currently I don't understand rsyslog queues well >>>> enough, which is why I'm writing this mail. The UDP thread in FIFO class >>>> should be used just to pick the packet from the socket and place it on >>>> another action queue. Let's say we have the following configuration: >>>> >>>> $Ruleset indata >>>> $RulesetCreateMainQueue on >>>> >>>> $UDPServerRun 514 >>>> $InputUDPServerBindRuleset indata >>>> >>>> $MainMsgQueueThreadSchedulingPolicy fifo >>>> $MainMsgQueueThreadSchedulingPriority 10 >>>> >>>> $ActionName local1 >>>> $ActionQueueType LinkedList >>>> $ActionQueueFileName local1 >>>> $ActionResumeRetryCount -1 >>>> $ActionQueueSaveOnShutdown on >>>> $ActionQueueDiscardSeverity 8 >>>> >>>> local1.* |/path/to/fifo >>>> & /path/to/file >>>> >>>> >>>> If I understand things correctly, there are two queues here. The first one >>>> is ruleset indata queue with the UDP input module and the second one is >>>> the action queue for the destinations (FIFO and file). I'm not sure about >>>> the following: >>>> >>>> 1. After a $Ruleset directive followed by "$RulesetCreateMainQueue on", >>>> are $MainMsgQueue* directives applied to that ruleset queue? It's not very >>>> intuitive. >>>> >>>> 2. Say we add a TCP listener (or any other input module) to the above >>>> configuration and we want UDP processing thread(s) to have SCHED_FIFO >>>> priority, but other input handling threads to stay at the default >>>> priority. I'm not sure if that can be specified with one ruleset. If two >>>> rulesets must be specified, then I suppose the action directives must be >>>> duplicated for each ruleset, which isn't very readable and >>>> maintenance-friendly. (This is without taking omruleset plugin into >>>> consideration). >>>> >>>> 3. The action queue is disk assisted. Which thread writes to disk when the >>>> need arises? The one from the ruleset indata queue or the one from the >>>> action queue? >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Thu Dec 16 18:41:31 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Dec 2010 18:41:31 +0100 Subject: [rsyslog] Losing UDP packages References: <20101215141340.GA6512@fly.srk.fer.hr><20101216101516.GA29280@fly.srk.fer.hr> <20101216164838.GA17175@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> > I suppose it can be done, but pthread_setschedparam(pthread_self(), > SCHED_FIFO, ...) is portable, simpler and better for my purposes. Also > slightly dangerous, if one doesn't have enough processors for all RT > threads. :-) If that's all that is needed, I could very probably hack this together while I wait for a longer running test... Rainer From david at lang.hm Thu Dec 16 19:40:24 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 16 Dec 2010 10:40:24 -0800 (PST) Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> References: <20101215141340.GA6512@fly.srk.fer.hr><20101216101516.GA29280@fly.srk.fer.hr> <20101216164838.GA17175@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> Message-ID: On Thu, 16 Dec 2010, Rainer Gerhards wrote: >> I suppose it can be done, but pthread_setschedparam(pthread_self(), >> SCHED_FIFO, ...) is portable, simpler and better for my purposes. Also >> slightly dangerous, if one doesn't have enough processors for all RT >> threads. :-) > > > If that's all that is needed, I could very probably hack this together while > I wait for a longer running test... make sure it's a config option, it could do bad things on a uni-processor machine. David Lang From champ at softwink.com Thu Dec 16 20:41:18 2010 From: champ at softwink.com (Champ Clark III [Softwink]) Date: Thu, 16 Dec 2010 14:41:18 -0500 Subject: [rsyslog] liblognorm problems... In-Reply-To: <20101215200722.GA21847@bundy.vistech.net> References: <005201cb9c8c$d9ff17ed$100013ac@intern.adiscon.com> <20101215200722.GA21847@bundy.vistech.net> Message-ID: <20101216194118.GA19290@bundy.vistech.net> > Understood. This is really rough PoC code. I know I have a > "ip=" field, but I'm unclear on how to "call" that from > ee_getEventField(event, ??) ... > > I'm assuming I inform ee_getEventField to obtain the "ip" > field, then use ee_getFieldValueAsStr to get the value. Correct? Argh. I'm such a dork :) You know, sometimes it's better to just go to bed and pick up the next morning. This morning, I reviewed what Rainer had said, and had it working in about 5 minutes. I wasn't using es_newStrFromBuf() buff correctly at all! So hence, ee_getEventField() would obviously fail. Anyways, in my PoC code I did a: propName = es_newStrFromBuf("ip", 2); /* Yes, terrible, just wanted see it work */ ... and what do you know? I get the value I was looking for. Man, sometimes I get to excited, I end up missing the most simple things :) Thanks for the suggestions and hard work on liblognorm. -- Champ Clark III | Softwink, Inc | 800-538-9357 x 101 http://www.softwink.com GPG Key ID: 58A2A58F Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F If it wasn't for C, we'd be using BASI, PASAL and OBOL. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From rgerhards at hq.adiscon.com Thu Dec 16 21:19:37 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 16 Dec 2010 21:19:37 +0100 Subject: [rsyslog] Losing UDP packages Message-ID: <005401cb9d5e$8d0cf0d6$100013ac@intern.adiscon.com> For sure ... And by default off ;) rainer ----- Urspr?ngliche Nachricht ----- Von: david at lang.hm Gesendet: Donnerstag, 16. Dezember 2010 19:40 An: rsyslog-users Betreff: Re: [rsyslog] Losing UDP packages On Thu, 16 Dec 2010, Rainer Gerhards wrote: >> I suppose it can be done, but pthread_setschedparam(pthread_self(), >> SCHED_FIFO, ...) is portable, simpler and better for my purposes. Also >> slightly dangerous, if one doesn't have enough processors for all RT >> threads. :-) > > > If that's all that is needed, I could very probably hack this together while > I wait for a longer running test... make sure it's a config option, it could do bad things on a uni-processor machine. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From dave at fly.srk.fer.hr Thu Dec 16 22:48:10 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Thu, 16 Dec 2010 22:48:10 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> References: <20101216164838.GA17175@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> Message-ID: <20101216214810.GA30537@fly.srk.fer.hr> Rainer Gerhards wrote: > > I suppose it can be done, but pthread_setschedparam(pthread_self(), > > SCHED_FIFO, ...) is portable, simpler and better for my purposes. Also > > slightly dangerous, if one doesn't have enough processors for all RT > > threads. :-) > > > If that's all that is needed, I could very probably hack this together while > I wait for a longer running test... There are two parameters needed: 1. policy: probably a string that maps to SCHED_xxx, ie: fifo => SCHED_FIFO rr => SCHED_RR other => SCHED_OTHER The code for those mappings should probably be in the appropriate "#ifdef SCHED_xxx" guards. They should be defined if pthread_setschedparam() is available, but you never know. Ignore SCHED_SPORADIC for the time being. 2. priority (integer), goes into the third argument for pthread_setschedparam(). Without defaults for any of these parameters. If one of them is specified in the config file and the other is not, report an error. Add a configure check (it's possible you'd need -lrt or something similar), and that's about that. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Fri Dec 17 12:38:59 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 17 Dec 2010 12:38:59 +0100 Subject: [rsyslog] Losing UDP packages References: <20101216164838.GA17175@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DD8F3@GRFEXC.intern.adiscon.com> <20101216214810.GA30537@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> I just committed the functionality: http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=cc8237736d11b54a3d6089d8 36da7ccb6972a29c Note that I am far from being an autotools expert, so I would appreciate if you could provide proper configure checks. I did only a very rough test of the functionality. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Thursday, December 16, 2010 10:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] Losing UDP packages > > Rainer Gerhards wrote: > > > I suppose it can be done, but pthread_setschedparam(pthread_self(), > > > SCHED_FIFO, ...) is portable, simpler and better for my purposes. > Also > > > slightly dangerous, if one doesn't have enough processors for all > RT > > > threads. :-) > > > > > > If that's all that is needed, I could very probably hack this > together while > > I wait for a longer running test... > > There are two parameters needed: > 1. policy: probably a string that maps to SCHED_xxx, ie: > fifo => SCHED_FIFO > rr => SCHED_RR > other => SCHED_OTHER > > The code for those mappings should probably be in the appropriate > "#ifdef SCHED_xxx" guards. They should be defined if > pthread_setschedparam() is available, but you never know. > > Ignore SCHED_SPORADIC for the time being. > > 2. priority (integer), goes into the third argument for > pthread_setschedparam(). > > Without defaults for any of these parameters. If one of them is > specified > in the config file and the other is not, report an error. > > Add a configure check (it's possible you'd need -lrt or something > similar), > and that's about that. > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From dave at fly.srk.fer.hr Fri Dec 17 16:26:48 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Fri, 17 Dec 2010 16:26:48 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> References: <20101216214810.GA30537@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> Message-ID: <20101217152648.GA24238@fly.srk.fer.hr> I'm not an expert either. :-) The attached diff is against rsyslog 5.6.2, it also checks for sched_get_priority_max() which can be used to check the proper priority range for a given policy. I've tested the patch on Linux and Solaris. Seems to work fine on both. Rainer Gerhards wrote: > I just committed the functionality: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=cc8237736d11b54a3d6089d8 > 36da7ccb6972a29c > > Note that I am far from being an autotools expert, so I would appreciate if > you could provide proper configure checks. I did only a very rough test of > the functionality. > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > > Sent: Thursday, December 16, 2010 10:48 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Losing UDP packages > > > > Rainer Gerhards wrote: > > > > I suppose it can be done, but pthread_setschedparam(pthread_self(), > > > > SCHED_FIFO, ...) is portable, simpler and better for my purposes. > > Also > > > > slightly dangerous, if one doesn't have enough processors for all > > RT > > > > threads. :-) > > > > > > > > > If that's all that is needed, I could very probably hack this > > together while > > > I wait for a longer running test... > > > > There are two parameters needed: > > 1. policy: probably a string that maps to SCHED_xxx, ie: > > fifo => SCHED_FIFO > > rr => SCHED_RR > > other => SCHED_OTHER > > > > The code for those mappings should probably be in the appropriate > > "#ifdef SCHED_xxx" guards. They should be defined if > > pthread_setschedparam() is available, but you never know. > > > > Ignore SCHED_SPORADIC for the time being. > > > > 2. priority (integer), goes into the third argument for > > pthread_setschedparam(). > > > > Without defaults for any of these parameters. If one of them is > > specified > > in the config file and the other is not, report an error. > > > > Add a configure check (it's possible you'd need -lrt or something > > similar), > > and that's about that. > > > > -- > > .-. .-. Yes, I am an agent of Satan, but my duties are largely > > (_ \ / _) ceremonial. > > | > > | dave at fly.srk.fer.hr > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr -------------- next part -------------- A non-text attachment was scrubbed... Name: rsyslog-sched.diff Type: text/x-diff Size: 1538 bytes Desc: not available URL: From ariel.p at hostdime.com Fri Dec 17 23:32:38 2010 From: ariel.p at hostdime.com (Ariel P.) Date: Fri, 17 Dec 2010 17:32:38 -0500 Subject: [rsyslog] Feedback on ommysql requested In-Reply-To: <20101202155126.GZ19162@aart.is.rice.edu> References: <20101202155126.GZ19162@aart.is.rice.edu> Message-ID: <4D0BE506.8090000@hostdime.com> I have created a new version of my patch, which creates two configuration settings for ommysql. The first setting sets the path for the 'my.cnf' file. If this setting is not specified in the configuration file, the mysql library is not informed of a file path (current behavior) The second setting is used to change the section of the 'my.cnf' file to use, which defaults to "client" if not specified in the configuration file (this is mysql standard behavior). Please see [ http://bugzilla.adiscon.com/show_bug.cgi?id=213#c3 ] for details. From Ward.P.Fontenot at wellsfargo.com Tue Dec 21 19:51:16 2010 From: Ward.P.Fontenot at wellsfargo.com (Ward.P.Fontenot at wellsfargo.com) Date: Tue, 21 Dec 2010 12:51:16 -0600 Subject: [rsyslog] Win32 vs Linux format Message-ID: Hi, I have a question about formatting - I'm certain I've simply overlooked something fairly simple I currently use syslog-ng on a linux central syslog server and have linux, solaris and Windows reporting to it just fine. We are building the replacement for this aging syslog-ng server and will be using rsyslog. My question is what can be causing the formatting between the linux and windows clients to be different. Syslog-ng didn't present these differences. Here is a log snippet for a linux machine and a Win32 machine Win32: Dec 21 11:48:30 Tue Dec 21 11:47:36 2010: MONITOR1P/Security (520) - "blah blah" Linux: Dec 21 11:50:02 oracle1p crond[26200]: (root) CMD (/usr/lib/sa/sa1 1 1) Why would there be two timestamps on the Win32 machine? The only difference is the change from syslog-ng to rsyslog. Thanks in advance for any help on this matter. Enterprise Key Management & Public Key Infrastructure | EKM/PKI Engineering Team 1305 W 23rd Street | Tempe, AZ 85282 MAC S4003-018 Tel 480-437-7795 | Cell 480-788-0730 ward.p.fontenot at wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. From david at lang.hm Wed Dec 22 00:23:43 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 21 Dec 2010 15:23:43 -0800 (PST) Subject: [rsyslog] Win32 vs Linux format In-Reply-To: References: Message-ID: On Tue, 21 Dec 2010, Ward.P.Fontenot at wellsfargo.com wrote: > Hi, > > I have a question about formatting - I'm certain I've simply overlooked something fairly simple > > I currently use syslog-ng on a linux central syslog server and have linux, solaris and Windows reporting to it just fine. We are building the replacement for this aging syslog-ng server and will be using rsyslog. My question is what can be causing the formatting between the linux and windows clients to be different. Syslog-ng didn't present these differences. > > Here is a log snippet for a linux machine and a Win32 machine > > Win32: > Dec 21 11:48:30 Tue Dec 21 11:47:36 2010: MONITOR1P/Security (520) - "blah blah" > > Linux: > Dec 21 11:50:02 oracle1p crond[26200]: (root) CMD (/usr/lib/sa/sa1 1 1) > > Why would there be two timestamps on the Win32 machine? The only difference is the change from syslog-ng to rsyslog. the reason is that the timestamp that was sent was "Tue Dec 21 11:47:36 2010:", which is not a valid syslog format. As a result, rsyslog creates it's own timestamp. unfortunantly rsyslog can't tell that "Tue" isn't a valid hostname, so it uses that for the hostname and "Dec" for the syslog tag (program name) what application is sending this message? is there a way to have it reformat the message? there are various things that you can do in rsyslog to clean this up (create a special parser module, creat a suxtom format), but it's cleaner to fix this at the source if you can. a good thing to do in this sort of situation is to create a log where you log with the format %rawmsg% (or use the RSYSLOG_DEBUG format) to see what you are actually getting over the wire. David Lang > Thanks in advance for any help on this matter. > > Enterprise Key Management & Public Key Infrastructure | EKM/PKI Engineering Team > > 1305 W 23rd Street | Tempe, AZ 85282 > MAC S4003-018 > Tel 480-437-7795 | Cell 480-788-0730 > > ward.p.fontenot at wellsfargo.com > > This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From raubvogel at gmail.com Wed Dec 22 08:09:28 2010 From: raubvogel at gmail.com (Mauricio Tavares) Date: Wed, 22 Dec 2010 02:09:28 -0500 Subject: [rsyslog] facilities and fail2ban Message-ID: <4D11A428.8070305@gmail.com> AFAIK, fail2ban is not a facility so something like this fail2ban.* /var/log/fail2ban/fail2ban.log should not work. However, it seems that *.*;auth,authpriv.none;\ fail2ban.none; \ local6.none;local7.none -/var/log/syslog/syslog.log works. How so? From david at lang.hm Wed Dec 22 08:16:27 2010 From: david at lang.hm (david at lang.hm) Date: Tue, 21 Dec 2010 23:16:27 -0800 (PST) Subject: [rsyslog] facilities and fail2ban In-Reply-To: <4D11A428.8070305@gmail.com> References: <4D11A428.8070305@gmail.com> Message-ID: On Wed, 22 Dec 2010, Mauricio Tavares wrote: > AFAIK, fail2ban is not a facility so something like this > > fail2ban.* /var/log/fail2ban/fail2ban.log > > should not work. However, it seems that > > *.*;auth,authpriv.none;\ > fail2ban.none; \ > local6.none;local7.none -/var/log/syslog/syslog.log > > works. How so? when you say it 'works', what does it do? David Lang From rgerhards at hq.adiscon.com Wed Dec 22 19:07:29 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 22 Dec 2010 19:07:29 +0100 Subject: [rsyslog] Feedback on ommysql requested References: <20101202155126.GZ19162@aart.is.rice.edu> <4D0BE506.8090000@hostdime.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD93D@GRFEXC.intern.adiscon.com> Hi all, thanks to Ariel for the hard work. I, too, think that all concerns are addressed. I intend to merge the patch when I am back in January if I don't here any good argument why I should not ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ariel P. > Sent: Friday, December 17, 2010 11:33 PM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Feedback on ommysql requested > > I have created a new version of my patch, which creates two > configuration settings for ommysql. > The first setting sets the path for the 'my.cnf' file. If this setting > is not specified in the configuration file, the mysql library is not > informed of a file path (current behavior) > The second setting is used to change the section of the 'my.cnf' file > to > use, which defaults to "client" if not specified in the configuration > file (this is mysql standard behavior). > > Please see [ http://bugzilla.adiscon.com/show_bug.cgi?id=213#c3 ] for > details. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From jason at jasonantman.com Wed Dec 22 21:41:41 2010 From: jason at jasonantman.com (Jason Antman) Date: Wed, 22 Dec 2010 15:41:41 -0500 Subject: [rsyslog] Multiple rulesets and queues - strange behavior, problems logging to MySQL Message-ID: <4D126285.50002@jasonantman.com> Greetings, I'm trying to implement multiple rulesets with per-ruleset queues in order to handle messages going to MySQL. Rsyslog is doing template/regex based parsing for the MySQL inserts - it's not raw log messages, or the Adiscon format. I seem to be running into some errors. I tried creating a disk-assisted queue, stopping mysql for a minute or so, grabbing some relatively unique strings from the text files containing the raw log data, starting MySQL back up, and then searching for the strings - which I presumed would eventually be flushed from the queue to the database. Not only did I never see any log data collected when MySQL is down end up in the database, but after about 3 minutes with MySQL down, not only did I not get any disk queue files, but I saw slightly garbled versions of the log messages ending up in /var/log/messages - which isn't even mentioned anywhere in the rulesets! I'm binding imtcp and imudp to a ruleset ("remote") that first logs everything to dynfiles based on hostname, and then (for a few specific source IPs) passes messages on to omruleset for further processing. Aside from some repetitive rules for string matching, my ruleset looks like: $RuleSet DHCP-parsing # create a Disk-Assisted LinkedList main queue specific to this ruleset $WorkDirectory /var/rsyslog/work $RulesetCreateMainQueue on $MainMsgQueueFilename DHCPqueue $MainMsgQueueType LinkedList $MainMsgQueueMaxDiskSpace 500M $MainMsgQueueSaveOnShutdown on # persist queue contents to disk on shutdown # TESTING ONLY *.* /var/log/dhcp-parsing # END TESTING ONLY $ActionResumeRetryCount -1 # retry infinitely :msg, startswith, " DHCPREQUEST for" :ommysql:localhost,test,syslogger,syslogger;DHCPREQUESTMAC & :ommysql:localhost,test,syslog,foo;DHCPREQUESTIP & ~ ### DISCARD :msg, startswith, " DHCPACK on" :ommysql:localhost,test,syslogger,syslogger;DHCPACKonIP & :ommysql:localhost,test,syslog,foo;DHCPACKonMAC & ~ ### DISCARD :msg, startswith, " DHCPINFORM from" :ommysql:localhost,test,syslog,foo;DHCPINFORM & ~ ### DISCARD if $msg startswith ' DHCPACK to' and ( not ( $msg contains 'no client hardware address' ) ) \ then :ommysql:localhost,test,syslog,foo;DHCPACKtoMAC & :ommysql:localhost,test,syslog,foo;DHCPACKtoIP & ~ ### DISCARD The garbage that I was seeing in /var/log/messages looked like (as you can see, right after the timestamp, it's obviously not properly formatted, though prior to killing mysql, it looked fine in the correct log files): Dec 22 15:20:41 : dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (Courtney-PC) via eth0 Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab (Courtney-PC) via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (Courtney-PC) via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPOFFER on 123.456.78.123 to 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 (123.456.78.123) from 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via 123.456.78.123 Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab (F52F2867C1364CC) via eth0 Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (F52F2867C1364CC) via eth0 Dec 22 15:20:42 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via 123.456.78.123 Dec 22 15:20:42 l dhcpd: DHCPOFFER on 123.456.78.123 to 01:23:45:67:89:ab via 123.456.78.123 Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab via eth0 Dec 22 15:20:42 l dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab via eth0 Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab via 123.456.78.123 Dec 22 15:20:42 ast dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab via 172.31.16 Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab via 123.456.78.123 Dec 22 15:20:47 l dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab (Blue-PC) via eth0 Dec 22 15:20:47 eER from 01:23:45:67:89:ab via 172.dhcpd: dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (Blue-PC) via eth0 Dec 22 15:20:47 l.25.114 dhcpd: DHCPREQUEST for 123.456.78.123 from 01:23:45:67:89:ab (Blue-PC) via 123.456.78.123 Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab (Blue-PC) via 123.456.78.123 Any ideas? I chose rsyslog in the hopes that I'd be able to replace our current cron-triggered log parsing scripts with rsyslog builtin templates and regex functions, but it seems like I'm running into more and more problems trying to develop complex rules that can also handle decent message volume and things like restarts of mysql. Thanks for any advice, Jason From rs at datacymru.net Wed Dec 22 21:59:23 2010 From: rs at datacymru.net (Rob Shepherd) Date: Wed, 22 Dec 2010 20:59:23 +0000 Subject: [rsyslog] converted %rawsmg% to %msg% Message-ID: <4D1266AB.3030805@datacymru.net> Dear rsyslogd users, I have a semicolon seperated msg but the first field in the submitted string has a space in it. Here is the %rawmsg% "<158>Dec 21 22:35:21 x-wing.datacymru.net 2010-12-21 22:35:21;2010-12-21 22:35:21+0000;null;null;35667;127.0.0.1;USER_ACCESS;USER_LOGGED_IN;User: abc at example.com Logged in" And here is the resulting %msg% " 22:35:21;2010-12-21 22:35:21+0000;null;null;35667;127.0.0.1;USER_ACCESS;USER_LOGGED_IN;User: channeladmin0 at zetaplc.com Logged in" (The double quotes aren't part of the resulting text, I've just put it there to indicate the leading in the %msg%) As you can see the first field has been stripped of it's 'yyyy-MM-dd' component. This makes the DB insert fail > pgsql, DBError(silent): db error (0): ERROR: invalid input syntax for > type timestamp: " 22:49:38" at character 136 I've found that the missing text makes it's way to the %app-name% and %syslogtag% properties I have a workaround, by using %syslogtag%%msg:F,59:1% to provide the two parts as required. I'd like to not use a workaround if possible. is there a command the defines how the message is pulled out from the raw.? can I encapsulate my semicolon-delimited string any better to avoid this behaviour? Many thanks Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd From david at lang.hm Wed Dec 22 22:08:21 2010 From: david at lang.hm (david at lang.hm) Date: Wed, 22 Dec 2010 13:08:21 -0800 (PST) Subject: [rsyslog] converted %rawsmg% to %msg% In-Reply-To: <4D1266AB.3030805@datacymru.net> References: <4D1266AB.3030805@datacymru.net> Message-ID: On Wed, 22 Dec 2010, Rob Shepherd wrote: > Dear rsyslogd users, > > I have a semicolon seperated msg but the first field in the submitted > string has a space in it. > > Here is the %rawmsg% > > "<158>Dec 21 22:35:21 x-wing.datacymru.net 2010-12-21 > 22:35:21;2010-12-21 > 22:35:21+0000;null;null;35667;127.0.0.1;USER_ACCESS;USER_LOGGED_IN;User: > abc at example.com Logged in" > > And here is the resulting %msg% > > " 22:35:21;2010-12-21 > 22:35:21+0000;null;null;35667;127.0.0.1;USER_ACCESS;USER_LOGGED_IN;User: > channeladmin0 at zetaplc.com Logged in" > > (The double quotes aren't part of the resulting text, I've just put it > there to indicate the leading in the %msg%) > > As you can see the first field has been stripped of it's 'yyyy-MM-dd' > component. > > This makes the DB insert fail >> pgsql, DBError(silent): db error (0): ERROR: invalid input syntax for type >> timestamp: " 22:49:38" at character 136 > > > I've found that the missing text makes it's way to the %app-name% and > %syslogtag% properties > > > I have a workaround, by using %syslogtag%%msg:F,59:1% to provide the two > parts as required. > > I'd like to not use a workaround if possible. > > is there a command the defines how the message is pulled out from the > raw.? can I encapsulate my semicolon-delimited string any better to > avoid this behaviour? the problem you are running into is that the syslog RFC defines a properly formatted message as being <###>MMM DD HH:MM:SS hostname syslogtag message where message can contain just about anything. the default parser tries very hard to figure out what is what to handle malformed messages, but in your case it has no way to know that the firs section of the message you want belongs as part of %msg% and there is no syslogtag field. creating a custom format to combine the fields is the easiest way to deal with the problem. The other option is to create a custom parser that will understand these messages and put everything into the %msg% field. This is a very new capaibility in rsyslog, and so there aren't many examples of how to do this yet. David Lang From rs at datacymru.net Thu Dec 23 11:33:38 2010 From: rs at datacymru.net (Rob Shepherd) Date: Thu, 23 Dec 2010 10:33:38 +0000 Subject: [rsyslog] converted %rawsmg% to %msg% In-Reply-To: References: <4D1266AB.3030805@datacymru.net> Message-ID: <4D132582.7040905@datacymru.net> > > the problem you are running into is that the syslog RFC defines a > properly formatted message as being > > <###>MMM DD HH:MM:SS hostname syslogtag message > > where message can contain just about anything. > > the default parser tries very hard to figure out what is what to > handle malformed messages, but in your case it has no way to know that > the firs section of the message you want belongs as part of %msg% and > there is no syslogtag field. > > creating a custom format to combine the fields is the easiest way to > deal with the problem. > > The other option is to create a custom parser that will understand > these messages and put everything into the %msg% field. This is a very > new capaibility in rsyslog, and so there aren't many examples of how > to do this yet. > > David Lang David, Thanks for the quick response. I'm grateful for your suggestion. In fact, the syslog implementation I am using is syslog4j. http://www.syslog4j.org/ It seems this library does not provide a tag by default (unless I am using it incorrectly). Using: SyslogIF.setIdent("appName"); ....inserted a tag and the rawmsg parser now produces all the %msg% I expect. Great stuff, Many thanks Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd From raubvogel at gmail.com Thu Dec 23 20:07:43 2010 From: raubvogel at gmail.com (Mauricio Tavares) Date: Thu, 23 Dec 2010 14:07:43 -0500 Subject: [rsyslog] facilities and fail2ban In-Reply-To: References: <4D11A428.8070305@gmail.com> Message-ID: <4D139DFF.20905@gmail.com> On 12/22/10 2:16 AM, david at lang.hm wrote: > On Wed, 22 Dec 2010, Mauricio Tavares wrote: > >> AFAIK, fail2ban is not a facility so something like this >> >> fail2ban.* /var/log/fail2ban/fail2ban.log >> >> should not work. However, it seems that >> >> *.*;auth,authpriv.none;\ >> fail2ban.none; \ >> local6.none;local7.none -/var/log/syslog/syslog.log >> >> works. How so? > > when you say it 'works', what does it do? > What I wanted to happen is that output from fail2ban should not be written into the syslog.log file. Instead it would go to its own file. How I ended up achieving the latter was to say if ($programname contains 'fail2ban') then /var/log/fail2ban/fail2ban.log I *want* to say fail2ban is not writing to syslog.log right now, but since we were not attached since last night I do not know if *.*;auth,authpriv.none;\ fail2ban.none; \ local6.none;local7.none -/var/log/syslog/syslog.log is working. From what I read, it shouldn't... > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Thu Dec 23 20:35:35 2010 From: david at lang.hm (david at lang.hm) Date: Thu, 23 Dec 2010 11:35:35 -0800 (PST) Subject: [rsyslog] facilities and fail2ban In-Reply-To: <4D139DFF.20905@gmail.com> References: <4D11A428.8070305@gmail.com> <4D139DFF.20905@gmail.com> Message-ID: On Thu, 23 Dec 2010, Mauricio Tavares wrote: > On 12/22/10 2:16 AM, david at lang.hm wrote: >> On Wed, 22 Dec 2010, Mauricio Tavares wrote: >> >>> AFAIK, fail2ban is not a facility so something like this >>> >>> fail2ban.* /var/log/fail2ban/fail2ban.log >>> >>> should not work. However, it seems that >>> >>> *.*;auth,authpriv.none;\ >>> fail2ban.none; \ >>> local6.none;local7.none -/var/log/syslog/syslog.log >>> >>> works. How so? >> >> when you say it 'works', what does it do? >> > What I wanted to happen is that output from fail2ban should not be > written into the syslog.log file. Instead it would go to its own file. How I > ended up achieving the latter was to say > > if ($programname contains 'fail2ban') then /var/log/fail2ban/fail2ban.log > > I *want* to say fail2ban is not writing to syslog.log right now, but since we > were not attached since last night I do not know if > > *.*;auth,authpriv.none;\ > fail2ban.none; \ > local6.none;local7.none -/var/log/syslog/syslog.log > > is working. From what I read, it shouldn't... no, it wouldn't because fail2ban.none is not valid. the part in front of the '.' is not an application name, it's a reserved name that maps to one of ~16 possible facility values. what I would expect would happen is that nothing would be written to syslog.log with your example, because I would expect rsyslog to declare that line invalid and ignore it at startup. David Lang From dave at fly.srk.fer.hr Mon Dec 27 18:19:41 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Mon, 27 Dec 2010 18:19:41 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> References: <20101216214810.GA30537@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> Message-ID: <20101227171941.GA18225@fly.srk.fer.hr> Rainer Gerhards wrote: > I just committed the functionality: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=cc8237736d11b54a3d6089d8 > 36da7ccb6972a29c There is a serious flaw there. The pthread_setschedparam() function is being called by the main thread before other threads are created, so all threads will inherit the scheduling parameters, which we don't want. That call has to be moved from willRun() function to the rcvMainLoop() which is being executed by a new thread so it will afect the UDP thread only. You can check this with: ps -eLo "cmd lwp policy rtprio" I'm attaching a patch which does that, against rsyslog 5.6.2. (After applying you need to call autoreconf). It also has improved validation. Note that the allowed scheduling priority range is OS specific, so the code shouldn't expect that it will be non-negative. I think that the better way to implement this feature would be to add pthread_attr_t * as an argument to the willRun() functions in order to allow them to change the thread properties. Scheduling properties can be set with pthread_attr_setschedpolicy() and pthread_attr_setschedparam(), for example. > Note that I am far from being an autotools expert, so I would appreciate if > you could provide proper configure checks. I did only a very rough test of > the functionality. Included in the patch. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr -------------- next part -------------- A non-text attachment was scrubbed... Name: rsyslog-imudp-setschedparam.diff Type: text/x-diff Size: 7315 bytes Desc: not available URL: From dave at fly.srk.fer.hr Tue Dec 28 15:57:25 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Tue, 28 Dec 2010 15:57:25 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <20101227171941.GA18225@fly.srk.fer.hr> References: <20101216214810.GA30537@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com> <20101227171941.GA18225@fly.srk.fer.hr> Message-ID: <20101228145725.GA6780@fly.srk.fer.hr> Dra?en Ka?ar wrote: > There is a serious flaw there. The pthread_setschedparam() function is > being called by the main thread before other threads are created, so all > threads will inherit the scheduling parameters, which we don't want. That > call has to be moved from willRun() function to the rcvMainLoop() which is > being executed by a new thread so it will afect the UDP thread only. You > can check this with: > > ps -eLo "cmd lwp policy rtprio" > > I'm attaching a patch which does that, against rsyslog 5.6.2. (After > applying you need to call autoreconf). It also has improved validation. > Note that the allowed scheduling priority range is OS specific, so the > code shouldn't expect that it will be non-negative. I've put the patch to test on the server under load and there were several threads with FIFO scheduling policy. One of them was receiving packets from the UDP socket. Another was locking and unlocking mutex (or mutexes) and strace didn't show anything else. Yet another thread was writing to the log file. There was one or two more, but I didn't bother to check what they were doing. This is not good. It seems like the UDP receiver thread is spawning other threads and then they inherit scheduling policy. That should be prevented somehow. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr From rgerhards at hq.adiscon.com Tue Dec 28 22:21:55 2010 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 28 Dec 2010 22:21:55 +0100 Subject: [rsyslog] Losing UDP packages References: <20101216214810.GA30537@fly.srk.fer.hr><9B6E2A8877C38245BFB15CC491A11DA71DD900@GRFEXC.intern.adiscon.com><20101227171941.GA18225@fly.srk.fer.hr> <20101228145725.GA6780@fly.srk.fer.hr> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com> I am on vacation right now. But I think what happens is that the worker threads inherit the priority setting from the UDP listener thread. You probably need to change thread creation in ./runtime/wtp.c. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Dra?en Kacar > Sent: Tuesday, December 28, 2010 3:57 PM > To: rsyslog-users > Subject: Re: [rsyslog] Losing UDP packages > > Dra?en Ka?ar wrote: > > > There is a serious flaw there. The pthread_setschedparam() function > is > > being called by the main thread before other threads are created, so > all > > threads will inherit the scheduling parameters, which we don't want. > That > > call has to be moved from willRun() function to the rcvMainLoop() > which is > > being executed by a new thread so it will afect the UDP thread only. > You > > can check this with: > > > > ps -eLo "cmd lwp policy rtprio" > > > > I'm attaching a patch which does that, against rsyslog 5.6.2. (After > > applying you need to call autoreconf). It also has improved > validation. > > Note that the allowed scheduling priority range is OS specific, so > the > > code shouldn't expect that it will be non-negative. > > I've put the patch to test on the server under load and there were > several > threads with FIFO scheduling policy. One of them was receiving packets > from the UDP socket. Another was locking and unlocking mutex (or > mutexes) > and strace didn't show anything else. Yet another thread was writing to > the log file. There was one or two more, but I didn't bother to check > what > they were doing. > > This is not good. It seems like the UDP receiver thread is spawning > other threads and then they inherit scheduling policy. That should be > prevented somehow. > > -- > .-. .-. Yes, I am an agent of Satan, but my duties are largely > (_ \ / _) ceremonial. > | > | dave at fly.srk.fer.hr > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From gallen at openworld.co.za Wed Dec 29 15:44:52 2010 From: gallen at openworld.co.za (Gareth Allen) Date: Wed, 29 Dec 2010 16:44:52 +0200 Subject: [rsyslog] removing characters from %syslogtag% Message-ID: Hi all, I've recently started setting up a remote rsyslog server for an Apache cluster. Sending the logs to the remote log server works well, but when I try and create a dynamic file name using the %syslogtag% tag I get a colon in the name e.g. /data/logs/sitelogs/otrs/httpd/foo.bar:-custom.log An example Apache log entry looks like this: 2010-12-29T15:41:31+02:00 foo foo.bar: - - [29/Dec/2010:15:41:31 +0200] "GET /otrs-web/skins/Agent/default/img/widget_bg_100.png HTTP/1.1" 304 - "http:///otrs-web/skins/Agent/default/css-cache/CommonCSS_ed85b1c7566e62a443ae9b4af8b335a8.css" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12" Does anyone know how I could remove the : when trying to create a file? Thanks in advance Gareth From dave at fly.srk.fer.hr Wed Dec 29 18:08:05 2010 From: dave at fly.srk.fer.hr (=?iso-8859-2?Q?Dra=BEen_Ka=E8ar?=) Date: Wed, 29 Dec 2010 18:08:05 +0100 Subject: [rsyslog] Losing UDP packages In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com> References: <20101228145725.GA6780@fly.srk.fer.hr> <9B6E2A8877C38245BFB15CC491A11DA71DD979@GRFEXC.intern.adiscon.com> Message-ID: <20101229170805.GB14863@fly.srk.fer.hr> Rainer Gerhards wrote: > I am on vacation right now. But I think what happens is that the worker > threads inherit the priority setting from the UDP listener thread. You > probably need to change thread creation in ./runtime/wtp.c. I hoped there would be a better method. Anyway, I've added thread attributes in every pthread_create call, since changing just the one in wtp.c wasn't enough. There is one pthread_create() in plugins/imsolaris/sun_cddl.c which I didn't touch because it seems buggy. It's using create_door_thr as pthread_attr_t, but create_door_thr is never initialized, as far as I can see. The updated patch against rsyslog 5.6.2 is attached. I have only UDP thread in real-time mode now. I'm not sure if the code which gets the default thread properties should go in rsyslog.c or somewhere else. It can be safely moved anywhere in the initialization sequence, before the first pthread_create is called. About configure check: the proper way to check for the functionality would be to check for _XOPEN_REALTIME_THREADS preprocessor macro. That's what's supposed to be defined if real-time thread functionality is available. However, there is no that symbol anywhere in /usr/include on Solaris 10 (update 6 is what I checked). The equivalent run-time check sysconf(_SC_XOPEN_REALTIME_THREADS) is returning 1, though, so the lack of _XOPEN_REALTIME_THREADS macro is a Solaris bug, as far as I can tell. Therefore I'm checking for the availability of pthread_setschedparam() and then have all real-time thread code in #ifdef HAVE_PTHREAD_SETSCHEDPARAM blocks. It's not ideal, but I hope it works. -- .-. .-. Yes, I am an agent of Satan, but my duties are largely (_ \ / _) ceremonial. | | dave at fly.srk.fer.hr -------------- next part -------------- A non-text attachment was scrubbed... Name: rsyslog-imudp-setschedparam.diff Type: text/x-diff Size: 11247 bytes Desc: not available URL: From fabien.bagard at parrot.com Thu Dec 30 10:30:38 2010 From: fabien.bagard at parrot.com (Fabien Bagard) Date: Thu, 30 Dec 2010 10:30:38 +0100 Subject: [rsyslog] Certificate not trusted problem In-Reply-To: References: <20101215141340.GA6512@fly.srk.fer.hr> Message-ID: <4D1C513E.6040203@parrot.com> Hi ! I tried to set up a centralized rsyslog server, with cyphered connections. Both server & client certificates were signed by the same CA. The problem is my certificates are not trusted : ==> logger/rsyslogd.log <== 2010-12-15T15:17:01.685026+01:00 logging-server rsyslogd: not permitted to talk to peer, certificate invalid: signer is not a CA 2010-12-15T15:17:01.685895+01:00 logging-server rsyslogd: invalid cert info: peer provided 1 certificate(s). Certificate 1 info: certificate valid from DATE to DATE; Certificate public key: RSA; DN: C=FR,ST=France,L=Paris,O=Parrot S.A.,OU=IT Department,CN=fr-b-701-006.parrot.biz,EMAIL=fabien.bagard_AT_parrot.com; Issuer DN: C=FR,ST=France,L=Paris,O=Parrot S.A. Root CA,OU=IT Department,CN=Parrot S.A. Root CA,EMAIL=root_AT_parrot.com; ==> logger/rsyslogd-2089.log <== 2010-12-15T15:17:01.725557+01:00 logger rsyslogd-2089: netstream session 0x90b55b8 will be closed due to error [try http://www.rsyslog.com/e/2089 ] When checking my certificates with gnutls-cli, I get : ... Other Information: MD5 fingerprint: 9dbbdeea9cee323e06553e632250d669 SHA-1 fingerprint: 008247f9dcc86bbfe958ef7a1b12ed9917320ad2 Public Key Id: 29dcaddda6746140e556804d7ea4af8fd47e62a5 - The hostname in the certificate matches 'logging-server.parrot.biz'. - Peer's certificate issuer is unknown - Peer's certificate is NOT trusted - Version: TLS1.1 - Key Exchange: DHE-RSA - Cipher: AES-128-CBC - MAC: SHA1 - Compression: NULL - Session ID: 61:2D:87:98:63:9F:14:EB:11:BB:7A:9F:1B:53:FC:DE:80:E2:4C:7C:C7:3F:15:9B:8A:18:25:6D:35:40:B1:4E *** Verifying server certificate failed... Any hints on what's going wrong ? By the way, I'm pretty sure my certificates are good enough because certificates were already signed with this CA. Thanks a lot,