From infofarmer at FreeBSD.org Tue Apr 1 08:36:16 2008 From: infofarmer at FreeBSD.org (Andrew Pantyukhin) Date: Tue, 1 Apr 2008 10:36:16 +0400 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <20080401063614.GD4692@amilo.cenkes.org> On Mon, Mar 31, 2008 at 07:52:37PM +0200, Michael Biebl wrote: > I don't see the use case yet. Why would someone want to build > the plugins without the rsyslogd binary? Does it actually work > if you mix different versions of plugins and rsyslogd? If > someone is interested in only a specific plugin, why not use > "make -C plugins/foo"? It's just a convenience feature. In FreeBSD, we have separate ports for core and each plugin. At present, each plugin has to build rsyslogd all over again. From now on, it can just require the core port and save all the wasted build-time. I'm sure there are other ways to do it (like the one you mentioned), but is there a merit in arguing which one's the best? From rgerhards at hq.adiscon.com Tue Apr 1 08:50:30 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:50:30 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: <20080401063614.GD4692@amilo.cenkes.org> References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> <20080401063614.GD4692@amilo.cenkes.org> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CC9@grfint2.intern.adiscon.com> As a side-note: > Does it actually work > if you mix different versions of plugins and rsyslogd? The answer is: it depends. Rsyslog uses versioning in the plugin interface and also uses versioning in all internal objects. But there are still some non-object accesses. As long as a plugin is of the same version as the plugin interface that rsyslogd is compiled to, it works - provided that all objects it needs are implemented in the core (for v3, this must not necessarily be the case, especially if you use a new plugin with an older rsylogd build). The bottom line, of course, is that mixing versions should be avoided. But the good news is that rsyslog and the plugins detect incompatible versions and, if found, the plugin is simply not activated. So it is a predictable failure case (again, not 100% safe for the v3 case as it is not purely object based so far). Rainer From rgerhards at hq.adiscon.com Tue Apr 1 08:56:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:56:45 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com><1206988699.9240.38.camel@cutter><1206991339.9240.45.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Related question: does anybody know of a (sufficiently good) library that abstracts differences between TLS implementation. I mean in the way libdbi does for databases? It's obviously not important to have more than one driver available at runtime, but it would be necessary to be able to link against different TLS implementation. If anybody has a suggestion, please let me know. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, March 31, 2008 9:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] openssl vs rsyslog > > Whatever I do - I'll try to create a small driver shim so that > hopefully > different libraries could be used together with rsyslog. It depends on > the APIs, but in general that should not be too hard (but practice > often > tells the difference). > > On FIPS I made I comment, where I think the mails crossed. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Monday, March 31, 2008 9:27 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] openssl vs rsyslog > > > > 2008/3/31, seth vidal : > > > On Mon, 2008-03-31 at 21:18 +0200, Michael Biebl wrote: > > > > Interesting to know. Do you know any technical > > advantages of NSS over > > > > GnuTLS (stability, features, nicer API, etc)? > > > > > > > > > openssl transition library > > > fips certification. > > > > > > http://www.mozilla.org/projects/security/pki/nss/fips/ > > > > > > the second one being the most important, I think. > > > > Yeah, the openssl transition library doesn't concern us. > > > > Why is the fips certification important for rsyslog? > > > > Cheers, > > Michael > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 1 09:19:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 09:19:28 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCB@grfint2.intern.adiscon.com> > > > It fixes two bugs, one is a potential segfault in the > syslog/plain tcp > > > receiver. The other one is removal of some debug instrumentation > that > > > accidently made it into 3.12.5. There is also a new ./configure > option > > > (--enable/disable-rsyslogd) which permits to build just specific > plugins > > > > > > Hm, the default of enable_rsyslogd should imho be "yes" That was a bug, which is now fixed. > Forgot to add: If you intend to build rsyslogd and rfc3195d > conditionally, the corresponding man pages should also be installed > conditionally. This is now done. Thanks for pointing both out. Rainer From pvrabec at redhat.com Tue Apr 1 11:12:43 2008 From: pvrabec at redhat.com (Peter Vrabec) Date: Tue, 1 Apr 2008 11:12:43 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Message-ID: <200804011112.43694.pvrabec@redhat.com> Hi, On Tuesday 01 April 2008 08:56:45 am Rainer Gerhards wrote: > Related question: does anybody know of a (sufficiently good) library > that abstracts differences between TLS implementation. I mean in the way > libdbi does for databases? It's obviously not important to have more > than one driver available at runtime, but it would be necessary to be > able to link against different TLS implementation. > > If anybody has a suggestion, please let me know. you can look at nss_compat_ossl http://rcritten.fedorapeople.org/nss_compat_ossl.html http://fedoraproject.org/wiki/nss_compat_ossl but when I talked to guy who work on Fedora Crypto Consolidation it was suggested not to use the abstract library, since all these TLS implementations have kind of different "philosophy". It would be much cleaner to use separate plugins for different TLS. Note, that we will appreciate a lot going with NSS. http://fedoraproject.org/wiki/FedoraCryptoConsolidation http://fedoraproject.org/wiki/CryptoConsolidationEval > Thanks, > Rainer From rgerhards at hq.adiscon.com Tue Apr 1 13:50:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 13:50:53 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <200804011112.43694.pvrabec@redhat.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> <200804011112.43694.pvrabec@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD1@grfint2.intern.adiscon.com> Hi all, > http://fedoraproject.org/wiki/FedoraCryptoConsolidation > http://fedoraproject.org/wiki/CryptoConsolidationEval This material (the big picture) is quite convincing. Are there similar efforts in other distros? To make a point, I'll probably start developing an as slim as possible abstraction layer as a side-project (much like librelp) and see what I can do with it. I'll probably start with NSS and then see what it take to move the other libs in. My initial focus will be on "get it going", so it will not be a very secure solution at first. All comments are still very welcome and will be considered. I just thought it would be good to do a sum-up of my current thinking. Rainer From rgerhards at hq.adiscon.com Tue Apr 1 19:00:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 19:00:19 +0200 Subject: [rsyslog] rsyslog 3.15.0 released - with RELP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD7@grfint2.intern.adiscon.com> Hi all, yes, I know it's April, 1st. But this release is not related to it ;) I have just released rsyslog 3.15.0, the release that finally provides initial RELP support. RELP provides superior reliability over plain tcp syslog, ensuring that no messages are lost. With plain TCP, this can happen if the remote server goes down. RELP protects from this by utilizing a full-duplex protocol where each message is acknowledged. RELP should also be safe to use with stunnel, where plain tcp syslog sometimes gets into trouble. The core RELP protocol support is NOT part of rsyslog. It comes in the form of librelp, available at http://www.librelp.com . You need to install librelp before you try to compile rsyslog with RELP support. That should be fairly simple. If you run into any troubles, please let me know - I am more than happy to help. I plan to do some more in-depth doc on the use cases soon. Change Log: http://www.rsyslog.com/Article203.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-93.phtml I hope this release is useful. Feedback is appreciated. Rainer From BLentz at channing-bete.com Tue Apr 1 21:53:16 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 15:53:16 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem Message-ID: <47F292AC.8040307@channing-bete.com> Greetings, list! I'm having trouble compiling 3.15.0. I get the following error on one machine (CentOS 5): rsyslogd-msg.o: In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch_4' And slightly different error on another machine (Fedora Core 3): rsyslogd-debug.o(.text+0x183d): In function `dbgEntrFunc': /usr/src/redhat/BUILD/rsyslog-3.15.0/debug.c:996: undefined reference to `__sync_fetch_and_add' rsyslogd-msg.o(.text+0x127): In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch' rsyslogd-msg.o(.text+0xc55): In function `MsgAddRef': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:449: undefined reference to `__sync_fetch_and_add' Has anyone else seen this or know of a workaround or fix? I can't seem to find any reference in the rsyslog install documentation which says that I need additional libraries on my system. Thanks in advance From mbiebl at gmail.com Tue Apr 1 23:10:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:10:56 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F292AC.8040307@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Ben Lentz : > Greetings, list! > I'm having trouble compiling 3.15.0. > > I get the following error on one machine (CentOS 5): > rsyslogd-msg.o: In function `msgDestruct': > /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to > `__sync_sub_and_fetch_4' See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 #undef DO_HAVE_ATOMICS 1 should help Obviously, ./configure should detect this automatically. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Tue Apr 1 23:23:47 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:23:47 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Michael Biebl : > > Obviously, ./configure should detect this automatically. We could steal from here: http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From BLentz at channing-bete.com Tue Apr 1 23:25:08 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 17:25:08 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <47F2A834.9000509@channing-bete.com> > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > #undef DO_HAVE_ATOMICS 1 should help > > Obviously, ./configure should detect this automatically. > > Thanks! I'm still getting errors with debug.c It seems that debug.c has a reference to ATOMIC_INC (line 996) without a set of ifdef DO_HAVE_ATOMICS around it, so the build still dies. If I comment that line out, it will compile, but the binary is unusable "rsyslogd run failed with error -3000." On a whim, I tried ./configure --disable-debug, no luck. Ugh! From rgerhards at hq.adiscon.com Wed Apr 2 08:38:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Apr 2008 08:38:40 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F2A834.9000509@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> Message-ID: <1207118320.22738.3.camel@localhost.localdomain> It's a bug, obviously. I have now fixed it. Please go to atomic.h and change the ATOMIC_INC macro to be #define ATOMIC_INC(data) (++(data)) That'll solve the problems. Doing so does not introduce any real weakness - the code has been tested always with this setting. The atomics are a recent addition to make it ultra-reliable on all CPU architectures ... but obviously we need to do more work on the feature selection than I initially thought... Thanks for bringing this up, it was just at the right time. I'll pull that code from today's stable release (that happens when you think you can add something laaate in the process because it is "unintrusive" ;)). Rainer On Tue, 2008-04-01 at 17:25 -0400, Ben Lentz wrote: > > > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > > > #undef DO_HAVE_ATOMICS 1 should help > > > > Obviously, ./configure should detect this automatically. > > > > > Thanks! > > I'm still getting errors with debug.c It seems that debug.c has a > reference to ATOMIC_INC (line 996) without a set of ifdef > DO_HAVE_ATOMICS around it, so the build still dies. > > If I comment that line out, it will compile, but the binary is unusable > "rsyslogd run failed with error -3000." > > On a whim, I tried ./configure --disable-debug, no luck. Ugh! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 08:40:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 08:40:27 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD9@grfint2.intern.adiscon.com> Ummm ;) Could you lend me a helping hand? That looks god, but it also looks like it goes above my autotools horizon ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Tuesday, April 01, 2008 11:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > 2008/4/1, Michael Biebl : > > > > Obviously, ./configure should detect this automatically. > > We could steal from here: > http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 10:35:29 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 04:35:29 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <1207118320.22738.3.camel@localhost.localdomain> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> Message-ID: <47F34551.4010902@channing-bete.com> > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > change the ATOMIC_INC macro to be > > #define ATOMIC_INC(data) (++(data)) > > That'll solve the problems. Hey Rainer, Thanks for the help. I still continue to have problems, now at runtime. I'm patching the source like so: --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 @@ -36,8 +36,8 @@ #define INCLUDED_ATOMIC_H /* set the following to 1 if we have atomic operations (and #undef it otherwise) */ -#define DO_HAVE_ATOMICS 1 -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) +#undef DO_HAVE_ATOMICS 1 +#define ATOMIC_INC(data) (++(data)) #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) #endif /* #ifndef INCLUDED_ATOMIC_H */ then: rsyslogd run failed with error -3000 I've compiled with ggdb and run through the debugger, but I'm not sure where to set the break points... tried a couple things but nothing worked. I've also run it with strace, but I can't seem to find any problems. I believe there's a ton of reasons that it would generate this message and I'm not sure where to start. From rgerhards at hq.adiscon.com Wed Apr 2 10:40:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:40:35 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com><1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CDF@grfint2.intern.adiscon.com> Hi, > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 10:35 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > > > > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > > change the ATOMIC_INC macro to be > > > > #define ATOMIC_INC(data) (++(data)) > > > > That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 - > 0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > That's fine.. > then: > rsyslogd run failed with error -3000 Nice - error codes are defined in rsyslog.h. But of course, -3000 means "an error occurred". In any case, it's something that fails cleanly... > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate > this > message and I'm not sure where to start. ... that's why you don't see anything in the debugger. So let's get a bit down on where it really happens. There is debug support inside rsyslog: http://www.rsyslog.com/doc-debug.html It would be best if you do ./configure --enable-rtinst but that's not strictly necessary. Set RSYSLOG_DEBUG to LogFuncFlow and run rsyslog interactively with -d -n. You'll get some debug output. That should tell us something (well, you could even start without seting RSYSLOG_DEBUG, I think that'll be sufficient and less output). Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From theinric at redhat.com Wed Apr 2 10:46:34 2008 From: theinric at redhat.com (theinric at redhat.com) Date: Wed, 02 Apr 2008 10:46:34 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <47F347EA.7000501@redhat.com> On 04/02/2008 10:35 AM, Ben Lentz wrote: > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h and >> change the ATOMIC_INC macro to be >> >> #define ATOMIC_INC(data) (++(data)) >> >> That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > then: > rsyslogd run failed with error -3000 > > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate this > message and I'm not sure where to start. Hi Ben, my guess is that rsyslog can't find its libraries (e.g. /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" or something like "ln -s $(find /path/to/rsyslog/sources/ -name "*.so") /usr/[local/]lib/rsyslog". You can run "strace ./rsyslogd -v" to see where it looks for its libraries. From rgerhards at hq.adiscon.com Wed Apr 2 10:48:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:48:28 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F347EA.7000501@redhat.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> You are probably right. After your message I checked the module loader and it just returns the generic error code. I'll fix that... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of theinric at redhat.com > Sent: Wednesday, April 02, 2008 10:47 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > On 04/02/2008 10:35 AM, Ben Lentz wrote: > > > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h > and > >> change the ATOMIC_INC macro to be > >> > >> #define ATOMIC_INC(data) (++(data)) > >> > >> That'll solve the problems. > > > > Hey Rainer, > > Thanks for the help. I still continue to have problems, now at > runtime. > > I'm patching the source like so: > > > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 > -0400 > > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > > @@ -36,8 +36,8 @@ > > #define INCLUDED_ATOMIC_H > > > > /* set the following to 1 if we have atomic operations (and #undef > it > > otherwise) */ > > -#define DO_HAVE_ATOMICS 1 > > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > > +#undef DO_HAVE_ATOMICS 1 > > +#define ATOMIC_INC(data) (++(data)) > > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > > > then: > > rsyslogd run failed with error -3000 > > > > I've compiled with ggdb and run through the debugger, but I'm not > sure > > where to set the break points... tried a couple things but nothing > > worked. I've also run it with strace, but I can't seem to find any > > problems. I believe there's a ton of reasons that it would generate > this > > message and I'm not sure where to start. > > Hi Ben, > > my guess is that rsyslog can't find its libraries (e.g. > /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" > or something like "ln -s $(find /path/to/rsyslog/sources/ -name > "*.so") /usr/[local/]lib/rsyslog". > You can run "strace ./rsyslogd -v" to see where it looks for its > libraries. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 11:03:12 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 05:03:12 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> Message-ID: <47F34BD0.8060606@channing-bete.com> > You are probably right. After your message I checked the module loader > and it just returns the generic error code. I'll fix that... > > Rainer > > >> Hi Ben, >> >> my guess is that rsyslog can't find its libraries (e.g. >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name >> "*.so") /usr/[local/]lib/rsyslog". >> You can run "strace ./rsyslogd -v" to see where it looks for its >> libraries. >> Yep, you're right. I was expecting: - Shared library issues at runtime to be visible using ldd(1) - To at least be able to get command-line help or version (-v) information from rsyslogd (without loadable modules) - A nicer error message ;-) strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file or directory) Can the module path (/usr/lib/rsyslog) be specified? Is there a ./configure option that I'm missing at compile time or a command line parameter to rsyslogd or config file parameter in rsyslog.conf? I've been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and can't seem to find anything? It's very early in the US and I might just be missing something obvious. From rgerhards at hq.adiscon.com Wed Apr 2 11:07:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 11:07:03 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34BD0.8060606@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com><577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> <47F34BD0.8060606@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE2@grfint2.intern.adiscon.com> > >> my guess is that rsyslog can't find its libraries (e.g. > >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make > install" > >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name > >> "*.so") /usr/[local/]lib/rsyslog". > >> You can run "strace ./rsyslogd -v" to see where it looks for its > >> libraries. > >> > Yep, you're right. I was expecting: > - Shared library issues at runtime to be visible using ldd(1) > - To at least be able to get command-line help or version (-v) > information from rsyslogd (without loadable modules) > - A nicer error message ;-) > > strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT > access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or > directory) > open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file > or > directory) > > Can the module path (/usr/lib/rsyslog) be specified? Is there a > ./configure option that I'm missing at compile time or a command line > parameter to rsyslogd or config file parameter in rsyslog.conf? I've > been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and > can't > seem to find anything? That's mostly distro-specific and this is where I have very limited knowledge. But you can set the module path via the environment, use RSYSLOG_MODDIR=/path/to/modules/ . There is also a -m option to do the same, but it currently is read too late to help with the core modules. On my 64Bit system adding --libdir=/lib64 often helped with such problems when I encountered them. Not sure if that's the right cure... > > It's very early in the US and I might just be missing something > obvious. You are *really* brave! From rgerhards at hq.adiscon.com Wed Apr 2 16:34:37 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 16:34:37 +0200 Subject: [rsyslog] RELP vs plain tcp syslog Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Hi folks, I just blogged about the unreliability of plain tcp syslog. This may be interesting for at least some of you: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl og.html Rainer From Gerrard.Geldenhuis at datacash.com Wed Apr 2 17:24:57 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Wed, 2 Apr 2008 16:24:57 +0100 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: Hi Thanks, that is good to know. Worth keeping in mind. I was under the impression tcp was the miracle cure for udp unreliability but it seems not. There is very expensive software out there that can do guarenteed packet deliver over tcp... tibco is one I have seen before. Regards > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 02 April 2008 15:35 > To: rsyslog-users > Subject: [rsyslog] RELP vs plain tcp syslog > > Hi folks, > > I just blogged about the unreliability of plain tcp syslog. This may be > interesting for at least some of you: > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl > og.html > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 17:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 17:34:56 +0200 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CFA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gerrard Geldenhuis > Sent: Wednesday, April 02, 2008 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] RELP vs plain tcp syslog > > Hi > Thanks, that is good to know. Worth keeping in mind. I was under the > impression tcp was the miracle cure for udp unreliability but it seems > not. > > There is very expensive software out there that can do guarenteed > packet > deliver over tcp... Well... RELP can *really* do the trick. It needs some help in the rsyslog engine to work under all cases, that is when the relp stack is terminated (when the client rsyslog is shut down). But that isn't rocket science and has just been pushed back by more urgent work (securing the channel). So far, relp does single ack, that is the server ack's every packet back to the client. Client discards packets only after ack. So when a session breaks, we know what the server has processed. If, however, some of the acks get lost, we resend packets that the server already processed, resulting in some mild message *duplication* (we currently have a 128 packet app-layer max window and typically acks come in rather quickly). To work around this, the client must ack the server's acks (double-acking). That sounds scary, but is not. It's also not performance intense and the protocol is modeled that those with ultra-slim bandwidth can turn it off and live with the message duplication scenario. If double-acks are active, relp client and server will go into a recovery phase after reconnect negotiation, in which mutally acked package numbers are exchanged. To do so, the client must provide a session cookie back to the server. This is a potential attack vector and thus I'd like to have secure transport in place before I do it. Once the recovery negotiation is done, client and server know, and have discarded, what each other peer has processed. The remaining packets are re-sent, but under the same reliability settings. So if the connection breaks at this point again (or during the renegotiation), we'll simply go into another recovery phase until we finally succeed. It is of course important that session caches be persisted to disk when an engine stops - that will be part of rsyslog. The recovery produced itself is librelp. Once this is done, and with proper rsyslog queue settings, sufficiently stable hardware and sufficient disk space, I guarantee (not in a lawyers sense, though ;)) that you'll never lose a message nor get a duplicate. This is when I think we have achieved our reliability goal. If all goes well... summer? ;) Rainer > > tibco is one I have seen before. > > Regards > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: 02 April 2008 15:35 > > To: rsyslog-users > > Subject: [rsyslog] RELP vs plain tcp syslog > > > > Hi folks, > > > > I just blogged about the unreliability of plain tcp syslog. This may > be > > interesting for at least some of you: > > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > sysl > > og.html > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 23:10:39 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 17:10:39 -0400 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter Message-ID: <47F3F64F.5090600@channing-bete.com> When I try to use $HOSTNAME in an expression-based filter, I get "INVALID PROPERTY" errors logged by the system. From the documentation, I'm lead to believe that expression-based filters use the same properties as expression-based filters, which are documented in the property replacer documentation. Okay. But it seems that maybe $HOSTNAME has been changed to $source in the expression-based filters, but I can't seem to find any evidence of this in the man or HTML documentation. From rgerhards at hq.adiscon.com Thu Apr 3 08:52:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Apr 2008 08:52:31 +0200 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter In-Reply-To: <47F3F64F.5090600@channing-bete.com> References: <47F3F64F.5090600@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D04@grfint2.intern.adiscon.com> Ah, good report. I see where the problem is stemming from. We use the same code for property access no matter which method is used (expression-based, property-based, ...). That's good. The bad thing is that property names were case-sensitive (a really bad idea). We were in the discussion to remove that. The RainerScript engine already is case-insensitive and consequently converts all variable names to lower case. However, the message object property access method has not yet been converted, so it expects it in upper case. The result is the mess that you see... I'll initiate a quick discussion, but I think we'll now drop case-sensitivity at all. That shouldn't hurt any existing installations but instead make life easier. Thanks for bringing this up. For the same reason "$source" is the solution and I have seen in the forum that it works for you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 11:11 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter > > When I try to use $HOSTNAME in an expression-based filter, I get > "INVALID PROPERTY" errors logged by the system. From the documentation, > I'm lead to believe that expression-based filters use the same > properties as expression-based filters, which are documented in the > property replacer documentation. Okay. But it seems that maybe > $HOSTNAME > has been changed to $source in the expression-based filters, but I > can't > seem to find any evidence of this in the man or HTML documentation. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 10:06:55 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:06:55 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: 2008/3/31, Rainer Gerhards : > Well... First thing is that I have urgent need to release -- there are a > couple of things in the queue. If I don't release them this week, I'll > probably don't have a chance to get to a stable any time soon. So I will > release, even at the price that the version number scheme may be less > than optimum this time. > > But now on to the real meat (and thanks for sticking around on this > topic! ;))... I like the idea of the odd/even numbering (of the minor number) to distinguish unstable/stable release. E.g. GNOME uses it and it seems to work fine for them. E.g. the latest stable release was 2.22.0 (and minor point/bug fix releases usually follow as 2.22.1, 2.22.2...) The major development is going on in 2.23. The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... As soon as the code base stabilises (reaching beta/rc quality), they will release 2.23.90,2.23.91,... and the final stable release will be 2.24.0. (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From r.bhatia at ipax.at Fri Apr 4 10:16:01 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 04 Apr 2008 10:16:01 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: <47F5E3C1.9090207@ipax.at> Michael Biebl wrote: > I like the idea of the odd/even numbering (of the minor number) to > distinguish unstable/stable release. E.g. GNOME uses it and it seems > to work fine for them. > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > releases usually follow as 2.22.1, 2.22.2...) > The major development is going on in 2.23. > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > As soon as the code base stabilises (reaching beta/rc quality), they > will release > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). its not new as afairc the linux kernel established this scheme. anyways, what bugs me is, that we then will have something like: 3.20.x - stable 3.21.x - unstable relp no 3.22.x 3.23.x - unstable tls when we then decide to implement yet-another-new-feature-in-a- seperate-tree, we will end up with a scenario like 3.20.x - oldstable 3.21.x - unstable relp - closed 3.23.x - unstable tls 3.25.x - unstable featureX 3.26.x - stable or similar. that is kind of odd ... rainer: what do you think about only having one dev tree and having some patchsets for not-that-much-worked-on features (e.g. tls, featureX). of course, you can arrange your repository as you wish, but i would stick to *one* stable version and *one* dev version for the testers. everything else should be build (as patches) on top of these 2 branches. as a nice effect, we will have only 1 stable core and 1 dev core and have no issues when it comes to merging core-relp with core-tls with core-featureX. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From mbiebl at gmail.com Fri Apr 4 10:45:09 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:45:09 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: 2008/4/4, Raoul Bhatia [IPAX] : > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). > > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable > > or similar. > > that is kind of odd ... > git feature branches would be ideal for this kind of development imho. > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 branches. > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. It would definitely make it easier if there was only one stable and one development tree (and maybe a third one, oldstable). Rainer could work on experimental new features in separate git feature branches and when he considers them ready for testing, merge them into the dev branch (usually called master in git). E.g. the relp functionality could be in a branch called feature-relp, tls in feature-tls etc. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:06:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:06:35 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Ah, the source of the misunderstanding finally hits me... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Friday, April 04, 2008 10:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, > 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered > correctly). > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like The trees are NOT independent of each other! There is always just one development tree. In the above scenario, 3.23.x *includes* 3.21.x. Let's look what I currently try: We will finally see 3.14.0 stable today. I have also done a number of improvements since the relp version. So I have (not yet released) 3.14.0 - stable, bug fixes only 3.15.0 - everything that is in stable, plus relp - set to stabilize, bug fixes only 3.17.0 - everything that is in 3.15.x (note the .x not .0) plus new features - devel Development only happens in 3.17.0. When a feature focus has been reached, I now plan to branch off that release and let it stabilize (that is no new features, just bug fixes). I am trialing this currently with the 3.15.x branch. So far, it looks manageable. The plan is to slowly migrate 3.15.x to become 3.16.0, the next stable, deprecating 3.14.x when it comes out. I think that'll on average will be every two month, but I don't know yet. With 3.15.x it may be quicker, as the whole RELP code is in an external library and thus the core engine has not been destabilized. > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable So this picture would actually be: 3.20.x - oldstable 3.21.x - unstable relp - closed 3.11.x - stable 3.23.x - unstable tls 3.25.x - unstable tls+featureX That actually bring us down to three branches (plus old legacy like v2 stable): - stable - stabilizing (branched off the dev tree and in the bugfix wait queue - devel There may be more than one stabilizing tree at a given time (not yet thought out). In recent history, we have one such sample. That is the queue engine. It was an extremely big overhaul of nearly everything. So it needed some extended time to mature. In the mean time, some more focus features could be implemented. With the version numbering system I have now on my mind, that would probably have looked as follows (I am intentionally using major version 999 to avoid confusion with existing versions): 999.2.0 - stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.9.x - devel Actually, we are happy with the feature introduced in 999.7.x. But we won't release a new stable because .5 is not yet ready for prime time. So nothing happens at that point. Then, we are confident in .5. I think the following happens then: 999.2.0 - deprecated stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.8.0 - stable 999.9.x - devel We will never see a 999.6. I personally think this is acceptable, especially as I think it won't happen very often. Does that make more sense? > or similar. > > that is kind of odd ... > > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 > branches. I am a bit concerned that I need to be too careful which patch I apply where. As you know, rsyslog is developed on a very fast pace and I'd rather prefer to keep coding instead of thinking where I add a "by-feature" (that maybe takes an hour to implement). The scheme I am currently trialing (outlined above) works pretty OK. It is some additional overhead, but I mostly do everything to the trunk, create a patch for bugfixes and apply that to the "old" version in question. Some more work than before, but quite bearable. Of course things are different if I work on a feature, abandon it, and go back after a few weeks. But currently I implement everything up to a usable state (maybe not 100%, but useful in what is there) before I take the next step. Again... comments please ;) Rainer > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 11:19:59 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 11:19:59 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 999.2.0 - stable > 999.3.x - big overhaul feature, stabilizing > 999.5.x - .3 + next focus feature, stabilizing > 999.7.x - .5 + next focus feature, stabilizing > 999.9.x - devel > > Again... comments please ;) I think you really should try to use git feature branches for that. Have a stable and master (development) branch, and develop the features in separate topic branches feature-A, feature-B etc. Whenever one feature is ready, merge it into master. This way, it doesn't matter which feature you have to concentrate on is released first (no skipped version numbers!). The strong merge suppport in git would also allow to cherrypick easily from the different feature branches or merge between them. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:42:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:42:42 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> > 2008/4/4, Rainer Gerhards : > > > > 999.2.0 - stable > > 999.3.x - big overhaul feature, stabilizing > > 999.5.x - .3 + next focus feature, stabilizing > > 999.7.x - .5 + next focus feature, stabilizing > > 999.9.x - devel > > > > Again... comments please ;) > > I think you really should try to use git feature branches for that. > Have a stable and master (development) branch, and develop the > features in separate topic branches feature-A, feature-B etc. > Whenever one feature is ready, merge it into master. > This way, it doesn't matter which feature you have to concentrate on > is released first (no skipped version numbers!). > The strong merge suppport in git would also allow to cherrypick easily > from the different feature branches or merge between them. Sounds good, but a honest question (NOT trying to give a bias, just a problem description): While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and implement them - all while FocusFeatureX is being developed. Where do I apply these? And don't I get into trouble if that interferes with things that I do in FocusFeatureX? Remember, I change a couple of hundered lines all over the project on a typical day... Rainer > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 12:02:20 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 12:02:20 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 2008/4/4, Rainer Gerhards : > > > > > > 999.2.0 - stable > > > 999.3.x - big overhaul feature, stabilizing > > > 999.5.x - .3 + next focus feature, stabilizing > > > 999.7.x - .5 + next focus feature, stabilizing > > > 999.9.x - devel > > > > > > Again... comments please ;) > > > > I think you really should try to use git feature branches for that. > > Have a stable and master (development) branch, and develop the > > features in separate topic branches feature-A, feature-B etc. > > Whenever one feature is ready, merge it into master. > > This way, it doesn't matter which feature you have to concentrate on > > is released first (no skipped version numbers!). > > The strong merge suppport in git would also allow to cherrypick easily > > from the different feature branches or merge between them. > > > Sounds good, but a honest question (NOT trying to give a bias, just a > problem description): > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > implement them - all while FocusFeatureX is being developed. Where do I > apply these? And don't I get into trouble if that interferes with things > that I do in FocusFeatureX? Remember, I change a couple of hundered > lines all over the project on a typical day... Say you work on a featureA branch. Now you get an unrelated feature request for featureB. You'd switch back to current master, and branch of featureB, starting to work on that. By the end of the day, say featureB is ready, you'd merge those branch back into master (and delete branch featureB if no longer required). If featureC is dependend on featureA, you can branch from there. If you now work again on featureA, and later on featureC, you can merge the new commits from featureA back into featureC again. Later, when featureA and C are ready, you merge them into master again. For small changes, I'd directly work on master and commit there. There is also a nice feature called git-stash, which allows to put uncommitted changes away, work temporarily on other stuff, an get back to the uncommited stuff later. I'd say, just test git and try to get a "feeling" for it. That probably helps to make a better decision. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 12:17:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 12:17:33 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> OK, let me come to a conclusion ;) What Michael writes and Raoul suggested makes an awful lot of sense. Right now, I have two problems: a) we are still having a bit of trouble with the git transistion of rsyslog - I hope to have that sorted out soon b) I need to invest some time to fully understand how git branches. The bigger problem is probably b). Thankfully, I have started to work with git on librelp and it was a very good experience. It still looks like I need to invest at least a day or two more into getting fully involved with it. That doesn't sound much, but there is a lot I can do in rsyslog in this time. I think I will proceed as follows: For the next few weeks, I'll use the scheme that I outlined this morning. It works and it is sufficiently clean for the time being. Especially as I don't see any reason for gaps, there is no such major overhaul in sight. While I do so, I'll get more acquainted to git and see how I can make utilize its branching capabilities. At some point in time (and if everything works as well as advertised, what I assume ;)), I'll switch to the git feature branch strategy. My hope is all this can be done in the next 10 weeks or so. I hope I don't disappoint anyone - but the problem is things to do. All needs to go by priorities and, quite honestly, TLS or the new config file format is higher on my priority scale than the branching strategy. And, yes, I know good knowledge with git will save in the long run. But I need to start somewhere ;) I someone has serious concerns on the route I am taking, please scream now ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, April 04, 2008 12:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > 2008/4/4, Rainer Gerhards : > > > 2008/4/4, Rainer Gerhards : > > > > > > > > 999.2.0 - stable > > > > 999.3.x - big overhaul feature, stabilizing > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > 999.9.x - devel > > > > > > > > Again... comments please ;) > > > > > > I think you really should try to use git feature branches for > that. > > > Have a stable and master (development) branch, and develop the > > > features in separate topic branches feature-A, feature-B etc. > > > Whenever one feature is ready, merge it into master. > > > This way, it doesn't matter which feature you have to concentrate > on > > > is released first (no skipped version numbers!). > > > The strong merge suppport in git would also allow to cherrypick > easily > > > from the different feature branches or merge between them. > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just a > > problem description): > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > > implement them - all while FocusFeatureX is being developed. Where > do I > > apply these? And don't I get into trouble if that interferes with > things > > that I do in FocusFeatureX? Remember, I change a couple of hundered > > lines all over the project on a typical day... > > Say you work on a featureA branch. Now you get an unrelated feature > request for featureB. > You'd switch back to current master, and branch of featureB, starting > to work on that. > By the end of the day, say featureB is ready, you'd merge those branch > back into master (and delete branch featureB if no longer required). > If featureC is dependend on featureA, you can branch from there. If > you now work again on featureA, and later on featureC, you can merge > the new commits from featureA back into featureC again. > Later, when featureA and C are ready, you merge them into master again. > For small changes, I'd directly work on master and commit there. There > is also a nice feature called git-stash, which allows to put > uncommitted changes away, work temporarily on other stuff, an get back > to the uncommited stuff later. > > I'd say, just test git and try to get a "feeling" for it. That > probably helps to make a better decision. > > Cheers, > Michael > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Fri Apr 4 17:21:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 17:21:38 +0200 Subject: [rsyslog] First rsyslog v3 stable released - 3.14.1 Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D3C@grfint2.intern.adiscon.com> Hi all, I am glad to finally announce rsyslog 3.14.1, the first version of the v3 stable branch. This version offers almost all features of the current development version to those in need of a stable branch. Among others, this includes support for multiple database backends, queued and offline operations, SNMP and text file support. It is a big step compared to the v2 stable branch. Users are advised to check the compatibility notes before the update. It's not strictly necessary, but will enable to use rsyslog in the most efficient and problem-free way. Please note that development continues inside the v3 branch. The v3 stable branch will see future feature enhancements after they have sufficiently matured. The v2 stable branch is still supported. It is quite featureless (compared to v3) but extremely solid. So if you are (or need to be) ultra-conservative, you can still take the v2 route. Feature-wise v2 is a dead end and only bug fixes will be provided. The general recommendation is that the v3 stable branch be used for regular production machines. The Fedora project will feature rsyslog v3 stable in its upcoming release 9. Please note that I made a mistake two days ago: I accidently released 3.14.0 to the web, without it being actually ready. For this reason, I have renamed the release to 3.14.1. There will never be an official 3.14.0 release. If you happen to have it downloaded and installed, please accept my apologies. You should get 3.14.1 whenever you are ready. Change Log: http://www.rsyslog.com/Article205.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-94.phtml I hope this release is useful. As always, feedback is appreciated. Best regards, Rainer Gerhards From rgerhards at hq.adiscon.com Mon Apr 7 11:21:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Apr 2008 11:21:31 +0200 Subject: [rsyslog] rsyslog on GIT now - RE: rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D51@grfint2.intern.adiscon.com> Hi folks, co-incidentally, CVS has just begun to give me some pain last Friday, when it came to actually working with the different branches I now have. And a rainy Sunday made my have a deep look at the git manual. End result: I won't wait 10 weeks from now but have started to finally convert. I have pulled Michael Biebl's git repository, which he thankfully kept synchronized with the CVS. I am doing some final checks right now, but as it looks we will be using git from now on. Rsyslog is available from Adiscon's gitweb: http://git.adiscon.com/ It can also be pulled via git protocol. Please keep in mind that it may receive some more finishing touches. I will now have four active branches: - v2-stable - v3-stable - beta -- what becomes the next release of v3-stable - master -- the current (b)leading development edge There is also v1-stable, which is deprecated and a few legacy branches. As suggested by Raoul and Michael, I'll now begin to work with feature-branches (as soon as I have finished current work in progress). I am new to git and I may mess up things. Bear with me if I do ;) Please note that I am still working on the initial setup, so if you pull the repository now, you may need to pull it again some time later ;) If I mess up, I'll let you know via the mailing list. I hope this will be a good move. Special thanks to Michael and Raoul, who were persistent enough to finally made me move (and, well, to CVS which provided the final bit of motivation ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, April 04, 2008 12:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > OK, let me come to a conclusion ;) > > What Michael writes and Raoul suggested makes an awful lot of sense. > Right now, I have two problems: > > a) we are still having a bit of trouble with the git transistion of > rsyslog - I hope to have that sorted out soon > b) I need to invest some time to fully understand how git branches. > > The bigger problem is probably b). Thankfully, I have started to work > with git on librelp and it was a very good experience. It still looks > like I need to invest at least a day or two more into getting fully > involved with it. That doesn't sound much, but there is a lot I can do > in rsyslog in this time. > > I think I will proceed as follows: > > For the next few weeks, I'll use the scheme that I outlined this > morning. It works and it is sufficiently clean for the time being. > Especially as I don't see any reason for gaps, there is no such major > overhaul in sight. > > While I do so, I'll get more acquainted to git and see how I can make > utilize its branching capabilities. At some point in time (and if > everything works as well as advertised, what I assume ;)), I'll switch > to the git feature branch strategy. My hope is all this can be done in > the next 10 weeks or so. > > I hope I don't disappoint anyone - but the problem is things to do. All > needs to go by priorities and, quite honestly, TLS or the new config > file format is higher on my priority scale than the branching strategy. > And, yes, I know good knowledge with git will save in the long run. But > I need to start somewhere ;) > > I someone has serious concerns on the route I am taking, please scream > now ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Friday, April 04, 2008 12:02 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog version numbering > > > > 2008/4/4, Rainer Gerhards : > > > > 2008/4/4, Rainer Gerhards : > > > > > > > > > > 999.2.0 - stable > > > > > 999.3.x - big overhaul feature, stabilizing > > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > > 999.9.x - devel > > > > > > > > > > Again... comments please ;) > > > > > > > > I think you really should try to use git feature branches for > > that. > > > > Have a stable and master (development) branch, and develop the > > > > features in separate topic branches feature-A, feature-B etc. > > > > Whenever one feature is ready, merge it into master. > > > > This way, it doesn't matter which feature you have to > concentrate > > on > > > > is released first (no skipped version numbers!). > > > > The strong merge suppport in git would also allow to cherrypick > > easily > > > > from the different feature branches or merge between them. > > > > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just > a > > > problem description): > > > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests > and > > > implement them - all while FocusFeatureX is being developed. Where > > do I > > > apply these? And don't I get into trouble if that interferes with > > things > > > that I do in FocusFeatureX? Remember, I change a couple of > hundered > > > lines all over the project on a typical day... > > > > Say you work on a featureA branch. Now you get an unrelated feature > > request for featureB. > > You'd switch back to current master, and branch of featureB, starting > > to work on that. > > By the end of the day, say featureB is ready, you'd merge those > branch > > back into master (and delete branch featureB if no longer required). > > If featureC is dependend on featureA, you can branch from there. If > > you now work again on featureA, and later on featureC, you can merge > > the new commits from featureA back into featureC again. > > Later, when featureA and C are ready, you merge them into master > again. > > For small changes, I'd directly work on master and commit there. > There > > is also a nice feature called git-stash, which allows to put > > uncommitted changes away, work temporarily on other stuff, an get > back > > to the uncommited stuff later. > > > > I'd say, just test git and try to get a "feeling" for it. That > > probably helps to make a better decision. > > > > Cheers, > > Michael > > > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 8 18:42:39 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Apr 2008 18:42:39 +0200 Subject: [rsyslog] rsyslog 3.17.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D85@grfint2.intern.adiscon.com> Hi all, rsyslog 3.17.0 has just been released. It is part of the development branch. The primary new feature is the ability to send email alerts based on syslog or file data. The action engine now has the ability to carry out an action only once within a configured interval and/or only during specific time frames. Option processing has been improved and a number of stability updates have been included. This is a recommended update for all users of the development version. More information on the major new feature can be found here: http://www.rsyslog.com/doc-ommail.html The following is a sample code snippet that alerts the operator when disk problems are detected: $ModLoad ommail $ActionMailSMTPServer mail.example.net $ActionMailFrom rsyslog at example.net $ActionMailTo operator at example.net $template mailSubject,"disk problem on %hostname%" $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" $ActionMailSubject mailSubject # make sure we receive a mail only once in six # hours (21,600 seconds ;)) $ActionExecOnlyOnceEveryInterval 21600 # the if ... then ... mailBody mus be on one line! if $msg contains 'hard disk fatal failure' then :ommail:;mailBody Note that we now have the ability to limit an action to be executed only once inside a specific period. In the above sample, the email alert happens only if there was no other such alert within the past 6 hours - this is absolutely vital to prevent an accidental DoS on your mailbox ;) ... but it may also be handy with other actions (e.g. SNMP trap notification etc etc). It's implemented at the action engine level, so it will work with any action, even file or database writes. Change Log: http://www.rsyslog.com/Article207.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-95.phtml I would be most interested in feedback on the new email feature, including clever use cases. I am sure it can be quite useful (especially if you think about imfile...), but would really appreciate to hear from you (and prove this in practice)! Thanks, Rainer Gerhards From aoz.syn at gmail.com Wed Apr 9 01:47:51 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:47:51 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 Message-ID: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Please forgive me if this is addressed in later versions; I'm just working against the version that was pulled into Gentoo yesterday. The atomic operations used in debug.c and msg.c were not introduced until gcc-4.1.0; as such, systems compiling with an earlier version (i.e. hardened-gentoo) fail compilation with undefined references to __sync_fetch_and_add and __sync_sub_and_fetch. I'm not sure of the right way to check for this in atomic.h, but it would be helpful if a check was made before setting DO_HAVE_ATOMICS. In addition, although msg.c makes appropriate checks for DO_HAVE_ATOMICS, debug.c does not and fails to compile under such systems. I would supply a patch, but being completely new to your codebase thought I'd start with reporting. Thanks for your time! RB From aoz.syn at gmail.com Wed Apr 9 01:53:23 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:53:23 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Message-ID: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> > The atomic operations used in debug.c and msg.c were not introduced > until gcc-4.1.0; Forgot to give specifics: the atomic operations as defined in atomic.h. The change made to bring debug.c to 1.49 in particular. From aoz.syn at gmail.com Wed Apr 9 01:55:49 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:55:49 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> Message-ID: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Strike that. Even better if I just read the ML before posting... *sigh* Long day, guys, sorry for the noise. On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > The atomic operations used in debug.c and msg.c were not introduced > > until gcc-4.1.0; > > Forgot to give specifics: the atomic operations as defined in > atomic.h. The change made to bring debug.c to 1.49 in particular. > From rgerhards at hq.adiscon.com Wed Apr 9 09:10:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:10:23 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com><4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> No problem at all, I think it's well-hidden in the ML. A good fix (via autoconf is still not done, so I should probably add a bug tracker ;). Please keep reporting. All others please note that I will release updates for v3-stable and beta soon. Its brewing ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, April 09, 2008 1:56 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Atomic operations in 3.15.0 > > Strike that. Even better if I just read the ML before posting... > *sigh* > > Long day, guys, sorry for the noise. > > On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > > The atomic operations used in debug.c and msg.c were not > introduced > > > until gcc-4.1.0; > > > > Forgot to give specifics: the atomic operations as defined in > > atomic.h. The change made to bring debug.c to 1.49 in particular. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From janfrode at tanso.net Wed Apr 9 09:33:02 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 9 Apr 2008 09:33:02 +0200 Subject: [rsyslog] git repo ? Message-ID: I just noticed "- converted to git" on http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html Is there a repo available online somewhere ? If not, maybe you could get http://github.com/ to host it so that there's a nice web interface to it and is easily forkable for others to work on it ? -jf From rgerhards at hq.adiscon.com Wed Apr 9 09:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:34:56 +0200 Subject: [rsyslog] git repo ? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8D@grfint2.intern.adiscon.com> Oops... too much going on? I thought I had posted it to the ML. Anyhow... it's at http://git.adiscon.com - git protocol is enabled Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jan-Frode Myklebust > Sent: Wednesday, April 09, 2008 9:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] git repo ? > > I just noticed "- converted to git" on > http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html > > Is there a repo available online somewhere ? If not, maybe you > could get http://github.com/ to host it so that there's a nice > web interface to it and is easily forkable for others to work > on it ? > > > > -jf > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From aoz.syn at gmail.com Wed Apr 9 17:08:22 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 9 Apr 2008 09:08:22 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> Message-ID: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> On 4/9/08, Rainer Gerhards wrote: > No problem at all, I think it's well-hidden in the ML. A good fix (via > autoconf is still not done, so I should probably add a bug tracker ;). I don't know about the remainder of the atomic changes you've made, but a check for >=gcc-4.1.0 will suffice for the stuff that's in atomic.h. I must say - I can't express how excited I was to find this project. It's been on my long-term plate for quite some time to fork syslog-ng and add all the nice features the Pro license grants; now I don't have to! I'm partial to the syslog-ng configuration syntax, but can most definitely live with this for not having to re-implement all that stuff. Tell me, are you also working on UDF-WORM? ;) From rgerhards at hq.adiscon.com Wed Apr 9 18:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 18:17:42 +0200 Subject: [rsyslog] rsyslog v3-stable/3.14.2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DA8@grfint2.intern.adiscon.com> Hi all, I have just released 3.14.2, an update to v3-stable. It is a purely bug-fixing release. Most importantly, it fixes a problem that caused imklog not to pull module symbols correctly from recent kernels. Also, a segfault caused by using expression-based filters is fixed. There are also some other fixes, see ChangeLog for details. This is a recommended update for all v3-stable branch users. Changelog: http://www.rsyslog.com/Article209.phtml Download (now pointing to the info page so that you see the md5sum ;)): http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-96.phtml I hope this release is useful. As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 9 18:51:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 09 Apr 2008 18:51:04 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> Message-ID: <1207759864.22738.94.camel@localhost.localdomain> On Wed, 2008-04-09 at 09:08 -0600, RB wrote: > On 4/9/08, Rainer Gerhards wrote: > > No problem at all, I think it's well-hidden in the ML. A good fix (via > > autoconf is still not done, so I should probably add a bug tracker ;). > > I don't know about the remainder of the atomic changes you've made, > but a check for >=gcc-4.1.0 will suffice for the stuff that's in > atomic.h. I just began, more outstanding... But it looks like that easy solution is the right one - thanks! > I must say - I can't express how excited I was to find this project. > It's been on my long-term plate for quite some time to fork syslog-ng > and add all the nice features the Pro license grants; now I don't have > to! Good to hear ;) And, believe it or not, I never ever installed syslog-ng. But,of course, it's the primary competitor in this space. IMHO we are now far ahead of it (I guess you already know that chart, but...): http://www.rsyslog.com/doc-rsyslog_ng_comparison.html Bazsi seem currently to brew something in regard to the log store, which is not (yet ;)) my priority. Also I don't think it helps to cryptographically sign the store (at least for court), the message itself must be signed (that's my route on that). So we will probably see a few features in the *paid* edition of syslog-ng that rsyslog does not yet have. Even with that, and even compared to the paid edition, I think we are now far ahead. > I'm partial to the syslog-ng configuration syntax, but can most > definitely live with this for not having to re-implement all that > stuff. rsyslog's config file syntax is just plain ugly. No way around. But wait a bit, I am working on a scripting engine: http://rgerhards.blogspot.com/2008/02/introducing-rainerscript-and-some.html http://www.rsyslog.com/doc-rscript_abnf.html > Tell me, are you also working on UDF-WORM? ;) Neither of them - just the boring stuff ;) Rainer From rgerhards at hq.adiscon.com Thu Apr 10 14:53:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 14:53:40 +0200 Subject: [rsyslog] Status of TLS development Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Hi folks, I just wanted to give you an update on my native TLS plans. I've been searching around and trying things the past days. As it looks now, I will use a wrapper class to cover nss and gnutls. Contrary to what I initially thought, I'll now probably start with GNUTls. The reason is that it seems to be much easier to start with it. For some insight, please follow this discussion on the NSS newsgroup: http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread /45e418ca0a556a42 That doesn't mean I settle for just GNUTls. Even if I start with it, NSS support will follow. I hope it will be easier to implement when I can check the NSS manuals for specific functionality. I thought I let you know. Rainer From mbiebl at gmail.com Thu Apr 10 15:29:06 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:29:06 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Hi folks, > > I just wanted to give you an update on my native TLS plans. I've been > searching around and trying things the past days. As it looks now, I > will use a wrapper class to cover nss and gnutls. Contrary to what I > initially thought, I'll now probably start with GNUTls. The reason is > that it seems to be much easier to start with it. For some insight, > please follow this discussion on the NSS newsgroup: > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread > /45e418ca0a556a42 > > That doesn't mean I settle for just GNUTls. Even if I start with it, NSS > support will follow. I hope it will be easier to implement when I can > check the NSS manuals for specific functionality. > > I thought I let you know. Is it technically necessary to implement the layer via a separate library (libsci), or could the layer also be implemented directly within rsyslog as (libtool) convenience lib? The point is, that a separate library means more works for packagers and people who want to compile rsyslog themselves. If you think, that libsci might be useful for other projects, then implementing it as a system library makes sense (or due to licensing matters). If libsci will only be used within rsyslog though, I think implementing it as a libtool convenience lib within rsyslog makes more sense. Just my 2? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:31:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:31:38 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Michael, I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Depending on how it works out, it may be useful for others, too. But that's not the primary focus. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Hi folks, > > > > I just wanted to give you an update on my native TLS plans. I've > been > > searching around and trying things the past days. As it looks now, I > > will use a wrapper class to cover nss and gnutls. Contrary to what I > > initially thought, I'll now probably start with GNUTls. The reason > is > > that it seems to be much easier to start with it. For some insight, > > please follow this discussion on the NSS newsgroup: > > > > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/threa > d > > /45e418ca0a556a42 > > > > That doesn't mean I settle for just GNUTls. Even if I start with it, > NSS > > support will follow. I hope it will be easier to implement when I > can > > check the NSS manuals for specific functionality. > > > > I thought I let you know. > > Is it technically necessary to implement the layer via a separate > library (libsci), or could the layer also be implemented directly > within rsyslog as (libtool) convenience lib? > > The point is, that a separate library means more works for packagers > and people who want to compile rsyslog themselves. > > If you think, that libsci might be useful for other projects, then > implementing it as a system library makes sense (or due to licensing > matters). > If libsci will only be used within rsyslog though, I think > implementing it as a libtool convenience lib within rsyslog makes more > sense. > > Just my 2? > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:38:18 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:38:18 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Michael, > > I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be fine with rsyslogs GPLv3). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:40:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:40:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go to link to GPLv3. Sorry... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Michael, > > > > I fully agree, but licensing is the culprit. I think it'll be either > LGPL or BSD. I'd really prefer to plug it directly into librelp and > rsyslog, but that will create license conflicts. > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > fine with rsyslogs GPLv3). > > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Thu Apr 10 15:43:01 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:43:01 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Bad typo. Should have been: librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go to link to GPLv3. Sorry... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, April 10, 2008 3:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go > to link to GPLv3. Sorry... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Thursday, April 10, 2008 3:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Status of TLS development > > > > 2008/4/10, Rainer Gerhards : > > > Michael, > > > > > > I fully agree, but licensing is the culprit. I think it'll be > either > > LGPL or BSD. I'd really prefer to plug it directly into librelp and > > rsyslog, but that will create license conflicts. > > > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > > fine with rsyslogs GPLv3). > > > > Cheers, > > Michael > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:50:34 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:50:34 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Bad typo. Should have been: > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > to link to GPLv3. Sorry... GPLv3+/commercial is not incompatible with LGPLv2.1+ Am I missing something? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:56:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:56:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: <1207835772.22738.132.camel@localhost.localdomain> On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > 2008/4/10, Rainer Gerhards : > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ If I put the crypto wrapper (code named so far libsci) into *rsyslog* the crypto wrapper will be under GPLv3. If librelp needs TLS functionality, it must link to the crypto-wrapper *inside* rsyslog, which is GPLv3. librelp can not do that if it is not used inside a GPLv3 program (because, in essence, I would need to detouch those parts from rsyslog that are needed by librelp and then add that to the non-GPLv3 application). Or am I overlooking something? A prominent example of what will use librelp functionality and is not under GPLv3 is Adiscon's MonitorWare products under Windows (and keep in mind that they currently sponsor rsyslog development ;)). Rainer From mbiebl at gmail.com Thu Apr 10 15:55:45 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:55:45 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Michael Biebl : > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > Am I missing something? The whole point of the LGPL is, to be useful for commercial/closed source applications. And GPLv3+ and LGPLv2.1+ is no problem either. I'm not a lawyer though, so take this with a grain of salt. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Apr 10 16:01:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 16:01:56 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <1207835772.22738.132.camel@localhost.localdomain> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: 2008/4/10, Rainer Gerhards : > > On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > > > to link to GPLv3. Sorry... > > > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > > If I put the crypto wrapper (code named so far libsci) into *rsyslog* > the crypto wrapper will be under GPLv3. > > If librelp needs TLS functionality, it must link to the crypto-wrapper Ok, this was probably my misconception. I thought, only rsyslog would need the TLS functionality. It wasn't clear to me, that *both* rsyslog and librelp would link against libsci. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 16:34:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 16:34:33 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: <1207838073.22738.144.camel@localhost.localdomain> On Thu, 2008-04-10 at 16:01 +0200, Michael Biebl wrote: > Ok, this was probably my misconception. > I thought, only rsyslog would need the TLS functionality. > It wasn't clear to me, that *both* rsyslog and librelp would link > against libsci. Sorry, probably I've been to brief. Let me sort out things at least now: Both rsyslog and librelp need native TLS functionality. For rsyslog, this is to support plain TCP syslog via TLS (hopefully compatible with the like syslog-ng and MonitorWare feature) as well as the upcoming IETF syslog-transport-tls standard. Librelps needs to do native TLS because it can not rely on an external entity (like stunnel) and claim to be reliable and pretty tamperproof. So it must talk to the remote peer directly. In RELP protocol, there will be a STARTTLS command verb, just like in SMTP. Negotiation on the availability of this verb is similar to SMTP's EHLO message or BEEP's greeting message. Both the RELP protocol as well as librelp already contain the necessary provisioning. My intention is to make libsci available to anyone who may have a need for it, but it will be focused on what I actually need. Please note that I have intentionally called it libsci - simple *crypto* interface - and not something like libsti - simple *tls* interface - because I intend to place all crypto functions into a single library (thus relieving at least a bit of the burden from an extra library). Other than TLS, rsyslog will need to support digital signatures as well as generic hashes (not only for cryptographic needs, but also for some other nice things). If JF finally convinces me to implement some of the advanced output plugin functionality as completely separate projects [he is seeding that thought quite well ;)], these projects will most probably also need to link to it. My intention is to put libsci under a very liberal license, probably either a BSD style one or at least LGPL. I have no idea yet when we will see any of libsci functionality in practice ;) I hope this finally clarifies. Sorry for the confusion. Please drop me any questions that are still around. Rainer From rgerhards at hq.adiscon.com Fri Apr 11 10:04:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Apr 2008 10:04:12 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... Message-ID: <1207901052.22738.230.camel@localhost.localdomain> Hi folks, I am sorry, this will be a long mail. But I would appreciate if you'd read it in full and comment on it. This mail covers a really important decision for rsyslog and will probably even influence if the project succeeds in the long term. Package maintainers and code contributors are especially requested to *really* read it. Though I t try hard to provide all relevant facts (that's why it is getting long), I will probably miss some and not properly convey others. Please feel free to ask. Let me start with explaining that the rsyslog project conceptually consists of three parts: - the modules - "helper" functions - rsyslog-specific functionality Modules are actually projects in their own right, just being distributed with the rsyslog tarball for convenience. A module may be released under any license. Note that modules call both rsyslog-specific functionality (e.g. to submit a message) as well as helper function (e.g. to handle tcp sessions). The "helper" functions are a growing set of generic objects. Examples are the module loader, the queue engine, networking support, the script engine and virtual machine, ... - you get the idea: Things that are used inside rsyslog but are not necessarily of use only for rsyslog. Actually, this could be called a "rsyslog runtime library". Rsyslog-specific functionality is primarily rsyslogd and everything it takes to glue together helpers and plugins to build the working syslog subsystem. Let's stick with that for a brief while. Now let me explain the idea of loosely coupled modules. This stems back to JF's effort to convince me to the Unix philosophy of "small tools that work well together". We had another good discussion yesterday (on the blog) and it made me change my mind a bit (well, probably, not 100% convinced yet, but it he managed to seed the thought ;)). While I still think that there are things that need to be really tightly coupled to the rsyslog core, there are others which not necessarily need to. Let me call the later "loosely coupled modules", in contrast to the (tightly coupled) plugins that actually become part of the rsyslogd process during runtime. The analysis plugins I have on my mind could become such loosely coupled modules. As an interface, the usual Unix "send it and forget it" pipe could be used, and it would probably be acceptable to allow for minor message loss during shutdown and plugin failure (anything else would require a pipe application protocol, e.g. relp over pipe, which sounds scary). The plus in doing so would be the ability to use those plugins in configurations where rsyslog is not present (e.g. driven by syslog-ng or a detached message generator [fed from a log file]). Done right, one could even select the (sligly lossy) pipe interface or the full blown plugin interface as a compile time switch. If you think it out, we may even end with an abstraction layer where each module can be compiled for either the plugin interface or the pipe (no promises, though). One problem with this approach is that modules call into the rsyslog helpers. For example, rsyslog's network support need to be available for all those modules that do something over the net. That's not a problem if I have a tightly coupled plugin as today (the rsyslog core makes the necessary bindings). It would become more problematic if I move the module to a pipe interface, because I now need to find a way to use the rsyslog objects. But that's still doable (though pretty ugly). It becomes really problematic if the same module, using a pipe interface, is to be used with e.g. syslog-ng. I don't think that syslog-ng will be able to provide it with an emulated rsyslog "net" object. Let's stick with this problem for a moment. Coincidentally, we had another discussion on the mailing list yesterday - on the TLS support wrapper for rsyslog and librelp. That discussion centered around licenses. Technically, there are also a number of issues. I have now involved myself enough with GnuTLS and a bit of NSS so that I am able to try draft a first abstraction layer. I thought hard and the "right solution" involves encapsulation stream network access. So the right thing to do is to have one object that handles network streams. That object then is configured to use either plain tcp, TLS (via whatever library) or even GSS-API. Nice and clean. It gets dirty if I think about the details. If I do it that way, it makes rsyslog depend on this object (so-far codenamed libsci). However, that would mean that any rsyslog installation would need to pull in libsci. Not a big deal, except, right, except if the crypto libraries are also pulled in by libsci. So would every desktop system running rsyslog need to have the crypto libraries installed? Scary... unacceptable. In rsyslog, we had the same problem a few month ago (at that time the mysql client libraries were the problem). The solution was the rsyslog loader, which dynamically loads other libraries (and their dependencies) on demand. The loader is what enables rsyslogd to be installed everywhere, but only with minimal core requirements (and have everything else in separate packages). So if libsci would be part of rsyslog, we would not have any problem at all. After all, the necessary plumbing is ready at hand in form of the rsyslog helper objects. This is where we come back to loosely coupled modules. You notice it is the same problem? Both them as well as libsci would need to call the rsyslog helpers. Now let's come a bit to licensing. In order to understand that, we need to talk about rsyslog funding first. Obviously, I am spending full time (and a bit more than that) on rsyslog for quite a while now. I even intend to do that for some more months as rsyslog is currently mabye 55% of what I would like it to be. Somehow I must get funding - for the time, for the hardware and for all the other things ;) What made the rsyslog project possible, and still 99.9% funds it is Adiscon, the company that provides (of course ;)) the best-ever logging solutions on Windows. Actually the Windows closed source pays for the rsyslog project. While we hope to find other sources of funding in the future, I can not ignore the fact. Once thing we would like to do at Adiscon is include select parts of the technology I am now developing into the closed source applications, too. The most prominent example is the RELP protocol. I obviously find this a fair policy - after all, the alternative would be to do it in closed source only and I was able to convince my folks at Adiscon that it is far better to contribute to the open source world. There is one drawback in this requirement: licensing. Of course, we could pick a BSD style license and every problem would be solved. But, I have to admit, we do not like to give everything to our competitors in the *closed source* space. We have made very bad experiences with folks building on our technology and even turning it against us. I won't get agreement from Adiscon to use a BSD license for everything (plus I personally, too, don't like to see that effect). We already discussed this on the mailing list here as part of dual-licensing in the past. The solution was that the technology in question was created as its own, dual-licensed, project. This lead to the creation of librelp. Rsyslog itself was left under GPLv3 (which I sincerely believe in because of its anti-patent, anti-drm clauses - even though the license gives myself obviously some troubles). Dual-licensing librelp lead to some duplicate code and made me not use some features which I could have used if I had access to all rsyslog helper objects. For librelp, that is not yet a big deal, because it is quite unique, very few needs to access the rsyslog helpers. With TLS, however, the situation changes and we get the dangling issue of rsyslog helpers in librelp, too. LETS TRY TO WRAP-UP If I put all of this together, I think I have taken a (slightly? ;)) wrong path. The core problem is monolithic design from a very high point of view. I have to admit I think this is what JF and some others were pointing out, but I didn't realize it quickly enough. Sure, rsyslog is quite modular by now. But rsyslog always requires rsyslog to do everything. It is very hard to do any rsyslog-related work without the rsyslog core. While rsyslog has a carefully crafted set of helper objects, these are not exposed to the outside world. And the licensing issues associated with that design begin to screw up everything in the long run. I think we need change. The obvious solution seems to be extracting the rsyslog helpers out of the rsyslog core project and create a "rsyslog runtime". That runtime than could individually be installed and be put under a different license (bear with me, explanation follows below). Let's consider a complicated case with the runtime. Assume we have a plugin "NeverBeforeSeenAnalysis". Let's say someone wants to use it with syslog-ng (!). With the runtime, all needed would be to compile it for the pipe interface and install rsyslog-runtime and the module onto the system. Now let's consider Adiscon's MonitorWare products on Windows. When they implement RELP, they need librelp and can pull in the rsyslog-runtime (for network access including TLS). For rsyslogd itself nothing really happens, the runtime is now just its own library - linking to it needs not to be modified. So for rsyslogd, the change would be transparent. Technically, this indeed solves the issues. Let me stress the point that it leads to code reuse, where I currently need to rewrite things (which increasingly concerns me, especially from a maintenance point of view). Now, on to the licensing. Obviously, the MonitorWare use case above would be totally incompatible with GPLv3. So the rsyslog-runtime would need to be under a different license. It could be dual-licensed, but I think that would probably do more bad than good. I think I can convince Adiscon to go with LGPL for the runtime part. Granted, it introduces risk of closed source competitors pulling it in, but the advantages should outweigh this risk. >From the ability to put this work under a different license, I think I am in good shape: most of the helper objects are freshly written and have only received limited patches (if at all) from contributors. I can contact them and ask for permission to change the license. Where I don't get permission, I think I can re-implement the contribution. Again, most of the code in question has been written in the past 4 month and is 99.999% non-contributed. There may be some few runtime objects which stem back to sysklogd. There, a license change is impractical. I'll have to life with the fact that those can not go into a re-licensed runtime. Depending on how important the functionality is, I either need to rewrite or drop it (for non-rsyslogd use). In any case, this looks (pending detail analysis) quite possible. Big question number one is what you think of this runtime approach? Have I overlooked something? Do you object it for some reason? If so, which? The next question is how to package this inside the source tree. Remember, currently rsyslog and the plugins (considered separate projects) are all packed together inside a single tarball. This is very convenient, both for me as well as for package maintainers and users. The question is if we split rsyslog into the rsyslogd and the rsyslog-runtime, will we continue to deliver the runtime as part of the rsyslog package? Or would it be better to move it to a librsyslog project? Other than with the plugins, we actually would have two different licenses, so it may be confusing to have both of it in the same project (but I have seen that GnuTLS uses exactly this approach, with the main library being LGPL and the - included - extras library GPL only). So that's the next question, obviously depending on the first: how to pack projects if we do a runtime split? I know this is a long and dense mail. My apologies for this. But I think the discussion is needed. I honestly believe that a number of discussions in the past weeks actually circled around this theme, we just didn't actually get down to the point. Please note that I will hold TLS development until we have reached consensus on the runtime/licensing topic. The reason is if we don't do a runtime split, I need to do things considerable different than when we do one (much more code, probably yet another external library). So, obviously, I have a current bias towards the split. However, experience shows that I (as everyone ;)) tend to overlook or misunderstand things. Thus your feedback is so important. I don't like the idea of jiggling back and forth on such an important topic as licensing and high-level modularization, so I would like to get it now done in "the right way" and keep it stable for at least the foreseeable future. Given the fact that the decision somehow affects rsyslog's development as whole, I would even appreciate quick feedback. In this spirit: please let the comments flow ;) Thanks, Rainer From friedl at hq.adiscon.com Fri Apr 11 11:24:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Fri, 11 Apr 2008 11:24:20 +0200 Subject: [rsyslog] rsyslog 3.15.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DD6@grfint2.intern.adiscon.com> Hi all, rsyslog 3.15.1 has been released. This is a refresh of the beta branch, providing new bug fixes. The beta branch currently features the RELP protocol and will be the next v3-stable once it has sufficiently matured. Changelog: http://www.rsyslog.com/Article210.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-97.phtml As always, feedback is appreciated. Florian Riedl From aoz.syn at gmail.com Sat Apr 12 00:10:04 2008 From: aoz.syn at gmail.com (RB) Date: Fri, 11 Apr 2008 16:10:04 -0600 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <1207901052.22738.230.camel@localhost.localdomain> References: <1207901052.22738.230.camel@localhost.localdomain> Message-ID: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Read once, digested for a few hours, read again. *whew* A monolithic message about a monolithic problem... ;) Also please understand I'm new here and am not completely familiar with your codebase, so may well miss the mark. Do I understand you correctly that part of your concern with looser coupling is the higher potential for lossiness? I don't think I can afford a line-by-line analysis at the moment, but got a persistent thought both times through: if you're considering pursuing a loosely-coupled model, why not consider going all the way and treat rsyslogd as a dispatch/routing agent among the various modules? That's by no means a simple solution, but I wonder why certain pieces of functionality must be tightly coupled, either to each other (network/crypt) or to the core itself (network). I *think* it becomes even more interesting when you start thinking about ways to 'stack' modules. Don't feel qualified to even start answering the licensing/packaging stuff, so will leave that be. From rgerhards at hq.adiscon.com Mon Apr 14 08:58:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 08:58:50 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> References: <1207901052.22738.230.camel@localhost.localdomain> <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Message-ID: <1208156330.22738.252.camel@localhost.localdomain> On Fri, 2008-04-11 at 16:10 -0600, RB wrote: > Read once, digested for a few hours, read again. *whew* A monolithic > message about a monolithic problem... ;) Also please understand I'm > new here and am not completely familiar with your codebase, so may > well miss the mark. > > Do I understand you correctly that part of your concern with looser > coupling is the higher potential for lossiness? Yes, there are two issues with that: increased lossiness (due to less knowledge inside rsyslog core of what happens & the same problem plain tcp has) and performance. A pipe interface is much slower than the native in-process plugin interface - this can turn out to become a problem in high performance big enterprise deployments. HOWEVER, all of this can be solved if the module can be compiled to utilize either one of both interfaces. I have also thought more about this and it looks like the absolutely perfect solution to all needs (the bottom line is: the user decides what to use and has all options at hand). > > I don't think I can afford a line-by-line analysis at the moment, but > got a persistent thought both times through: if you're considering > pursuing a loosely-coupled model, why not consider going all the way > and treat rsyslogd as a dispatch/routing agent among the various > modules? Actually, this is the approach rsyslog takes. Though this page is a bit old (and generic), it still covers the base ideas: http://www.rsyslog.com/doc-generic_design.html This document was originally written for an IETF discussion, so it is not rsyslog-specific (but given the same authorship... ;)). > That's by no means a simple solution, but I wonder why > certain pieces of functionality must be tightly coupled, either to > each other (network/crypt) or to the core itself (network). I *think* > it becomes even more interesting when you start thinking about ways to > 'stack' modules. The most recent rsyslog loader supports stacking. We now have a call where an object can request it to find other objects, which can lead to an object load request if the object is not yet in core. Of course, if a new object is loaded, it can request the loader to find other objects it needs... Run a recent build of rsyslogd with -d -n and you'll see these calls in action ;) However, not all of the rsyslog modules are full blown objects as of now. Converting them is part of the ongoing modularization effort. But this also touches the core of my long message: the loader and all helpers currently live in rsyslogd core ONLY. They are not usable by any entity that is not either part of rsyslog or being *called by rsyslog* (plugins). So it is currently impossible to use any of rsyslog's technology without rsyslogd. Among others, this prevents loosely coupling modules (because the module would need access to the runtime objects without rsyslog). It also prevents cases where someone wants to use parts that depend on rsyslog technology, but without using rsyslogd itself. While, of course, I do not find that smart, it may be just the right thing for the user in question. I intend to no longer limit the user in his options, freeing the code base even more. This is both a technical issue as well as a licensing one. The solution, the more I think about it, is really obvious: create a runtime system and place it under LGPL. That'll solve all issues and make rsyslog modular not only on a detail level but also from the high-level perspective. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 12:46:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 12:46:23 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux Message-ID: <1208169983.22738.290.camel@localhost.localdomain> Hi all, I am currently working on the imklog module. I noticed that - for years now - internally generated messages of this module are also emitted with the "kernel" facility. For example, the startup message: imklog 3.17.1-bsd, log source = /proc/kmsg started. rsyslog has taken over that behavior from the sysklogd project, so it is really ages old. HOWEVER, I personally think this is the wrong facility. IMHO the syslogd facility would be the right one, as this message is not from the kernel log but instead locally generated by the syslogd (the syslog subsystem, to be precise). On BSD, no similar problem exists so far, because there no kernel message was added. At the current snapshot, I add few messages if things go wrong. Actually, this triggered the question on how to handle internally generated messages... Question now: shall I change that to LOG_SYSLOG? Or should it remain as is? Final option: a config switch where the user can configure which facility to use (that would then default to LOG_SYSLOG IMO). Feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 14:20:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 14:20:52 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux In-Reply-To: <1208169983.22738.290.camel@localhost.localdomain> References: <1208169983.22738.290.camel@localhost.localdomain> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E00@grfint2.intern.adiscon.com> Hi all, I have thought a bit more about it. I ended up with a configuration variable, but the default is depending on the klog driver. So on Linux, we continue to log those messages as kernel, while on BSD we continue to log them as syslogd. The default can be overwritten by the user. However, I still wonder if it really is desirable to log the messages as kernel on Linux. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, April 14, 2008 12:46 PM > To: rsyslog-users > Subject: [rsyslog] Facility of imklog-generated messages on linux > > Hi all, > > I am currently working on the imklog module. I noticed that - for years > now - internally generated messages of this module are also emitted > with > the "kernel" facility. For example, the startup message: > > imklog 3.17.1-bsd, log source = /proc/kmsg started. > > rsyslog has taken over that behavior from the sysklogd project, so it > is > really ages old. HOWEVER, I personally think this is the wrong > facility. > IMHO the syslogd facility would be the right one, as this message is > not > from the kernel log but instead locally generated by the syslogd (the > syslog subsystem, to be precise). > > On BSD, no similar problem exists so far, because there no kernel > message was added. At the current snapshot, I add few messages if > things > go wrong. Actually, this triggered the question on how to handle > internally generated messages... > > Question now: shall I change that to LOG_SYSLOG? Or should it remain as > is? Final option: a config switch where the user can configure which > facility to use (that would then default to LOG_SYSLOG IMO). > > Feedback is appreciated. > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 15 09:41:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 09:41:56 +0200 Subject: [rsyslog] some rsyslog files under LGLP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E12@grfint2.intern.adiscon.com> Hi all, I will now begin to change the license for some rsyslog files from GPLv3 only to LGPLv3. I have contacted the major contributors and will contact others on an as-needed basis. The license change will probably take some time. Rainer From rgerhards at hq.adiscon.com Tue Apr 15 10:51:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 10:51:45 +0200 Subject: [rsyslog] rsyslog runtime library Message-ID: <1208249505.22738.309.camel@localhost.localdomain> Hi all, I have begun to lay the foundation for a well-defined rsyslog runtime library. I had a couple of discussions and the current (still open to change ;)) strategy is to build gradually build the runtime library as part of the main rsyslog project. This resembles the way plugins, also individual projects, are shipped together in the main tarball. What convinced me to work on this route, at least for now, is some technical advantages. Using this approach, I can incrementally work on the runtime library. While it is useful to have, it requires quite some effort to build a clean, abstracted runtime library. Remember the main effort for v3 (which will lead to v4) is to create a fully modular rsyslog subsystem. I started from the quite monolithic v2 source based and have already extracted a great deal of objects. But this is not yet completed - I do this work in parallel to introducing new features and when it best fits. Similarly, splitting off the runtime library right now would mean I stopping feature development for a few weeks, finally cleaning up the object model and putting it all nicely together. While it obviously sound like the right thing from the software engineering point of view, it sounds pretty bad from a feature perspective (bringing everything to a halt plus introducing many new, untested, bug possibilities). So I now believe the right route is again a compromise: I continue to add new features, keeping the runtime idea in mind and contributing to it whenever it is a good time to do so. For v4, of course, we should have a full blow abstraction, because then we finally have the clean object model. So I will spend some limited time on initial creation of a skeleton library, but will continue to work on rsyslog features on a fast pace. I honestly believe this will lead to the overall best result. Feedback is appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Tue Apr 15 15:24:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 15:24:38 +0200 Subject: [rsyslog] rsyslog 3.17.1 (devel) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E1D@grfint2.intern.adiscon.com> Hi all: Rsyslog 3.17.1 has just been released. The primary new feature is full BSD platform support. Rsyslog now not only compiles cleanly on BSD, but also now supports BSD's kernel log natively. In general, kernel logging has been improved and better documented. High-precision timestamps for the kernel log have been enabled. There are also some minor other enhancements as well as a couple of bug fixes. Rsyslog 3.17.1 is a recommended update for all users of the devel branch. Changelog: http://www.rsyslog.com/Article213.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-98.phtml As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 16 17:27:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Apr 2008 17:27:04 +0200 Subject: [rsyslog] Syslog Traffic Data & Alerting Message-ID: <1208359624.22738.353.camel@localhost.localdomain> Hi all, I am currently thinking about a traffic-volume based approach to detect unusual situations and issue alerts. This is in the context of advanced analysis output plugins (which will later also be able to run standalone). I have some ideas, but in order to refine them, I need some real-world syslog traffic data. I am in need of the volume of syslog messages per second, as it evolves over multiple days (preferably over a period of one or two month - and to get started a week worth of this data). I wonder if anyone of you would be interested in providing me such a traffic sample. The data I need would be totally anonymous. All I need is a file with one line per second: timestamp and number of messages per second (no information about message content or any other message property). However, these two fields must be correct and not be any further processed. After all, the root idea is to detect something from the patterns and mangled ones would be very counter-productive. If there is interest (multiple contributors would be happily accepted), I would write an output plugin for rsyslog that gathers this data. It would be very useful to have at least one high-volume environment inside the sample. Please let me know if you could contribute to this effort. Your help would be deeply appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 18 18:38:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 18 Apr 2008 18:38:06 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon Message-ID: <1208536686.22738.360.camel@localhost.localdomain> Hi folks, of course, rsyslog can work with any web interface. Nevertheless, we have begun to rewrite our phpLogCon in the hope that it will be an even useful tool. From Andre, the main project author, I have heard that we will probably see an initial v2 release next week. He has also set up a demo server, based on today's codebase: http://demo.phplogcon.org/ I would appreciate if you could drop by and provide us some feedback. It's the first time phpLogCon v2 is visible online and I we are very interested to learn what people like and what not. Of course, the project will expand much in the next few month, and you can help drive its direction. However, I think it is quite useful as it currently is. We have pushed some limited test data into the demo. If someone would like to try it out on his/her own system, please let us know. We will help you get started for sure :) Thanks, Rainer From bakers at web-ster.com Fri Apr 18 18:46:37 2008 From: bakers at web-ster.com (Scott Baker) Date: Fri, 18 Apr 2008 09:46:37 -0700 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808D06D.5000309@web-ster.com> Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer My first response is... WOW! I've never seen that web interface before but that's pretty impressive. I'm much more of a CLI/grep kinda guy but I can see a lot of PHBs loving this. It's very impressive and seems to work quite well. Only complaint is that the date column wraps to two lines, but that's just cosmetic. Kick ass. Keep up the good work. -- Scott Baker - Canby Telcom RHCE - System Administrator - 503.266.8253 From r.bhatia at ipax.at Fri Apr 18 21:18:47 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 18 Apr 2008 21:18:47 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808F417.2000805@ipax.at> hi rainer, looks pretty decent. i pulled the git repository and will have a further look ;) as far as i can tell by now, i like modular design where you can fetch data from different streams. cheers, raoul Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From lanas at securenet.net Wed Apr 23 01:31:00 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 19:31:00 -0400 Subject: [rsyslog] rsyslog in general Message-ID: <20080422193100.2b2a09a1@mistral.stie> Hello, I'd be interested to use rsyslog in a production environment. On one side I have the impression that it is a stable product but then if I look at the releases, they are quite numerous over a small amount of time. I would like to know what is the average count of bug fixes in general and to which severity. Thanks, Al From mbiebl at gmail.com Wed Apr 23 02:37:15 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 02:37:15 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422193100.2b2a09a1@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: 2008/4/23, lanas : > Hello, > > I'd be interested to use rsyslog in a production environment. On one > side I have the impression that it is a stable product but then if I > look at the releases, they are quite numerous over a small amount of > time. I would like to know what is the average count of bug fixes in > general and to which severity. There are frequent releases of devel "snapshots". For stable releases you should check out http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml (not sure if you are already aware of that) Or do you also have concerns about the frequency of stable releases? Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Wed Apr 23 02:54:47 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 20:54:47 -0400 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: <20080422205447.0772e890@mistral.stie> Le Mercredi, 23 Avril 2008 02:37:15 +0200, "Michael Biebl" a ?crit : >> I'd be interested to use rsyslog in a production environment. On >> one side I have the impression that it is a stable product but then >> if I look at the releases, they are quite numerous over a small >> amount of time. I would like to know what is the average count of >> bug fixes in general and to which severity. > There are frequent releases of devel "snapshots". > For stable releases you should check out > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > (not sure if you are already aware of that) Thanks. The package I ended up with was version 3.17.1. What about this version ? It is not tagged as beta, although it is development. What is the schedule for a stable release of that version ? I see that the page you gave lists 3.14 as stable. > Or do you also have concerns about the frequency of stable releases? Well, to have a grasp at how the development is planned and one would be great. Cheers, Al From mbiebl at gmail.com Wed Apr 23 03:16:27 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 03:16:27 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422205447.0772e890@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> <20080422205447.0772e890@mistral.stie> Message-ID: 2008/4/23, lanas : > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > "Michael Biebl" a ?crit : > > > >> I'd be interested to use rsyslog in a production environment. On > >> one side I have the impression that it is a stable product but then > >> if I look at the releases, they are quite numerous over a small > >> amount of time. I would like to know what is the average count of > >> bug fixes in general and to which severity. > > > There are frequent releases of devel "snapshots". > > > For stable releases you should check out > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > (not sure if you are already aware of that) > > > Thanks. The package I ended up with was version 3.17.1. What about > this version ? It is not tagged as beta, although it is development. > What is the schedule for a stable release of that version ? I see that > the page you gave lists 3.14 as stable. 3.17.1 is from the devel branch, so I wouldn't recommend it for production use. The version numbering has been in discussion not long ago, so take the following with a grain of salt (Rainer, the lead author, will probably have additional information). 2.0.x is the old-stable branch (the current version being 2.0.4). If you want something ultra-stable, without the need for some of the nice feature of the v3 branch, take this one. The first stable release of the v3 branch was 3.14.1 (the current version being 3.14.2). If you need some of the features of the v3 branch, go with 3.14.2. Stable versions are even numbered (second digit). See also http://www.rsyslog.com/doc-status.html The devel branch is usually for bleeding edge stuff and has a odd numbered second digit. The beta branch is for stabilizing features, resulting in the next stable release. I also has a odd numbered second digit. > > > > Or do you also have concerns about the frequency of stable releases? > > > Well, to have a grasp at how the development is planned and one would > be great. > http://www.rsyslog.com/doc-status.html http://www.rsyslog.com/doc-features.html http://www.rsyslog.com/doc-v3compatibility.html http://www.rsyslog.com/doc HTH, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Apr 23 08:20:58 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Apr 2008 08:20:58 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie><20080422205447.0772e890@mistral.stie> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EB0@grfint2.intern.adiscon.com> Hi there, Michael has done an excellent job of summing up things :) Just let me add a few thoughts. First of all, the versioning document needs to be updated and I'll do that soon ;) The frequent releases are a result of rapid development. There is lots to do and we do it quickly. This, of course, introduces bugs. To develop rapidly and still provide stable releases, we have settled on this for now: The v2-stable is a feature-dead end. It receives patches only. Ultra-stable, quite featureless. V3-stable primarily receives patches but from time to time new functionality is integrated. This functionality ripens in the beta branch. The devel version has the newest code and most bugs. It is NOT recommended for production use. At some point in time, beta is branched off devel. Beta than receives bug fixes, but no new features. After some time (between 3 and 10 weeks, depending on bug reports), beta becomes v3-stable. Even inside stable or beta or devel there are several levels of maturity - some code is unmodified for more than a year, while other just recently made it into stable. I have recently begun to add some information about when features have been introduced into a version. The general rule is that the newer a feature is, the more likely it contains bugs. In general, rsyslog is rock-solid. However, some so far seldomly-used features (expression based filters are a good example) has seen limited exposure to practice. Such features are more likely to have problems. If you try judge project and feature stability, it may be a good idea to look at the bugzilla and the code commits: http://bugzilla.adiscon.com/buglist.cgi?cmdtype=runnamed&namedcmd=rsyslog-all http://git.adiscon.com/?p=rsyslog.git;a=summary You may also want to have a look at my blog, where I post anything of note. It's also a good place to see what is going on and actually even to see what is giving me a hard time every now and then: http://rgerhards.blogspot.com Finally, if you let me know what you intend to do, I can provide you my personal view of how stable the required features are. But as I said - it's pretty stable now. Rsyslog also recently made it into Red Hat Enterprise Linux (5.2) and you probably know that these folks are quite conservative when it comes to stability ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, April 23, 2008 3:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in general > > 2008/4/23, lanas : > > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > > "Michael Biebl" a ?crit : > > > > > > >> I'd be interested to use rsyslog in a production environment. > On > > >> one side I have the impression that it is a stable product but > then > > >> if I look at the releases, they are quite numerous over a small > > >> amount of time. I would like to know what is the average count > of > > >> bug fixes in general and to which severity. > > > > > There are frequent releases of devel "snapshots". > > > > > For stable releases you should check out > > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > > (not sure if you are already aware of that) > > > > > > Thanks. The package I ended up with was version 3.17.1. What about > > this version ? It is not tagged as beta, although it is > development. > > What is the schedule for a stable release of that version ? I see > that > > the page you gave lists 3.14 as stable. > > 3.17.1 is from the devel branch, so I wouldn't recommend it for > production use. > The version numbering has been in discussion not long ago, so take the > following with a grain of salt (Rainer, the lead author, will probably > have additional information). > > 2.0.x is the old-stable branch (the current version being 2.0.4). If > you want something ultra-stable, without the need for some of the nice > feature of the v3 branch, take this one. > > The first stable release of the v3 branch was 3.14.1 (the current > version being 3.14.2). If you need some of the features of the v3 > branch, go with 3.14.2. > > Stable versions are even numbered (second digit). > > See also > http://www.rsyslog.com/doc-status.html > > The devel branch is usually for bleeding edge stuff and has a odd > numbered second digit. > The beta branch is for stabilizing features, resulting in the next > stable release. I also has a odd numbered second digit. > > > > > > > > Or do you also have concerns about the frequency of stable > releases? > > > > > > Well, to have a grasp at how the development is planned and one would > > be great. > > > > http://www.rsyslog.com/doc-status.html > http://www.rsyslog.com/doc-features.html > http://www.rsyslog.com/doc-v3compatibility.html > http://www.rsyslog.com/doc > > HTH, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From friedl at hq.adiscon.com Thu Apr 24 14:36:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 24 Apr 2008 14:36:20 +0200 Subject: [rsyslog] rsyslog 3.16.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Hi all, I have just released rsyslog 3.16.0, a v3-stable release. This release brings a major new feature, RELP support, to the stable branch. RELP enables lossless transmission of syslog messages, in cases where even plain tcp syslog fails. Other than that, the release contains a few bugfixes. Users of the v3-stable branch are advised to update to the new release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-99.phtml Changelog: http://www.rsyslog.com/Article215.phtml Florian Riedl From rgerhards at hq.adiscon.com Fri Apr 25 07:49:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 07:49:53 +0200 Subject: [rsyslog] phpLogCon v2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ED7@grfint2.intern.adiscon.com> Hi all, I'd like to pass the info that phpLogCon v2's initial release is now available. I thought a while if I should use this list to announce it, but came to the conclusion that it is justified for now. I will refrain from doing frequent phpLogCon announcements here. We are currently setting up the infrastructure (mailing list et al) for phpLogCon. I'll do one more announcement here when this is completed. In the meantime, I suggest subscribing to freshmeat's announcements, which we maintain in a timely way. You can do so at: http://freshmeat.net/projects/phplogcon/ We have just released version 2.1.0 which is the first official beta version of the v2 branch. PhpLogCon has been completely rewritten from scratch. It now offers a state-of-the art modern user interface and also is able to work with log files and not just databases. For example, it can be used to view a remote server's log files over the web (proper authentication settings highly recommended). It will evolve to a very capable search, reporting and analysis frontend for syslog data. Let me stress the point that it can work with log files directly. For example, we have set it up on one of our mail relays so that we can review mail logs without the need to login onto that machine. Obviously, this functionality should only be available to authenticated users, but then it is quite useful. I would appreciate to learn about any more thought of how this tool can be put to good use. Download: http://www.phplogcon.org/Downloads-req-viewdownloaddetails-lid-7.phtml Many thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 25 18:18:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:18:23 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EEA@grfint2.intern.adiscon.com> Folks, mailman does currently not accept this message, so I forward it to the list. A reply follows. Should someone else also have problems sending list mail, please let me know. Rainer > -----Original Message----- > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Friday, April 25, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Cc: Rainer Gerhards > Subject: FW: RSYSLOG "Best Practices" & General Questions > > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all data > from the remote sites. I am configuring it to store all SYSLOG data > with in the database, but I have followed the recommendations made to > "buffer" it to a spool first. My question is this, I do not want to > just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format on > the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this... It looked like it should able to be done, > but I am just not sure the best way to accomplish this. > > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site... This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to just > send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, and > let the server process and insert like it would local logs? > > 3) Within the scenario listed in question 2, how can I, 1) preserve > all the original IP addresses of the machines that are transmitting > information, and 2) tag that information with a specific account code > identifying the site that the information was sent from. Within the > database, I have created a column called "customerid" that I would like > to do this with. In this, I would like to designate a name or integer > like "1" for site A, "2" for site B, etc. The reason for this is that > I will run into situations where multiple customers will have the same > IP addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this...? > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based machines. > I would like to use the same table that is used for storage of the rest > of SYSLOG data, and I have the associated columns already built. I > just want to make sure that what I setup now will be completely > compatible and able to process NT Event Log information. > > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > From rgerhards at hq.adiscon.com Fri Apr 25 18:48:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:48:43 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions In-Reply-To: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> References: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> Message-ID: <1209142123.22738.461.camel@localhost.localdomain> On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > ?central? server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to ?buffer? it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in ?standard? log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this? It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. > 2) At each customer site, there will be a server called a > ?collector? that will accept all SYSLOG related information for that > site? This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, ?I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.html rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called ?customerid? that I would like to do this > with. In this, I would like to designate a name or integer like ?1? > for site A, ?2? for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site?s > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more ?elegant? method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this?? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > > > From mic at npgx.com.au Sat Apr 26 03:11:24 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sat, 26 Apr 2008 12:11:24 +1100 Subject: [rsyslog] rsyslog 3.16.0 released In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Message-ID: <20080426011021.M97077@npgx.com.au> Hi, > Hi all, > > I have just released rsyslog 3.16.0, a v3-stable release. This > release brings a major new feature, RELP support, to the stable > branch. RELP enables lossless transmission of syslog messages, in > cases where even plain tcp syslog fails. Other than that, the > release contains a few bugfixes. > > Users of the v3-stable branch are advised to update to the new release. Where are the RPM spec files for these releases so I can build the RPM's? Thanks. Michael. From mic at npgx.com.au Sun Apr 27 13:56:13 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sun, 27 Apr 2008 22:56:13 +1100 Subject: [rsyslog] rsyslog 3.16.0 on el4 Message-ID: <20080427115258.M2863@npgx.com.au> Hi, I just created a spec file and compiled an RPM for this version. In the source is the ./redhat/rsyslog.init file. This file references klogd through, so when I ended up installing the RPM, I couldn't start rsyslog using the script as it would exit. If klogd is no longer used in this rsyslog version, could you modify/correct the rsyslog.init file so that it no longer makes reference to it? I've already commented out the relevant entries in the file so I could start up rsyslogd. Thanks. Michael. From mic at npgx.com.au Sun Apr 27 18:12:26 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 03:12:26 +1100 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <20080427160908.M29132@npgx.com.au> Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. From rgerhards at hq.adiscon.com Sun Apr 27 18:31:48 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 27 Apr 2008 18:31:48 +0200 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> I am on a cell phone... Check out the v3 compatibility document. Its available right from the doc main page. rainer ----- Urspr?ngliche Nachricht ----- Von: "Michael Mansour" An: "rsyslog-users" Gesendet: 27.04.08 18:13 Betreff: [rsyslog] Date format on rsyslog 3.16.0 Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From mic at npgx.com.au Mon Apr 28 01:15:49 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 10:15:49 +1100 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> Message-ID: <20080427231354.M42594@npgx.com.au> Hi Rainer, > I am on a cell phone... > > Check out the v3 compatibility document. Its available right from > the doc main page. Yes, from that document I found the option: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat Which made it go back to the older time format. While reading it more though, I think I will actually keep it on the new format as the precision time isn't actually too bad. It may break some of my scripts which analyse daily log files and if it does, I'll enable the traditional format again, but for now I'll wait and see if anything breaks. Thanks. Michael. > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Michael Mansour" > An: "rsyslog-users" > Gesendet: 27.04.08 18:13 > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > Hi, > > The date format logged to the messages/maillog/etc has changed from: > > Apr 27 21:36:58 > > to: > > 2008-04-27T21:44:35.788251+10:00 > > I'm new to v3 so how can I change this format back to the original? > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog ------- End of Original Message ------- From rgerhards at hq.adiscon.com Mon Apr 28 08:01:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 08:01:40 +0200 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <20080427231354.M42594@npgx.com.au> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> <20080427231354.M42594@npgx.com.au> Message-ID: <1209362500.22738.466.camel@localhost.localdomain> Hi Michael, On Mon, 2008-04-28 at 10:15 +1100, Michael Mansour wrote: > Hi Rainer, > > > I am on a cell phone... > > > > Check out the v3 compatibility document. Its available right from > > the doc main page. > > Yes, from that document I found the option: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > Which made it go back to the older time format. While reading it more though, > I think I will actually keep it on the new format as the precision time isn't > actually too bad. It may break some of my scripts which analyse daily log > files and if it does, I'll enable the traditional format again, but for now > I'll wait and see if anything breaks. > Yes, scripts is the big Problem. Rsyslog actually had high-precision timestamps right from the beginning, but nobody ever used them. So I decided to change the default ;) On the scripts: I am thinking about a simple converter that takes high precision log files and converts them to old-style low precision ones. Such a converter could be run before the actual script. Do you (or someone else) think this is useful and worth the effort? Rainer > Thanks. > > Michael. > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Michael Mansour" > > An: "rsyslog-users" > > Gesendet: 27.04.08 18:13 > > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > > > Hi, > > > > The date format logged to the messages/maillog/etc has changed from: > > > > Apr 27 21:36:58 > > > > to: > > > > 2008-04-27T21:44:35.788251+10:00 > > > > I'm new to v3 so how can I change this format back to the original? > > > > Thanks. > > > > Michael. > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > ------- End of Original Message ------- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Mon Apr 28 14:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 14:34:52 +0200 Subject: [rsyslog] rsyslog 3.16.0 on el4 In-Reply-To: <20080427115258.M2863@npgx.com.au> References: <20080427115258.M2863@npgx.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F01@grfint2.intern.adiscon.com> Michael, this was contributed code. We even thought about dropping these files altogether. But as it looks, they seem to be of some value. If you have a corrected version, I'd appreciate if you could send me these, so that I can include them into the tarball- I have very limited platform know-how, so I am depending on other folks to send me startup scripts and such... Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Mansour > Sent: Sunday, April 27, 2008 1:56 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog 3.16.0 on el4 > > Hi, > > I just created a spec file and compiled an RPM for this version. > > In the source is the ./redhat/rsyslog.init file. This file references > klogd > through, so when I ended up installing the RPM, I couldn't start > rsyslog > using the script as it would exit. > > If klogd is no longer used in this rsyslog version, could you > modify/correct > the rsyslog.init file so that it no longer makes reference to it? > > I've already commented out the relevant entries in the file so I could > start > up rsyslogd. > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From smalenshek at skyline-ats.com Mon Apr 28 17:01:36 2008 From: smalenshek at skyline-ats.com (Stephen Malenshek) Date: Mon, 28 Apr 2008 09:01:36 -0600 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions Message-ID: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] Sent: Monday, April 28, 2008 8:55 AM To: rsyslog at lists.adiscon.com; Rainer Gerhards Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to "buffer" it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this. It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. In my present configuration, I am doing an if/then like the following: if $source == 'somemachine' \ and $syslogfacility-text == 'authpriv' \ then ?LocalSecure $template LocalSecure,"/var/log/secure" With this, how can I define multiple directions within this template? > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site. This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.h tml rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. At each site, there will be a single server tat is aggregating all SYSLOG data before transmitting it to the central repository. Are you talking about the machine sending the aggregate information or the hosts, I.E. router, server, switch, etc., that we will be monitoring? > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called "customerid" that I would like to do this > with. In this, I would like to designate a name or integer like "1" > for site A, "2" for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. I need a little more direction on this. I believe I understand, but I want to make absolutely sure. Could you provide me an example of what would be within the remote server's config and what the central server config would look like? > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this.? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html I want to look for things like the following: Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, terminal=ssh res=success)' With this, I want to be notified with the host information, address, etc. via e-mail if someone logs into a particular host as "root". > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. I have taken a good look at it and will integrate it within our administration interface. It looks like an excellent product and will save me a lot of code to write. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer From rgerhards at hq.adiscon.com Mon Apr 28 22:28:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 22:28:26 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F0F@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General > Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would > like to use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like > you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have > them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? I do not fully get you - do you mean how to write to different files based on some message property? If so, you can use the property replacer to do so: $template LocalSecure,"/var/log/%HOSTNAME%/secure" > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* > @{IP Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to > firewalls). Also, you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plai n-tcp-syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is > aggregating all SYSLOG > data before transmitting it to the central repository. Are > you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? Ah, ok. The host is taken from the message, and as such depends on the system that formatted it. You can define a template to consistently define the hostname on the relay. However, that requires that no further relay is between the aggregator and the original senders. > > > > > and 2) tag that information with a specific account code > identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like > to do this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, > if I have > > > to, with a subquery within the insert query to another > table to lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being > separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property > replace is > > part of the template system. It's quite powerful, but you > need to play a > > bit with it. > > > > I need a little more direction on this. I believe I > understand, but I want > to make absolutely sure. Could you provide me an example of > what would be > within the remote server's config and what the central server > config would > look like? > I can't do a lab right now (sorry, busy), but you can try along these lines. On the server's config, use a template along those lines: $template tpl,"%TIMESTAMP% %HOSTNAME% %syslogtag%,%msg:::drop-last-lf%\n" Where sitename is to be replaced by a site name (e.g. 1, SiteA ...). On the central server, you can extract that field inside a template by such a property specification: %msg:F,44:1% Again, all of this is untested but should either work or is pretty close to working ;) > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples > of efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably > has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, > address, etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our > clients use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used > for storage > > > of the rest of SYSLOG data, and I have the associated > columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare > schema", which is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will > define a special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, > so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the > parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product > and will save > me a lot of code to write. If you have any feedback, comments, suggestions or feature request, please let us know. It is very much in its infancy and Andre, the lead developer, is working very hard on it. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but > disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From rgerhards at hq.adiscon.com Tue Apr 29 16:27:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Apr 2008 16:27:33 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F1D@grfint2.intern.adiscon.com> I created a small wiki entry that has details on how to use a site id inside the message. I hope it is useful: http://wiki.rsyslog.com/index.php/Splitting_messages_based_on_a_site_ID Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would like to > use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* @{IP > Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to firewalls). Also, > you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is aggregating all > SYSLOG > data before transmitting it to the central repository. Are you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? > > > > > and 2) tag that information with a specific account code identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like to do > this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, if I have > > > to, with a subquery within the insert query to another table to > lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property replace > is > > part of the template system. It's quite powerful, but you need to play > a > > bit with it. > > > > I need a little more direction on this. I believe I understand, but I > want > to make absolutely sure. Could you provide me an example of what would > be > within the remote server's config and what the central server config > would > look like? > > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples of > efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, address, > etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our clients > use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used for > storage > > > of the rest of SYSLOG data, and I have the associated columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare schema", which > is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will define a > special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product and will > save > me a lot of code to write. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From infofarmer at FreeBSD.org Tue Apr 1 08:36:16 2008 From: infofarmer at FreeBSD.org (Andrew Pantyukhin) Date: Tue, 1 Apr 2008 10:36:16 +0400 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <20080401063614.GD4692@amilo.cenkes.org> On Mon, Mar 31, 2008 at 07:52:37PM +0200, Michael Biebl wrote: > I don't see the use case yet. Why would someone want to build > the plugins without the rsyslogd binary? Does it actually work > if you mix different versions of plugins and rsyslogd? If > someone is interested in only a specific plugin, why not use > "make -C plugins/foo"? It's just a convenience feature. In FreeBSD, we have separate ports for core and each plugin. At present, each plugin has to build rsyslogd all over again. From now on, it can just require the core port and save all the wasted build-time. I'm sure there are other ways to do it (like the one you mentioned), but is there a merit in arguing which one's the best? From rgerhards at hq.adiscon.com Tue Apr 1 08:50:30 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:50:30 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: <20080401063614.GD4692@amilo.cenkes.org> References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> <20080401063614.GD4692@amilo.cenkes.org> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CC9@grfint2.intern.adiscon.com> As a side-note: > Does it actually work > if you mix different versions of plugins and rsyslogd? The answer is: it depends. Rsyslog uses versioning in the plugin interface and also uses versioning in all internal objects. But there are still some non-object accesses. As long as a plugin is of the same version as the plugin interface that rsyslogd is compiled to, it works - provided that all objects it needs are implemented in the core (for v3, this must not necessarily be the case, especially if you use a new plugin with an older rsylogd build). The bottom line, of course, is that mixing versions should be avoided. But the good news is that rsyslog and the plugins detect incompatible versions and, if found, the plugin is simply not activated. So it is a predictable failure case (again, not 100% safe for the v3 case as it is not purely object based so far). Rainer From rgerhards at hq.adiscon.com Tue Apr 1 08:56:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:56:45 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com><1206988699.9240.38.camel@cutter><1206991339.9240.45.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Related question: does anybody know of a (sufficiently good) library that abstracts differences between TLS implementation. I mean in the way libdbi does for databases? It's obviously not important to have more than one driver available at runtime, but it would be necessary to be able to link against different TLS implementation. If anybody has a suggestion, please let me know. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, March 31, 2008 9:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] openssl vs rsyslog > > Whatever I do - I'll try to create a small driver shim so that > hopefully > different libraries could be used together with rsyslog. It depends on > the APIs, but in general that should not be too hard (but practice > often > tells the difference). > > On FIPS I made I comment, where I think the mails crossed. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Monday, March 31, 2008 9:27 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] openssl vs rsyslog > > > > 2008/3/31, seth vidal : > > > On Mon, 2008-03-31 at 21:18 +0200, Michael Biebl wrote: > > > > Interesting to know. Do you know any technical > > advantages of NSS over > > > > GnuTLS (stability, features, nicer API, etc)? > > > > > > > > > openssl transition library > > > fips certification. > > > > > > http://www.mozilla.org/projects/security/pki/nss/fips/ > > > > > > the second one being the most important, I think. > > > > Yeah, the openssl transition library doesn't concern us. > > > > Why is the fips certification important for rsyslog? > > > > Cheers, > > Michael > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 1 09:19:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 09:19:28 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCB@grfint2.intern.adiscon.com> > > > It fixes two bugs, one is a potential segfault in the > syslog/plain tcp > > > receiver. The other one is removal of some debug instrumentation > that > > > accidently made it into 3.12.5. There is also a new ./configure > option > > > (--enable/disable-rsyslogd) which permits to build just specific > plugins > > > > > > Hm, the default of enable_rsyslogd should imho be "yes" That was a bug, which is now fixed. > Forgot to add: If you intend to build rsyslogd and rfc3195d > conditionally, the corresponding man pages should also be installed > conditionally. This is now done. Thanks for pointing both out. Rainer From pvrabec at redhat.com Tue Apr 1 11:12:43 2008 From: pvrabec at redhat.com (Peter Vrabec) Date: Tue, 1 Apr 2008 11:12:43 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Message-ID: <200804011112.43694.pvrabec@redhat.com> Hi, On Tuesday 01 April 2008 08:56:45 am Rainer Gerhards wrote: > Related question: does anybody know of a (sufficiently good) library > that abstracts differences between TLS implementation. I mean in the way > libdbi does for databases? It's obviously not important to have more > than one driver available at runtime, but it would be necessary to be > able to link against different TLS implementation. > > If anybody has a suggestion, please let me know. you can look at nss_compat_ossl http://rcritten.fedorapeople.org/nss_compat_ossl.html http://fedoraproject.org/wiki/nss_compat_ossl but when I talked to guy who work on Fedora Crypto Consolidation it was suggested not to use the abstract library, since all these TLS implementations have kind of different "philosophy". It would be much cleaner to use separate plugins for different TLS. Note, that we will appreciate a lot going with NSS. http://fedoraproject.org/wiki/FedoraCryptoConsolidation http://fedoraproject.org/wiki/CryptoConsolidationEval > Thanks, > Rainer From rgerhards at hq.adiscon.com Tue Apr 1 13:50:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 13:50:53 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <200804011112.43694.pvrabec@redhat.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> <200804011112.43694.pvrabec@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD1@grfint2.intern.adiscon.com> Hi all, > http://fedoraproject.org/wiki/FedoraCryptoConsolidation > http://fedoraproject.org/wiki/CryptoConsolidationEval This material (the big picture) is quite convincing. Are there similar efforts in other distros? To make a point, I'll probably start developing an as slim as possible abstraction layer as a side-project (much like librelp) and see what I can do with it. I'll probably start with NSS and then see what it take to move the other libs in. My initial focus will be on "get it going", so it will not be a very secure solution at first. All comments are still very welcome and will be considered. I just thought it would be good to do a sum-up of my current thinking. Rainer From rgerhards at hq.adiscon.com Tue Apr 1 19:00:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 19:00:19 +0200 Subject: [rsyslog] rsyslog 3.15.0 released - with RELP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD7@grfint2.intern.adiscon.com> Hi all, yes, I know it's April, 1st. But this release is not related to it ;) I have just released rsyslog 3.15.0, the release that finally provides initial RELP support. RELP provides superior reliability over plain tcp syslog, ensuring that no messages are lost. With plain TCP, this can happen if the remote server goes down. RELP protects from this by utilizing a full-duplex protocol where each message is acknowledged. RELP should also be safe to use with stunnel, where plain tcp syslog sometimes gets into trouble. The core RELP protocol support is NOT part of rsyslog. It comes in the form of librelp, available at http://www.librelp.com . You need to install librelp before you try to compile rsyslog with RELP support. That should be fairly simple. If you run into any troubles, please let me know - I am more than happy to help. I plan to do some more in-depth doc on the use cases soon. Change Log: http://www.rsyslog.com/Article203.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-93.phtml I hope this release is useful. Feedback is appreciated. Rainer From BLentz at channing-bete.com Tue Apr 1 21:53:16 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 15:53:16 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem Message-ID: <47F292AC.8040307@channing-bete.com> Greetings, list! I'm having trouble compiling 3.15.0. I get the following error on one machine (CentOS 5): rsyslogd-msg.o: In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch_4' And slightly different error on another machine (Fedora Core 3): rsyslogd-debug.o(.text+0x183d): In function `dbgEntrFunc': /usr/src/redhat/BUILD/rsyslog-3.15.0/debug.c:996: undefined reference to `__sync_fetch_and_add' rsyslogd-msg.o(.text+0x127): In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch' rsyslogd-msg.o(.text+0xc55): In function `MsgAddRef': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:449: undefined reference to `__sync_fetch_and_add' Has anyone else seen this or know of a workaround or fix? I can't seem to find any reference in the rsyslog install documentation which says that I need additional libraries on my system. Thanks in advance From mbiebl at gmail.com Tue Apr 1 23:10:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:10:56 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F292AC.8040307@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Ben Lentz : > Greetings, list! > I'm having trouble compiling 3.15.0. > > I get the following error on one machine (CentOS 5): > rsyslogd-msg.o: In function `msgDestruct': > /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to > `__sync_sub_and_fetch_4' See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 #undef DO_HAVE_ATOMICS 1 should help Obviously, ./configure should detect this automatically. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Tue Apr 1 23:23:47 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:23:47 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Michael Biebl : > > Obviously, ./configure should detect this automatically. We could steal from here: http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From BLentz at channing-bete.com Tue Apr 1 23:25:08 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 17:25:08 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <47F2A834.9000509@channing-bete.com> > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > #undef DO_HAVE_ATOMICS 1 should help > > Obviously, ./configure should detect this automatically. > > Thanks! I'm still getting errors with debug.c It seems that debug.c has a reference to ATOMIC_INC (line 996) without a set of ifdef DO_HAVE_ATOMICS around it, so the build still dies. If I comment that line out, it will compile, but the binary is unusable "rsyslogd run failed with error -3000." On a whim, I tried ./configure --disable-debug, no luck. Ugh! From rgerhards at hq.adiscon.com Wed Apr 2 08:38:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Apr 2008 08:38:40 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F2A834.9000509@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> Message-ID: <1207118320.22738.3.camel@localhost.localdomain> It's a bug, obviously. I have now fixed it. Please go to atomic.h and change the ATOMIC_INC macro to be #define ATOMIC_INC(data) (++(data)) That'll solve the problems. Doing so does not introduce any real weakness - the code has been tested always with this setting. The atomics are a recent addition to make it ultra-reliable on all CPU architectures ... but obviously we need to do more work on the feature selection than I initially thought... Thanks for bringing this up, it was just at the right time. I'll pull that code from today's stable release (that happens when you think you can add something laaate in the process because it is "unintrusive" ;)). Rainer On Tue, 2008-04-01 at 17:25 -0400, Ben Lentz wrote: > > > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > > > #undef DO_HAVE_ATOMICS 1 should help > > > > Obviously, ./configure should detect this automatically. > > > > > Thanks! > > I'm still getting errors with debug.c It seems that debug.c has a > reference to ATOMIC_INC (line 996) without a set of ifdef > DO_HAVE_ATOMICS around it, so the build still dies. > > If I comment that line out, it will compile, but the binary is unusable > "rsyslogd run failed with error -3000." > > On a whim, I tried ./configure --disable-debug, no luck. Ugh! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 08:40:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 08:40:27 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD9@grfint2.intern.adiscon.com> Ummm ;) Could you lend me a helping hand? That looks god, but it also looks like it goes above my autotools horizon ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Tuesday, April 01, 2008 11:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > 2008/4/1, Michael Biebl : > > > > Obviously, ./configure should detect this automatically. > > We could steal from here: > http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 10:35:29 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 04:35:29 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <1207118320.22738.3.camel@localhost.localdomain> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> Message-ID: <47F34551.4010902@channing-bete.com> > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > change the ATOMIC_INC macro to be > > #define ATOMIC_INC(data) (++(data)) > > That'll solve the problems. Hey Rainer, Thanks for the help. I still continue to have problems, now at runtime. I'm patching the source like so: --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 @@ -36,8 +36,8 @@ #define INCLUDED_ATOMIC_H /* set the following to 1 if we have atomic operations (and #undef it otherwise) */ -#define DO_HAVE_ATOMICS 1 -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) +#undef DO_HAVE_ATOMICS 1 +#define ATOMIC_INC(data) (++(data)) #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) #endif /* #ifndef INCLUDED_ATOMIC_H */ then: rsyslogd run failed with error -3000 I've compiled with ggdb and run through the debugger, but I'm not sure where to set the break points... tried a couple things but nothing worked. I've also run it with strace, but I can't seem to find any problems. I believe there's a ton of reasons that it would generate this message and I'm not sure where to start. From rgerhards at hq.adiscon.com Wed Apr 2 10:40:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:40:35 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com><1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CDF@grfint2.intern.adiscon.com> Hi, > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 10:35 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > > > > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > > change the ATOMIC_INC macro to be > > > > #define ATOMIC_INC(data) (++(data)) > > > > That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 - > 0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > That's fine.. > then: > rsyslogd run failed with error -3000 Nice - error codes are defined in rsyslog.h. But of course, -3000 means "an error occurred". In any case, it's something that fails cleanly... > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate > this > message and I'm not sure where to start. ... that's why you don't see anything in the debugger. So let's get a bit down on where it really happens. There is debug support inside rsyslog: http://www.rsyslog.com/doc-debug.html It would be best if you do ./configure --enable-rtinst but that's not strictly necessary. Set RSYSLOG_DEBUG to LogFuncFlow and run rsyslog interactively with -d -n. You'll get some debug output. That should tell us something (well, you could even start without seting RSYSLOG_DEBUG, I think that'll be sufficient and less output). Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From theinric at redhat.com Wed Apr 2 10:46:34 2008 From: theinric at redhat.com (theinric at redhat.com) Date: Wed, 02 Apr 2008 10:46:34 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <47F347EA.7000501@redhat.com> On 04/02/2008 10:35 AM, Ben Lentz wrote: > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h and >> change the ATOMIC_INC macro to be >> >> #define ATOMIC_INC(data) (++(data)) >> >> That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > then: > rsyslogd run failed with error -3000 > > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate this > message and I'm not sure where to start. Hi Ben, my guess is that rsyslog can't find its libraries (e.g. /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" or something like "ln -s $(find /path/to/rsyslog/sources/ -name "*.so") /usr/[local/]lib/rsyslog". You can run "strace ./rsyslogd -v" to see where it looks for its libraries. From rgerhards at hq.adiscon.com Wed Apr 2 10:48:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:48:28 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F347EA.7000501@redhat.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> You are probably right. After your message I checked the module loader and it just returns the generic error code. I'll fix that... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of theinric at redhat.com > Sent: Wednesday, April 02, 2008 10:47 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > On 04/02/2008 10:35 AM, Ben Lentz wrote: > > > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h > and > >> change the ATOMIC_INC macro to be > >> > >> #define ATOMIC_INC(data) (++(data)) > >> > >> That'll solve the problems. > > > > Hey Rainer, > > Thanks for the help. I still continue to have problems, now at > runtime. > > I'm patching the source like so: > > > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 > -0400 > > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > > @@ -36,8 +36,8 @@ > > #define INCLUDED_ATOMIC_H > > > > /* set the following to 1 if we have atomic operations (and #undef > it > > otherwise) */ > > -#define DO_HAVE_ATOMICS 1 > > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > > +#undef DO_HAVE_ATOMICS 1 > > +#define ATOMIC_INC(data) (++(data)) > > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > > > then: > > rsyslogd run failed with error -3000 > > > > I've compiled with ggdb and run through the debugger, but I'm not > sure > > where to set the break points... tried a couple things but nothing > > worked. I've also run it with strace, but I can't seem to find any > > problems. I believe there's a ton of reasons that it would generate > this > > message and I'm not sure where to start. > > Hi Ben, > > my guess is that rsyslog can't find its libraries (e.g. > /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" > or something like "ln -s $(find /path/to/rsyslog/sources/ -name > "*.so") /usr/[local/]lib/rsyslog". > You can run "strace ./rsyslogd -v" to see where it looks for its > libraries. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 11:03:12 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 05:03:12 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> Message-ID: <47F34BD0.8060606@channing-bete.com> > You are probably right. After your message I checked the module loader > and it just returns the generic error code. I'll fix that... > > Rainer > > >> Hi Ben, >> >> my guess is that rsyslog can't find its libraries (e.g. >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name >> "*.so") /usr/[local/]lib/rsyslog". >> You can run "strace ./rsyslogd -v" to see where it looks for its >> libraries. >> Yep, you're right. I was expecting: - Shared library issues at runtime to be visible using ldd(1) - To at least be able to get command-line help or version (-v) information from rsyslogd (without loadable modules) - A nicer error message ;-) strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file or directory) Can the module path (/usr/lib/rsyslog) be specified? Is there a ./configure option that I'm missing at compile time or a command line parameter to rsyslogd or config file parameter in rsyslog.conf? I've been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and can't seem to find anything? It's very early in the US and I might just be missing something obvious. From rgerhards at hq.adiscon.com Wed Apr 2 11:07:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 11:07:03 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34BD0.8060606@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com><577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> <47F34BD0.8060606@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE2@grfint2.intern.adiscon.com> > >> my guess is that rsyslog can't find its libraries (e.g. > >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make > install" > >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name > >> "*.so") /usr/[local/]lib/rsyslog". > >> You can run "strace ./rsyslogd -v" to see where it looks for its > >> libraries. > >> > Yep, you're right. I was expecting: > - Shared library issues at runtime to be visible using ldd(1) > - To at least be able to get command-line help or version (-v) > information from rsyslogd (without loadable modules) > - A nicer error message ;-) > > strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT > access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or > directory) > open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file > or > directory) > > Can the module path (/usr/lib/rsyslog) be specified? Is there a > ./configure option that I'm missing at compile time or a command line > parameter to rsyslogd or config file parameter in rsyslog.conf? I've > been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and > can't > seem to find anything? That's mostly distro-specific and this is where I have very limited knowledge. But you can set the module path via the environment, use RSYSLOG_MODDIR=/path/to/modules/ . There is also a -m option to do the same, but it currently is read too late to help with the core modules. On my 64Bit system adding --libdir=/lib64 often helped with such problems when I encountered them. Not sure if that's the right cure... > > It's very early in the US and I might just be missing something > obvious. You are *really* brave! From rgerhards at hq.adiscon.com Wed Apr 2 16:34:37 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 16:34:37 +0200 Subject: [rsyslog] RELP vs plain tcp syslog Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Hi folks, I just blogged about the unreliability of plain tcp syslog. This may be interesting for at least some of you: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl og.html Rainer From Gerrard.Geldenhuis at datacash.com Wed Apr 2 17:24:57 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Wed, 2 Apr 2008 16:24:57 +0100 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: Hi Thanks, that is good to know. Worth keeping in mind. I was under the impression tcp was the miracle cure for udp unreliability but it seems not. There is very expensive software out there that can do guarenteed packet deliver over tcp... tibco is one I have seen before. Regards > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 02 April 2008 15:35 > To: rsyslog-users > Subject: [rsyslog] RELP vs plain tcp syslog > > Hi folks, > > I just blogged about the unreliability of plain tcp syslog. This may be > interesting for at least some of you: > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl > og.html > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 17:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 17:34:56 +0200 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CFA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gerrard Geldenhuis > Sent: Wednesday, April 02, 2008 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] RELP vs plain tcp syslog > > Hi > Thanks, that is good to know. Worth keeping in mind. I was under the > impression tcp was the miracle cure for udp unreliability but it seems > not. > > There is very expensive software out there that can do guarenteed > packet > deliver over tcp... Well... RELP can *really* do the trick. It needs some help in the rsyslog engine to work under all cases, that is when the relp stack is terminated (when the client rsyslog is shut down). But that isn't rocket science and has just been pushed back by more urgent work (securing the channel). So far, relp does single ack, that is the server ack's every packet back to the client. Client discards packets only after ack. So when a session breaks, we know what the server has processed. If, however, some of the acks get lost, we resend packets that the server already processed, resulting in some mild message *duplication* (we currently have a 128 packet app-layer max window and typically acks come in rather quickly). To work around this, the client must ack the server's acks (double-acking). That sounds scary, but is not. It's also not performance intense and the protocol is modeled that those with ultra-slim bandwidth can turn it off and live with the message duplication scenario. If double-acks are active, relp client and server will go into a recovery phase after reconnect negotiation, in which mutally acked package numbers are exchanged. To do so, the client must provide a session cookie back to the server. This is a potential attack vector and thus I'd like to have secure transport in place before I do it. Once the recovery negotiation is done, client and server know, and have discarded, what each other peer has processed. The remaining packets are re-sent, but under the same reliability settings. So if the connection breaks at this point again (or during the renegotiation), we'll simply go into another recovery phase until we finally succeed. It is of course important that session caches be persisted to disk when an engine stops - that will be part of rsyslog. The recovery produced itself is librelp. Once this is done, and with proper rsyslog queue settings, sufficiently stable hardware and sufficient disk space, I guarantee (not in a lawyers sense, though ;)) that you'll never lose a message nor get a duplicate. This is when I think we have achieved our reliability goal. If all goes well... summer? ;) Rainer > > tibco is one I have seen before. > > Regards > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: 02 April 2008 15:35 > > To: rsyslog-users > > Subject: [rsyslog] RELP vs plain tcp syslog > > > > Hi folks, > > > > I just blogged about the unreliability of plain tcp syslog. This may > be > > interesting for at least some of you: > > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > sysl > > og.html > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 23:10:39 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 17:10:39 -0400 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter Message-ID: <47F3F64F.5090600@channing-bete.com> When I try to use $HOSTNAME in an expression-based filter, I get "INVALID PROPERTY" errors logged by the system. From the documentation, I'm lead to believe that expression-based filters use the same properties as expression-based filters, which are documented in the property replacer documentation. Okay. But it seems that maybe $HOSTNAME has been changed to $source in the expression-based filters, but I can't seem to find any evidence of this in the man or HTML documentation. From rgerhards at hq.adiscon.com Thu Apr 3 08:52:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Apr 2008 08:52:31 +0200 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter In-Reply-To: <47F3F64F.5090600@channing-bete.com> References: <47F3F64F.5090600@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D04@grfint2.intern.adiscon.com> Ah, good report. I see where the problem is stemming from. We use the same code for property access no matter which method is used (expression-based, property-based, ...). That's good. The bad thing is that property names were case-sensitive (a really bad idea). We were in the discussion to remove that. The RainerScript engine already is case-insensitive and consequently converts all variable names to lower case. However, the message object property access method has not yet been converted, so it expects it in upper case. The result is the mess that you see... I'll initiate a quick discussion, but I think we'll now drop case-sensitivity at all. That shouldn't hurt any existing installations but instead make life easier. Thanks for bringing this up. For the same reason "$source" is the solution and I have seen in the forum that it works for you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 11:11 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter > > When I try to use $HOSTNAME in an expression-based filter, I get > "INVALID PROPERTY" errors logged by the system. From the documentation, > I'm lead to believe that expression-based filters use the same > properties as expression-based filters, which are documented in the > property replacer documentation. Okay. But it seems that maybe > $HOSTNAME > has been changed to $source in the expression-based filters, but I > can't > seem to find any evidence of this in the man or HTML documentation. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 10:06:55 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:06:55 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: 2008/3/31, Rainer Gerhards : > Well... First thing is that I have urgent need to release -- there are a > couple of things in the queue. If I don't release them this week, I'll > probably don't have a chance to get to a stable any time soon. So I will > release, even at the price that the version number scheme may be less > than optimum this time. > > But now on to the real meat (and thanks for sticking around on this > topic! ;))... I like the idea of the odd/even numbering (of the minor number) to distinguish unstable/stable release. E.g. GNOME uses it and it seems to work fine for them. E.g. the latest stable release was 2.22.0 (and minor point/bug fix releases usually follow as 2.22.1, 2.22.2...) The major development is going on in 2.23. The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... As soon as the code base stabilises (reaching beta/rc quality), they will release 2.23.90,2.23.91,... and the final stable release will be 2.24.0. (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From r.bhatia at ipax.at Fri Apr 4 10:16:01 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 04 Apr 2008 10:16:01 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: <47F5E3C1.9090207@ipax.at> Michael Biebl wrote: > I like the idea of the odd/even numbering (of the minor number) to > distinguish unstable/stable release. E.g. GNOME uses it and it seems > to work fine for them. > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > releases usually follow as 2.22.1, 2.22.2...) > The major development is going on in 2.23. > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > As soon as the code base stabilises (reaching beta/rc quality), they > will release > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). its not new as afairc the linux kernel established this scheme. anyways, what bugs me is, that we then will have something like: 3.20.x - stable 3.21.x - unstable relp no 3.22.x 3.23.x - unstable tls when we then decide to implement yet-another-new-feature-in-a- seperate-tree, we will end up with a scenario like 3.20.x - oldstable 3.21.x - unstable relp - closed 3.23.x - unstable tls 3.25.x - unstable featureX 3.26.x - stable or similar. that is kind of odd ... rainer: what do you think about only having one dev tree and having some patchsets for not-that-much-worked-on features (e.g. tls, featureX). of course, you can arrange your repository as you wish, but i would stick to *one* stable version and *one* dev version for the testers. everything else should be build (as patches) on top of these 2 branches. as a nice effect, we will have only 1 stable core and 1 dev core and have no issues when it comes to merging core-relp with core-tls with core-featureX. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From mbiebl at gmail.com Fri Apr 4 10:45:09 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:45:09 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: 2008/4/4, Raoul Bhatia [IPAX] : > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). > > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable > > or similar. > > that is kind of odd ... > git feature branches would be ideal for this kind of development imho. > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 branches. > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. It would definitely make it easier if there was only one stable and one development tree (and maybe a third one, oldstable). Rainer could work on experimental new features in separate git feature branches and when he considers them ready for testing, merge them into the dev branch (usually called master in git). E.g. the relp functionality could be in a branch called feature-relp, tls in feature-tls etc. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:06:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:06:35 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Ah, the source of the misunderstanding finally hits me... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Friday, April 04, 2008 10:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, > 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered > correctly). > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like The trees are NOT independent of each other! There is always just one development tree. In the above scenario, 3.23.x *includes* 3.21.x. Let's look what I currently try: We will finally see 3.14.0 stable today. I have also done a number of improvements since the relp version. So I have (not yet released) 3.14.0 - stable, bug fixes only 3.15.0 - everything that is in stable, plus relp - set to stabilize, bug fixes only 3.17.0 - everything that is in 3.15.x (note the .x not .0) plus new features - devel Development only happens in 3.17.0. When a feature focus has been reached, I now plan to branch off that release and let it stabilize (that is no new features, just bug fixes). I am trialing this currently with the 3.15.x branch. So far, it looks manageable. The plan is to slowly migrate 3.15.x to become 3.16.0, the next stable, deprecating 3.14.x when it comes out. I think that'll on average will be every two month, but I don't know yet. With 3.15.x it may be quicker, as the whole RELP code is in an external library and thus the core engine has not been destabilized. > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable So this picture would actually be: 3.20.x - oldstable 3.21.x - unstable relp - closed 3.11.x - stable 3.23.x - unstable tls 3.25.x - unstable tls+featureX That actually bring us down to three branches (plus old legacy like v2 stable): - stable - stabilizing (branched off the dev tree and in the bugfix wait queue - devel There may be more than one stabilizing tree at a given time (not yet thought out). In recent history, we have one such sample. That is the queue engine. It was an extremely big overhaul of nearly everything. So it needed some extended time to mature. In the mean time, some more focus features could be implemented. With the version numbering system I have now on my mind, that would probably have looked as follows (I am intentionally using major version 999 to avoid confusion with existing versions): 999.2.0 - stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.9.x - devel Actually, we are happy with the feature introduced in 999.7.x. But we won't release a new stable because .5 is not yet ready for prime time. So nothing happens at that point. Then, we are confident in .5. I think the following happens then: 999.2.0 - deprecated stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.8.0 - stable 999.9.x - devel We will never see a 999.6. I personally think this is acceptable, especially as I think it won't happen very often. Does that make more sense? > or similar. > > that is kind of odd ... > > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 > branches. I am a bit concerned that I need to be too careful which patch I apply where. As you know, rsyslog is developed on a very fast pace and I'd rather prefer to keep coding instead of thinking where I add a "by-feature" (that maybe takes an hour to implement). The scheme I am currently trialing (outlined above) works pretty OK. It is some additional overhead, but I mostly do everything to the trunk, create a patch for bugfixes and apply that to the "old" version in question. Some more work than before, but quite bearable. Of course things are different if I work on a feature, abandon it, and go back after a few weeks. But currently I implement everything up to a usable state (maybe not 100%, but useful in what is there) before I take the next step. Again... comments please ;) Rainer > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 11:19:59 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 11:19:59 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 999.2.0 - stable > 999.3.x - big overhaul feature, stabilizing > 999.5.x - .3 + next focus feature, stabilizing > 999.7.x - .5 + next focus feature, stabilizing > 999.9.x - devel > > Again... comments please ;) I think you really should try to use git feature branches for that. Have a stable and master (development) branch, and develop the features in separate topic branches feature-A, feature-B etc. Whenever one feature is ready, merge it into master. This way, it doesn't matter which feature you have to concentrate on is released first (no skipped version numbers!). The strong merge suppport in git would also allow to cherrypick easily from the different feature branches or merge between them. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:42:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:42:42 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> > 2008/4/4, Rainer Gerhards : > > > > 999.2.0 - stable > > 999.3.x - big overhaul feature, stabilizing > > 999.5.x - .3 + next focus feature, stabilizing > > 999.7.x - .5 + next focus feature, stabilizing > > 999.9.x - devel > > > > Again... comments please ;) > > I think you really should try to use git feature branches for that. > Have a stable and master (development) branch, and develop the > features in separate topic branches feature-A, feature-B etc. > Whenever one feature is ready, merge it into master. > This way, it doesn't matter which feature you have to concentrate on > is released first (no skipped version numbers!). > The strong merge suppport in git would also allow to cherrypick easily > from the different feature branches or merge between them. Sounds good, but a honest question (NOT trying to give a bias, just a problem description): While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and implement them - all while FocusFeatureX is being developed. Where do I apply these? And don't I get into trouble if that interferes with things that I do in FocusFeatureX? Remember, I change a couple of hundered lines all over the project on a typical day... Rainer > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 12:02:20 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 12:02:20 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 2008/4/4, Rainer Gerhards : > > > > > > 999.2.0 - stable > > > 999.3.x - big overhaul feature, stabilizing > > > 999.5.x - .3 + next focus feature, stabilizing > > > 999.7.x - .5 + next focus feature, stabilizing > > > 999.9.x - devel > > > > > > Again... comments please ;) > > > > I think you really should try to use git feature branches for that. > > Have a stable and master (development) branch, and develop the > > features in separate topic branches feature-A, feature-B etc. > > Whenever one feature is ready, merge it into master. > > This way, it doesn't matter which feature you have to concentrate on > > is released first (no skipped version numbers!). > > The strong merge suppport in git would also allow to cherrypick easily > > from the different feature branches or merge between them. > > > Sounds good, but a honest question (NOT trying to give a bias, just a > problem description): > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > implement them - all while FocusFeatureX is being developed. Where do I > apply these? And don't I get into trouble if that interferes with things > that I do in FocusFeatureX? Remember, I change a couple of hundered > lines all over the project on a typical day... Say you work on a featureA branch. Now you get an unrelated feature request for featureB. You'd switch back to current master, and branch of featureB, starting to work on that. By the end of the day, say featureB is ready, you'd merge those branch back into master (and delete branch featureB if no longer required). If featureC is dependend on featureA, you can branch from there. If you now work again on featureA, and later on featureC, you can merge the new commits from featureA back into featureC again. Later, when featureA and C are ready, you merge them into master again. For small changes, I'd directly work on master and commit there. There is also a nice feature called git-stash, which allows to put uncommitted changes away, work temporarily on other stuff, an get back to the uncommited stuff later. I'd say, just test git and try to get a "feeling" for it. That probably helps to make a better decision. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 12:17:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 12:17:33 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> OK, let me come to a conclusion ;) What Michael writes and Raoul suggested makes an awful lot of sense. Right now, I have two problems: a) we are still having a bit of trouble with the git transistion of rsyslog - I hope to have that sorted out soon b) I need to invest some time to fully understand how git branches. The bigger problem is probably b). Thankfully, I have started to work with git on librelp and it was a very good experience. It still looks like I need to invest at least a day or two more into getting fully involved with it. That doesn't sound much, but there is a lot I can do in rsyslog in this time. I think I will proceed as follows: For the next few weeks, I'll use the scheme that I outlined this morning. It works and it is sufficiently clean for the time being. Especially as I don't see any reason for gaps, there is no such major overhaul in sight. While I do so, I'll get more acquainted to git and see how I can make utilize its branching capabilities. At some point in time (and if everything works as well as advertised, what I assume ;)), I'll switch to the git feature branch strategy. My hope is all this can be done in the next 10 weeks or so. I hope I don't disappoint anyone - but the problem is things to do. All needs to go by priorities and, quite honestly, TLS or the new config file format is higher on my priority scale than the branching strategy. And, yes, I know good knowledge with git will save in the long run. But I need to start somewhere ;) I someone has serious concerns on the route I am taking, please scream now ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, April 04, 2008 12:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > 2008/4/4, Rainer Gerhards : > > > 2008/4/4, Rainer Gerhards : > > > > > > > > 999.2.0 - stable > > > > 999.3.x - big overhaul feature, stabilizing > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > 999.9.x - devel > > > > > > > > Again... comments please ;) > > > > > > I think you really should try to use git feature branches for > that. > > > Have a stable and master (development) branch, and develop the > > > features in separate topic branches feature-A, feature-B etc. > > > Whenever one feature is ready, merge it into master. > > > This way, it doesn't matter which feature you have to concentrate > on > > > is released first (no skipped version numbers!). > > > The strong merge suppport in git would also allow to cherrypick > easily > > > from the different feature branches or merge between them. > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just a > > problem description): > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > > implement them - all while FocusFeatureX is being developed. Where > do I > > apply these? And don't I get into trouble if that interferes with > things > > that I do in FocusFeatureX? Remember, I change a couple of hundered > > lines all over the project on a typical day... > > Say you work on a featureA branch. Now you get an unrelated feature > request for featureB. > You'd switch back to current master, and branch of featureB, starting > to work on that. > By the end of the day, say featureB is ready, you'd merge those branch > back into master (and delete branch featureB if no longer required). > If featureC is dependend on featureA, you can branch from there. If > you now work again on featureA, and later on featureC, you can merge > the new commits from featureA back into featureC again. > Later, when featureA and C are ready, you merge them into master again. > For small changes, I'd directly work on master and commit there. There > is also a nice feature called git-stash, which allows to put > uncommitted changes away, work temporarily on other stuff, an get back > to the uncommited stuff later. > > I'd say, just test git and try to get a "feeling" for it. That > probably helps to make a better decision. > > Cheers, > Michael > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Fri Apr 4 17:21:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 17:21:38 +0200 Subject: [rsyslog] First rsyslog v3 stable released - 3.14.1 Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D3C@grfint2.intern.adiscon.com> Hi all, I am glad to finally announce rsyslog 3.14.1, the first version of the v3 stable branch. This version offers almost all features of the current development version to those in need of a stable branch. Among others, this includes support for multiple database backends, queued and offline operations, SNMP and text file support. It is a big step compared to the v2 stable branch. Users are advised to check the compatibility notes before the update. It's not strictly necessary, but will enable to use rsyslog in the most efficient and problem-free way. Please note that development continues inside the v3 branch. The v3 stable branch will see future feature enhancements after they have sufficiently matured. The v2 stable branch is still supported. It is quite featureless (compared to v3) but extremely solid. So if you are (or need to be) ultra-conservative, you can still take the v2 route. Feature-wise v2 is a dead end and only bug fixes will be provided. The general recommendation is that the v3 stable branch be used for regular production machines. The Fedora project will feature rsyslog v3 stable in its upcoming release 9. Please note that I made a mistake two days ago: I accidently released 3.14.0 to the web, without it being actually ready. For this reason, I have renamed the release to 3.14.1. There will never be an official 3.14.0 release. If you happen to have it downloaded and installed, please accept my apologies. You should get 3.14.1 whenever you are ready. Change Log: http://www.rsyslog.com/Article205.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-94.phtml I hope this release is useful. As always, feedback is appreciated. Best regards, Rainer Gerhards From rgerhards at hq.adiscon.com Mon Apr 7 11:21:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Apr 2008 11:21:31 +0200 Subject: [rsyslog] rsyslog on GIT now - RE: rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D51@grfint2.intern.adiscon.com> Hi folks, co-incidentally, CVS has just begun to give me some pain last Friday, when it came to actually working with the different branches I now have. And a rainy Sunday made my have a deep look at the git manual. End result: I won't wait 10 weeks from now but have started to finally convert. I have pulled Michael Biebl's git repository, which he thankfully kept synchronized with the CVS. I am doing some final checks right now, but as it looks we will be using git from now on. Rsyslog is available from Adiscon's gitweb: http://git.adiscon.com/ It can also be pulled via git protocol. Please keep in mind that it may receive some more finishing touches. I will now have four active branches: - v2-stable - v3-stable - beta -- what becomes the next release of v3-stable - master -- the current (b)leading development edge There is also v1-stable, which is deprecated and a few legacy branches. As suggested by Raoul and Michael, I'll now begin to work with feature-branches (as soon as I have finished current work in progress). I am new to git and I may mess up things. Bear with me if I do ;) Please note that I am still working on the initial setup, so if you pull the repository now, you may need to pull it again some time later ;) If I mess up, I'll let you know via the mailing list. I hope this will be a good move. Special thanks to Michael and Raoul, who were persistent enough to finally made me move (and, well, to CVS which provided the final bit of motivation ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, April 04, 2008 12:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > OK, let me come to a conclusion ;) > > What Michael writes and Raoul suggested makes an awful lot of sense. > Right now, I have two problems: > > a) we are still having a bit of trouble with the git transistion of > rsyslog - I hope to have that sorted out soon > b) I need to invest some time to fully understand how git branches. > > The bigger problem is probably b). Thankfully, I have started to work > with git on librelp and it was a very good experience. It still looks > like I need to invest at least a day or two more into getting fully > involved with it. That doesn't sound much, but there is a lot I can do > in rsyslog in this time. > > I think I will proceed as follows: > > For the next few weeks, I'll use the scheme that I outlined this > morning. It works and it is sufficiently clean for the time being. > Especially as I don't see any reason for gaps, there is no such major > overhaul in sight. > > While I do so, I'll get more acquainted to git and see how I can make > utilize its branching capabilities. At some point in time (and if > everything works as well as advertised, what I assume ;)), I'll switch > to the git feature branch strategy. My hope is all this can be done in > the next 10 weeks or so. > > I hope I don't disappoint anyone - but the problem is things to do. All > needs to go by priorities and, quite honestly, TLS or the new config > file format is higher on my priority scale than the branching strategy. > And, yes, I know good knowledge with git will save in the long run. But > I need to start somewhere ;) > > I someone has serious concerns on the route I am taking, please scream > now ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Friday, April 04, 2008 12:02 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog version numbering > > > > 2008/4/4, Rainer Gerhards : > > > > 2008/4/4, Rainer Gerhards : > > > > > > > > > > 999.2.0 - stable > > > > > 999.3.x - big overhaul feature, stabilizing > > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > > 999.9.x - devel > > > > > > > > > > Again... comments please ;) > > > > > > > > I think you really should try to use git feature branches for > > that. > > > > Have a stable and master (development) branch, and develop the > > > > features in separate topic branches feature-A, feature-B etc. > > > > Whenever one feature is ready, merge it into master. > > > > This way, it doesn't matter which feature you have to > concentrate > > on > > > > is released first (no skipped version numbers!). > > > > The strong merge suppport in git would also allow to cherrypick > > easily > > > > from the different feature branches or merge between them. > > > > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just > a > > > problem description): > > > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests > and > > > implement them - all while FocusFeatureX is being developed. Where > > do I > > > apply these? And don't I get into trouble if that interferes with > > things > > > that I do in FocusFeatureX? Remember, I change a couple of > hundered > > > lines all over the project on a typical day... > > > > Say you work on a featureA branch. Now you get an unrelated feature > > request for featureB. > > You'd switch back to current master, and branch of featureB, starting > > to work on that. > > By the end of the day, say featureB is ready, you'd merge those > branch > > back into master (and delete branch featureB if no longer required). > > If featureC is dependend on featureA, you can branch from there. If > > you now work again on featureA, and later on featureC, you can merge > > the new commits from featureA back into featureC again. > > Later, when featureA and C are ready, you merge them into master > again. > > For small changes, I'd directly work on master and commit there. > There > > is also a nice feature called git-stash, which allows to put > > uncommitted changes away, work temporarily on other stuff, an get > back > > to the uncommited stuff later. > > > > I'd say, just test git and try to get a "feeling" for it. That > > probably helps to make a better decision. > > > > Cheers, > > Michael > > > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 8 18:42:39 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Apr 2008 18:42:39 +0200 Subject: [rsyslog] rsyslog 3.17.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D85@grfint2.intern.adiscon.com> Hi all, rsyslog 3.17.0 has just been released. It is part of the development branch. The primary new feature is the ability to send email alerts based on syslog or file data. The action engine now has the ability to carry out an action only once within a configured interval and/or only during specific time frames. Option processing has been improved and a number of stability updates have been included. This is a recommended update for all users of the development version. More information on the major new feature can be found here: http://www.rsyslog.com/doc-ommail.html The following is a sample code snippet that alerts the operator when disk problems are detected: $ModLoad ommail $ActionMailSMTPServer mail.example.net $ActionMailFrom rsyslog at example.net $ActionMailTo operator at example.net $template mailSubject,"disk problem on %hostname%" $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" $ActionMailSubject mailSubject # make sure we receive a mail only once in six # hours (21,600 seconds ;)) $ActionExecOnlyOnceEveryInterval 21600 # the if ... then ... mailBody mus be on one line! if $msg contains 'hard disk fatal failure' then :ommail:;mailBody Note that we now have the ability to limit an action to be executed only once inside a specific period. In the above sample, the email alert happens only if there was no other such alert within the past 6 hours - this is absolutely vital to prevent an accidental DoS on your mailbox ;) ... but it may also be handy with other actions (e.g. SNMP trap notification etc etc). It's implemented at the action engine level, so it will work with any action, even file or database writes. Change Log: http://www.rsyslog.com/Article207.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-95.phtml I would be most interested in feedback on the new email feature, including clever use cases. I am sure it can be quite useful (especially if you think about imfile...), but would really appreciate to hear from you (and prove this in practice)! Thanks, Rainer Gerhards From aoz.syn at gmail.com Wed Apr 9 01:47:51 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:47:51 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 Message-ID: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Please forgive me if this is addressed in later versions; I'm just working against the version that was pulled into Gentoo yesterday. The atomic operations used in debug.c and msg.c were not introduced until gcc-4.1.0; as such, systems compiling with an earlier version (i.e. hardened-gentoo) fail compilation with undefined references to __sync_fetch_and_add and __sync_sub_and_fetch. I'm not sure of the right way to check for this in atomic.h, but it would be helpful if a check was made before setting DO_HAVE_ATOMICS. In addition, although msg.c makes appropriate checks for DO_HAVE_ATOMICS, debug.c does not and fails to compile under such systems. I would supply a patch, but being completely new to your codebase thought I'd start with reporting. Thanks for your time! RB From aoz.syn at gmail.com Wed Apr 9 01:53:23 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:53:23 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Message-ID: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> > The atomic operations used in debug.c and msg.c were not introduced > until gcc-4.1.0; Forgot to give specifics: the atomic operations as defined in atomic.h. The change made to bring debug.c to 1.49 in particular. From aoz.syn at gmail.com Wed Apr 9 01:55:49 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:55:49 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> Message-ID: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Strike that. Even better if I just read the ML before posting... *sigh* Long day, guys, sorry for the noise. On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > The atomic operations used in debug.c and msg.c were not introduced > > until gcc-4.1.0; > > Forgot to give specifics: the atomic operations as defined in > atomic.h. The change made to bring debug.c to 1.49 in particular. > From rgerhards at hq.adiscon.com Wed Apr 9 09:10:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:10:23 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com><4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> No problem at all, I think it's well-hidden in the ML. A good fix (via autoconf is still not done, so I should probably add a bug tracker ;). Please keep reporting. All others please note that I will release updates for v3-stable and beta soon. Its brewing ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, April 09, 2008 1:56 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Atomic operations in 3.15.0 > > Strike that. Even better if I just read the ML before posting... > *sigh* > > Long day, guys, sorry for the noise. > > On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > > The atomic operations used in debug.c and msg.c were not > introduced > > > until gcc-4.1.0; > > > > Forgot to give specifics: the atomic operations as defined in > > atomic.h. The change made to bring debug.c to 1.49 in particular. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From janfrode at tanso.net Wed Apr 9 09:33:02 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 9 Apr 2008 09:33:02 +0200 Subject: [rsyslog] git repo ? Message-ID: I just noticed "- converted to git" on http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html Is there a repo available online somewhere ? If not, maybe you could get http://github.com/ to host it so that there's a nice web interface to it and is easily forkable for others to work on it ? -jf From rgerhards at hq.adiscon.com Wed Apr 9 09:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:34:56 +0200 Subject: [rsyslog] git repo ? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8D@grfint2.intern.adiscon.com> Oops... too much going on? I thought I had posted it to the ML. Anyhow... it's at http://git.adiscon.com - git protocol is enabled Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jan-Frode Myklebust > Sent: Wednesday, April 09, 2008 9:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] git repo ? > > I just noticed "- converted to git" on > http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html > > Is there a repo available online somewhere ? If not, maybe you > could get http://github.com/ to host it so that there's a nice > web interface to it and is easily forkable for others to work > on it ? > > > > -jf > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From aoz.syn at gmail.com Wed Apr 9 17:08:22 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 9 Apr 2008 09:08:22 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> Message-ID: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> On 4/9/08, Rainer Gerhards wrote: > No problem at all, I think it's well-hidden in the ML. A good fix (via > autoconf is still not done, so I should probably add a bug tracker ;). I don't know about the remainder of the atomic changes you've made, but a check for >=gcc-4.1.0 will suffice for the stuff that's in atomic.h. I must say - I can't express how excited I was to find this project. It's been on my long-term plate for quite some time to fork syslog-ng and add all the nice features the Pro license grants; now I don't have to! I'm partial to the syslog-ng configuration syntax, but can most definitely live with this for not having to re-implement all that stuff. Tell me, are you also working on UDF-WORM? ;) From rgerhards at hq.adiscon.com Wed Apr 9 18:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 18:17:42 +0200 Subject: [rsyslog] rsyslog v3-stable/3.14.2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DA8@grfint2.intern.adiscon.com> Hi all, I have just released 3.14.2, an update to v3-stable. It is a purely bug-fixing release. Most importantly, it fixes a problem that caused imklog not to pull module symbols correctly from recent kernels. Also, a segfault caused by using expression-based filters is fixed. There are also some other fixes, see ChangeLog for details. This is a recommended update for all v3-stable branch users. Changelog: http://www.rsyslog.com/Article209.phtml Download (now pointing to the info page so that you see the md5sum ;)): http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-96.phtml I hope this release is useful. As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 9 18:51:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 09 Apr 2008 18:51:04 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> Message-ID: <1207759864.22738.94.camel@localhost.localdomain> On Wed, 2008-04-09 at 09:08 -0600, RB wrote: > On 4/9/08, Rainer Gerhards wrote: > > No problem at all, I think it's well-hidden in the ML. A good fix (via > > autoconf is still not done, so I should probably add a bug tracker ;). > > I don't know about the remainder of the atomic changes you've made, > but a check for >=gcc-4.1.0 will suffice for the stuff that's in > atomic.h. I just began, more outstanding... But it looks like that easy solution is the right one - thanks! > I must say - I can't express how excited I was to find this project. > It's been on my long-term plate for quite some time to fork syslog-ng > and add all the nice features the Pro license grants; now I don't have > to! Good to hear ;) And, believe it or not, I never ever installed syslog-ng. But,of course, it's the primary competitor in this space. IMHO we are now far ahead of it (I guess you already know that chart, but...): http://www.rsyslog.com/doc-rsyslog_ng_comparison.html Bazsi seem currently to brew something in regard to the log store, which is not (yet ;)) my priority. Also I don't think it helps to cryptographically sign the store (at least for court), the message itself must be signed (that's my route on that). So we will probably see a few features in the *paid* edition of syslog-ng that rsyslog does not yet have. Even with that, and even compared to the paid edition, I think we are now far ahead. > I'm partial to the syslog-ng configuration syntax, but can most > definitely live with this for not having to re-implement all that > stuff. rsyslog's config file syntax is just plain ugly. No way around. But wait a bit, I am working on a scripting engine: http://rgerhards.blogspot.com/2008/02/introducing-rainerscript-and-some.html http://www.rsyslog.com/doc-rscript_abnf.html > Tell me, are you also working on UDF-WORM? ;) Neither of them - just the boring stuff ;) Rainer From rgerhards at hq.adiscon.com Thu Apr 10 14:53:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 14:53:40 +0200 Subject: [rsyslog] Status of TLS development Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Hi folks, I just wanted to give you an update on my native TLS plans. I've been searching around and trying things the past days. As it looks now, I will use a wrapper class to cover nss and gnutls. Contrary to what I initially thought, I'll now probably start with GNUTls. The reason is that it seems to be much easier to start with it. For some insight, please follow this discussion on the NSS newsgroup: http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread /45e418ca0a556a42 That doesn't mean I settle for just GNUTls. Even if I start with it, NSS support will follow. I hope it will be easier to implement when I can check the NSS manuals for specific functionality. I thought I let you know. Rainer From mbiebl at gmail.com Thu Apr 10 15:29:06 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:29:06 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Hi folks, > > I just wanted to give you an update on my native TLS plans. I've been > searching around and trying things the past days. As it looks now, I > will use a wrapper class to cover nss and gnutls. Contrary to what I > initially thought, I'll now probably start with GNUTls. The reason is > that it seems to be much easier to start with it. For some insight, > please follow this discussion on the NSS newsgroup: > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread > /45e418ca0a556a42 > > That doesn't mean I settle for just GNUTls. Even if I start with it, NSS > support will follow. I hope it will be easier to implement when I can > check the NSS manuals for specific functionality. > > I thought I let you know. Is it technically necessary to implement the layer via a separate library (libsci), or could the layer also be implemented directly within rsyslog as (libtool) convenience lib? The point is, that a separate library means more works for packagers and people who want to compile rsyslog themselves. If you think, that libsci might be useful for other projects, then implementing it as a system library makes sense (or due to licensing matters). If libsci will only be used within rsyslog though, I think implementing it as a libtool convenience lib within rsyslog makes more sense. Just my 2? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:31:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:31:38 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Michael, I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Depending on how it works out, it may be useful for others, too. But that's not the primary focus. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Hi folks, > > > > I just wanted to give you an update on my native TLS plans. I've > been > > searching around and trying things the past days. As it looks now, I > > will use a wrapper class to cover nss and gnutls. Contrary to what I > > initially thought, I'll now probably start with GNUTls. The reason > is > > that it seems to be much easier to start with it. For some insight, > > please follow this discussion on the NSS newsgroup: > > > > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/threa > d > > /45e418ca0a556a42 > > > > That doesn't mean I settle for just GNUTls. Even if I start with it, > NSS > > support will follow. I hope it will be easier to implement when I > can > > check the NSS manuals for specific functionality. > > > > I thought I let you know. > > Is it technically necessary to implement the layer via a separate > library (libsci), or could the layer also be implemented directly > within rsyslog as (libtool) convenience lib? > > The point is, that a separate library means more works for packagers > and people who want to compile rsyslog themselves. > > If you think, that libsci might be useful for other projects, then > implementing it as a system library makes sense (or due to licensing > matters). > If libsci will only be used within rsyslog though, I think > implementing it as a libtool convenience lib within rsyslog makes more > sense. > > Just my 2? > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:38:18 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:38:18 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Michael, > > I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be fine with rsyslogs GPLv3). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:40:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:40:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go to link to GPLv3. Sorry... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Michael, > > > > I fully agree, but licensing is the culprit. I think it'll be either > LGPL or BSD. I'd really prefer to plug it directly into librelp and > rsyslog, but that will create license conflicts. > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > fine with rsyslogs GPLv3). > > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Thu Apr 10 15:43:01 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:43:01 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Bad typo. Should have been: librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go to link to GPLv3. Sorry... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, April 10, 2008 3:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go > to link to GPLv3. Sorry... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Thursday, April 10, 2008 3:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Status of TLS development > > > > 2008/4/10, Rainer Gerhards : > > > Michael, > > > > > > I fully agree, but licensing is the culprit. I think it'll be > either > > LGPL or BSD. I'd really prefer to plug it directly into librelp and > > rsyslog, but that will create license conflicts. > > > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > > fine with rsyslogs GPLv3). > > > > Cheers, > > Michael > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:50:34 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:50:34 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Bad typo. Should have been: > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > to link to GPLv3. Sorry... GPLv3+/commercial is not incompatible with LGPLv2.1+ Am I missing something? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:56:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:56:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: <1207835772.22738.132.camel@localhost.localdomain> On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > 2008/4/10, Rainer Gerhards : > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ If I put the crypto wrapper (code named so far libsci) into *rsyslog* the crypto wrapper will be under GPLv3. If librelp needs TLS functionality, it must link to the crypto-wrapper *inside* rsyslog, which is GPLv3. librelp can not do that if it is not used inside a GPLv3 program (because, in essence, I would need to detouch those parts from rsyslog that are needed by librelp and then add that to the non-GPLv3 application). Or am I overlooking something? A prominent example of what will use librelp functionality and is not under GPLv3 is Adiscon's MonitorWare products under Windows (and keep in mind that they currently sponsor rsyslog development ;)). Rainer From mbiebl at gmail.com Thu Apr 10 15:55:45 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:55:45 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Michael Biebl : > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > Am I missing something? The whole point of the LGPL is, to be useful for commercial/closed source applications. And GPLv3+ and LGPLv2.1+ is no problem either. I'm not a lawyer though, so take this with a grain of salt. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Apr 10 16:01:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 16:01:56 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <1207835772.22738.132.camel@localhost.localdomain> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: 2008/4/10, Rainer Gerhards : > > On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > > > to link to GPLv3. Sorry... > > > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > > If I put the crypto wrapper (code named so far libsci) into *rsyslog* > the crypto wrapper will be under GPLv3. > > If librelp needs TLS functionality, it must link to the crypto-wrapper Ok, this was probably my misconception. I thought, only rsyslog would need the TLS functionality. It wasn't clear to me, that *both* rsyslog and librelp would link against libsci. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 16:34:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 16:34:33 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: <1207838073.22738.144.camel@localhost.localdomain> On Thu, 2008-04-10 at 16:01 +0200, Michael Biebl wrote: > Ok, this was probably my misconception. > I thought, only rsyslog would need the TLS functionality. > It wasn't clear to me, that *both* rsyslog and librelp would link > against libsci. Sorry, probably I've been to brief. Let me sort out things at least now: Both rsyslog and librelp need native TLS functionality. For rsyslog, this is to support plain TCP syslog via TLS (hopefully compatible with the like syslog-ng and MonitorWare feature) as well as the upcoming IETF syslog-transport-tls standard. Librelps needs to do native TLS because it can not rely on an external entity (like stunnel) and claim to be reliable and pretty tamperproof. So it must talk to the remote peer directly. In RELP protocol, there will be a STARTTLS command verb, just like in SMTP. Negotiation on the availability of this verb is similar to SMTP's EHLO message or BEEP's greeting message. Both the RELP protocol as well as librelp already contain the necessary provisioning. My intention is to make libsci available to anyone who may have a need for it, but it will be focused on what I actually need. Please note that I have intentionally called it libsci - simple *crypto* interface - and not something like libsti - simple *tls* interface - because I intend to place all crypto functions into a single library (thus relieving at least a bit of the burden from an extra library). Other than TLS, rsyslog will need to support digital signatures as well as generic hashes (not only for cryptographic needs, but also for some other nice things). If JF finally convinces me to implement some of the advanced output plugin functionality as completely separate projects [he is seeding that thought quite well ;)], these projects will most probably also need to link to it. My intention is to put libsci under a very liberal license, probably either a BSD style one or at least LGPL. I have no idea yet when we will see any of libsci functionality in practice ;) I hope this finally clarifies. Sorry for the confusion. Please drop me any questions that are still around. Rainer From rgerhards at hq.adiscon.com Fri Apr 11 10:04:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Apr 2008 10:04:12 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... Message-ID: <1207901052.22738.230.camel@localhost.localdomain> Hi folks, I am sorry, this will be a long mail. But I would appreciate if you'd read it in full and comment on it. This mail covers a really important decision for rsyslog and will probably even influence if the project succeeds in the long term. Package maintainers and code contributors are especially requested to *really* read it. Though I t try hard to provide all relevant facts (that's why it is getting long), I will probably miss some and not properly convey others. Please feel free to ask. Let me start with explaining that the rsyslog project conceptually consists of three parts: - the modules - "helper" functions - rsyslog-specific functionality Modules are actually projects in their own right, just being distributed with the rsyslog tarball for convenience. A module may be released under any license. Note that modules call both rsyslog-specific functionality (e.g. to submit a message) as well as helper function (e.g. to handle tcp sessions). The "helper" functions are a growing set of generic objects. Examples are the module loader, the queue engine, networking support, the script engine and virtual machine, ... - you get the idea: Things that are used inside rsyslog but are not necessarily of use only for rsyslog. Actually, this could be called a "rsyslog runtime library". Rsyslog-specific functionality is primarily rsyslogd and everything it takes to glue together helpers and plugins to build the working syslog subsystem. Let's stick with that for a brief while. Now let me explain the idea of loosely coupled modules. This stems back to JF's effort to convince me to the Unix philosophy of "small tools that work well together". We had another good discussion yesterday (on the blog) and it made me change my mind a bit (well, probably, not 100% convinced yet, but it he managed to seed the thought ;)). While I still think that there are things that need to be really tightly coupled to the rsyslog core, there are others which not necessarily need to. Let me call the later "loosely coupled modules", in contrast to the (tightly coupled) plugins that actually become part of the rsyslogd process during runtime. The analysis plugins I have on my mind could become such loosely coupled modules. As an interface, the usual Unix "send it and forget it" pipe could be used, and it would probably be acceptable to allow for minor message loss during shutdown and plugin failure (anything else would require a pipe application protocol, e.g. relp over pipe, which sounds scary). The plus in doing so would be the ability to use those plugins in configurations where rsyslog is not present (e.g. driven by syslog-ng or a detached message generator [fed from a log file]). Done right, one could even select the (sligly lossy) pipe interface or the full blown plugin interface as a compile time switch. If you think it out, we may even end with an abstraction layer where each module can be compiled for either the plugin interface or the pipe (no promises, though). One problem with this approach is that modules call into the rsyslog helpers. For example, rsyslog's network support need to be available for all those modules that do something over the net. That's not a problem if I have a tightly coupled plugin as today (the rsyslog core makes the necessary bindings). It would become more problematic if I move the module to a pipe interface, because I now need to find a way to use the rsyslog objects. But that's still doable (though pretty ugly). It becomes really problematic if the same module, using a pipe interface, is to be used with e.g. syslog-ng. I don't think that syslog-ng will be able to provide it with an emulated rsyslog "net" object. Let's stick with this problem for a moment. Coincidentally, we had another discussion on the mailing list yesterday - on the TLS support wrapper for rsyslog and librelp. That discussion centered around licenses. Technically, there are also a number of issues. I have now involved myself enough with GnuTLS and a bit of NSS so that I am able to try draft a first abstraction layer. I thought hard and the "right solution" involves encapsulation stream network access. So the right thing to do is to have one object that handles network streams. That object then is configured to use either plain tcp, TLS (via whatever library) or even GSS-API. Nice and clean. It gets dirty if I think about the details. If I do it that way, it makes rsyslog depend on this object (so-far codenamed libsci). However, that would mean that any rsyslog installation would need to pull in libsci. Not a big deal, except, right, except if the crypto libraries are also pulled in by libsci. So would every desktop system running rsyslog need to have the crypto libraries installed? Scary... unacceptable. In rsyslog, we had the same problem a few month ago (at that time the mysql client libraries were the problem). The solution was the rsyslog loader, which dynamically loads other libraries (and their dependencies) on demand. The loader is what enables rsyslogd to be installed everywhere, but only with minimal core requirements (and have everything else in separate packages). So if libsci would be part of rsyslog, we would not have any problem at all. After all, the necessary plumbing is ready at hand in form of the rsyslog helper objects. This is where we come back to loosely coupled modules. You notice it is the same problem? Both them as well as libsci would need to call the rsyslog helpers. Now let's come a bit to licensing. In order to understand that, we need to talk about rsyslog funding first. Obviously, I am spending full time (and a bit more than that) on rsyslog for quite a while now. I even intend to do that for some more months as rsyslog is currently mabye 55% of what I would like it to be. Somehow I must get funding - for the time, for the hardware and for all the other things ;) What made the rsyslog project possible, and still 99.9% funds it is Adiscon, the company that provides (of course ;)) the best-ever logging solutions on Windows. Actually the Windows closed source pays for the rsyslog project. While we hope to find other sources of funding in the future, I can not ignore the fact. Once thing we would like to do at Adiscon is include select parts of the technology I am now developing into the closed source applications, too. The most prominent example is the RELP protocol. I obviously find this a fair policy - after all, the alternative would be to do it in closed source only and I was able to convince my folks at Adiscon that it is far better to contribute to the open source world. There is one drawback in this requirement: licensing. Of course, we could pick a BSD style license and every problem would be solved. But, I have to admit, we do not like to give everything to our competitors in the *closed source* space. We have made very bad experiences with folks building on our technology and even turning it against us. I won't get agreement from Adiscon to use a BSD license for everything (plus I personally, too, don't like to see that effect). We already discussed this on the mailing list here as part of dual-licensing in the past. The solution was that the technology in question was created as its own, dual-licensed, project. This lead to the creation of librelp. Rsyslog itself was left under GPLv3 (which I sincerely believe in because of its anti-patent, anti-drm clauses - even though the license gives myself obviously some troubles). Dual-licensing librelp lead to some duplicate code and made me not use some features which I could have used if I had access to all rsyslog helper objects. For librelp, that is not yet a big deal, because it is quite unique, very few needs to access the rsyslog helpers. With TLS, however, the situation changes and we get the dangling issue of rsyslog helpers in librelp, too. LETS TRY TO WRAP-UP If I put all of this together, I think I have taken a (slightly? ;)) wrong path. The core problem is monolithic design from a very high point of view. I have to admit I think this is what JF and some others were pointing out, but I didn't realize it quickly enough. Sure, rsyslog is quite modular by now. But rsyslog always requires rsyslog to do everything. It is very hard to do any rsyslog-related work without the rsyslog core. While rsyslog has a carefully crafted set of helper objects, these are not exposed to the outside world. And the licensing issues associated with that design begin to screw up everything in the long run. I think we need change. The obvious solution seems to be extracting the rsyslog helpers out of the rsyslog core project and create a "rsyslog runtime". That runtime than could individually be installed and be put under a different license (bear with me, explanation follows below). Let's consider a complicated case with the runtime. Assume we have a plugin "NeverBeforeSeenAnalysis". Let's say someone wants to use it with syslog-ng (!). With the runtime, all needed would be to compile it for the pipe interface and install rsyslog-runtime and the module onto the system. Now let's consider Adiscon's MonitorWare products on Windows. When they implement RELP, they need librelp and can pull in the rsyslog-runtime (for network access including TLS). For rsyslogd itself nothing really happens, the runtime is now just its own library - linking to it needs not to be modified. So for rsyslogd, the change would be transparent. Technically, this indeed solves the issues. Let me stress the point that it leads to code reuse, where I currently need to rewrite things (which increasingly concerns me, especially from a maintenance point of view). Now, on to the licensing. Obviously, the MonitorWare use case above would be totally incompatible with GPLv3. So the rsyslog-runtime would need to be under a different license. It could be dual-licensed, but I think that would probably do more bad than good. I think I can convince Adiscon to go with LGPL for the runtime part. Granted, it introduces risk of closed source competitors pulling it in, but the advantages should outweigh this risk. >From the ability to put this work under a different license, I think I am in good shape: most of the helper objects are freshly written and have only received limited patches (if at all) from contributors. I can contact them and ask for permission to change the license. Where I don't get permission, I think I can re-implement the contribution. Again, most of the code in question has been written in the past 4 month and is 99.999% non-contributed. There may be some few runtime objects which stem back to sysklogd. There, a license change is impractical. I'll have to life with the fact that those can not go into a re-licensed runtime. Depending on how important the functionality is, I either need to rewrite or drop it (for non-rsyslogd use). In any case, this looks (pending detail analysis) quite possible. Big question number one is what you think of this runtime approach? Have I overlooked something? Do you object it for some reason? If so, which? The next question is how to package this inside the source tree. Remember, currently rsyslog and the plugins (considered separate projects) are all packed together inside a single tarball. This is very convenient, both for me as well as for package maintainers and users. The question is if we split rsyslog into the rsyslogd and the rsyslog-runtime, will we continue to deliver the runtime as part of the rsyslog package? Or would it be better to move it to a librsyslog project? Other than with the plugins, we actually would have two different licenses, so it may be confusing to have both of it in the same project (but I have seen that GnuTLS uses exactly this approach, with the main library being LGPL and the - included - extras library GPL only). So that's the next question, obviously depending on the first: how to pack projects if we do a runtime split? I know this is a long and dense mail. My apologies for this. But I think the discussion is needed. I honestly believe that a number of discussions in the past weeks actually circled around this theme, we just didn't actually get down to the point. Please note that I will hold TLS development until we have reached consensus on the runtime/licensing topic. The reason is if we don't do a runtime split, I need to do things considerable different than when we do one (much more code, probably yet another external library). So, obviously, I have a current bias towards the split. However, experience shows that I (as everyone ;)) tend to overlook or misunderstand things. Thus your feedback is so important. I don't like the idea of jiggling back and forth on such an important topic as licensing and high-level modularization, so I would like to get it now done in "the right way" and keep it stable for at least the foreseeable future. Given the fact that the decision somehow affects rsyslog's development as whole, I would even appreciate quick feedback. In this spirit: please let the comments flow ;) Thanks, Rainer From friedl at hq.adiscon.com Fri Apr 11 11:24:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Fri, 11 Apr 2008 11:24:20 +0200 Subject: [rsyslog] rsyslog 3.15.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DD6@grfint2.intern.adiscon.com> Hi all, rsyslog 3.15.1 has been released. This is a refresh of the beta branch, providing new bug fixes. The beta branch currently features the RELP protocol and will be the next v3-stable once it has sufficiently matured. Changelog: http://www.rsyslog.com/Article210.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-97.phtml As always, feedback is appreciated. Florian Riedl From aoz.syn at gmail.com Sat Apr 12 00:10:04 2008 From: aoz.syn at gmail.com (RB) Date: Fri, 11 Apr 2008 16:10:04 -0600 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <1207901052.22738.230.camel@localhost.localdomain> References: <1207901052.22738.230.camel@localhost.localdomain> Message-ID: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Read once, digested for a few hours, read again. *whew* A monolithic message about a monolithic problem... ;) Also please understand I'm new here and am not completely familiar with your codebase, so may well miss the mark. Do I understand you correctly that part of your concern with looser coupling is the higher potential for lossiness? I don't think I can afford a line-by-line analysis at the moment, but got a persistent thought both times through: if you're considering pursuing a loosely-coupled model, why not consider going all the way and treat rsyslogd as a dispatch/routing agent among the various modules? That's by no means a simple solution, but I wonder why certain pieces of functionality must be tightly coupled, either to each other (network/crypt) or to the core itself (network). I *think* it becomes even more interesting when you start thinking about ways to 'stack' modules. Don't feel qualified to even start answering the licensing/packaging stuff, so will leave that be. From rgerhards at hq.adiscon.com Mon Apr 14 08:58:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 08:58:50 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> References: <1207901052.22738.230.camel@localhost.localdomain> <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Message-ID: <1208156330.22738.252.camel@localhost.localdomain> On Fri, 2008-04-11 at 16:10 -0600, RB wrote: > Read once, digested for a few hours, read again. *whew* A monolithic > message about a monolithic problem... ;) Also please understand I'm > new here and am not completely familiar with your codebase, so may > well miss the mark. > > Do I understand you correctly that part of your concern with looser > coupling is the higher potential for lossiness? Yes, there are two issues with that: increased lossiness (due to less knowledge inside rsyslog core of what happens & the same problem plain tcp has) and performance. A pipe interface is much slower than the native in-process plugin interface - this can turn out to become a problem in high performance big enterprise deployments. HOWEVER, all of this can be solved if the module can be compiled to utilize either one of both interfaces. I have also thought more about this and it looks like the absolutely perfect solution to all needs (the bottom line is: the user decides what to use and has all options at hand). > > I don't think I can afford a line-by-line analysis at the moment, but > got a persistent thought both times through: if you're considering > pursuing a loosely-coupled model, why not consider going all the way > and treat rsyslogd as a dispatch/routing agent among the various > modules? Actually, this is the approach rsyslog takes. Though this page is a bit old (and generic), it still covers the base ideas: http://www.rsyslog.com/doc-generic_design.html This document was originally written for an IETF discussion, so it is not rsyslog-specific (but given the same authorship... ;)). > That's by no means a simple solution, but I wonder why > certain pieces of functionality must be tightly coupled, either to > each other (network/crypt) or to the core itself (network). I *think* > it becomes even more interesting when you start thinking about ways to > 'stack' modules. The most recent rsyslog loader supports stacking. We now have a call where an object can request it to find other objects, which can lead to an object load request if the object is not yet in core. Of course, if a new object is loaded, it can request the loader to find other objects it needs... Run a recent build of rsyslogd with -d -n and you'll see these calls in action ;) However, not all of the rsyslog modules are full blown objects as of now. Converting them is part of the ongoing modularization effort. But this also touches the core of my long message: the loader and all helpers currently live in rsyslogd core ONLY. They are not usable by any entity that is not either part of rsyslog or being *called by rsyslog* (plugins). So it is currently impossible to use any of rsyslog's technology without rsyslogd. Among others, this prevents loosely coupling modules (because the module would need access to the runtime objects without rsyslog). It also prevents cases where someone wants to use parts that depend on rsyslog technology, but without using rsyslogd itself. While, of course, I do not find that smart, it may be just the right thing for the user in question. I intend to no longer limit the user in his options, freeing the code base even more. This is both a technical issue as well as a licensing one. The solution, the more I think about it, is really obvious: create a runtime system and place it under LGPL. That'll solve all issues and make rsyslog modular not only on a detail level but also from the high-level perspective. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 12:46:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 12:46:23 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux Message-ID: <1208169983.22738.290.camel@localhost.localdomain> Hi all, I am currently working on the imklog module. I noticed that - for years now - internally generated messages of this module are also emitted with the "kernel" facility. For example, the startup message: imklog 3.17.1-bsd, log source = /proc/kmsg started. rsyslog has taken over that behavior from the sysklogd project, so it is really ages old. HOWEVER, I personally think this is the wrong facility. IMHO the syslogd facility would be the right one, as this message is not from the kernel log but instead locally generated by the syslogd (the syslog subsystem, to be precise). On BSD, no similar problem exists so far, because there no kernel message was added. At the current snapshot, I add few messages if things go wrong. Actually, this triggered the question on how to handle internally generated messages... Question now: shall I change that to LOG_SYSLOG? Or should it remain as is? Final option: a config switch where the user can configure which facility to use (that would then default to LOG_SYSLOG IMO). Feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 14:20:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 14:20:52 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux In-Reply-To: <1208169983.22738.290.camel@localhost.localdomain> References: <1208169983.22738.290.camel@localhost.localdomain> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E00@grfint2.intern.adiscon.com> Hi all, I have thought a bit more about it. I ended up with a configuration variable, but the default is depending on the klog driver. So on Linux, we continue to log those messages as kernel, while on BSD we continue to log them as syslogd. The default can be overwritten by the user. However, I still wonder if it really is desirable to log the messages as kernel on Linux. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, April 14, 2008 12:46 PM > To: rsyslog-users > Subject: [rsyslog] Facility of imklog-generated messages on linux > > Hi all, > > I am currently working on the imklog module. I noticed that - for years > now - internally generated messages of this module are also emitted > with > the "kernel" facility. For example, the startup message: > > imklog 3.17.1-bsd, log source = /proc/kmsg started. > > rsyslog has taken over that behavior from the sysklogd project, so it > is > really ages old. HOWEVER, I personally think this is the wrong > facility. > IMHO the syslogd facility would be the right one, as this message is > not > from the kernel log but instead locally generated by the syslogd (the > syslog subsystem, to be precise). > > On BSD, no similar problem exists so far, because there no kernel > message was added. At the current snapshot, I add few messages if > things > go wrong. Actually, this triggered the question on how to handle > internally generated messages... > > Question now: shall I change that to LOG_SYSLOG? Or should it remain as > is? Final option: a config switch where the user can configure which > facility to use (that would then default to LOG_SYSLOG IMO). > > Feedback is appreciated. > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 15 09:41:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 09:41:56 +0200 Subject: [rsyslog] some rsyslog files under LGLP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E12@grfint2.intern.adiscon.com> Hi all, I will now begin to change the license for some rsyslog files from GPLv3 only to LGPLv3. I have contacted the major contributors and will contact others on an as-needed basis. The license change will probably take some time. Rainer From rgerhards at hq.adiscon.com Tue Apr 15 10:51:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 10:51:45 +0200 Subject: [rsyslog] rsyslog runtime library Message-ID: <1208249505.22738.309.camel@localhost.localdomain> Hi all, I have begun to lay the foundation for a well-defined rsyslog runtime library. I had a couple of discussions and the current (still open to change ;)) strategy is to build gradually build the runtime library as part of the main rsyslog project. This resembles the way plugins, also individual projects, are shipped together in the main tarball. What convinced me to work on this route, at least for now, is some technical advantages. Using this approach, I can incrementally work on the runtime library. While it is useful to have, it requires quite some effort to build a clean, abstracted runtime library. Remember the main effort for v3 (which will lead to v4) is to create a fully modular rsyslog subsystem. I started from the quite monolithic v2 source based and have already extracted a great deal of objects. But this is not yet completed - I do this work in parallel to introducing new features and when it best fits. Similarly, splitting off the runtime library right now would mean I stopping feature development for a few weeks, finally cleaning up the object model and putting it all nicely together. While it obviously sound like the right thing from the software engineering point of view, it sounds pretty bad from a feature perspective (bringing everything to a halt plus introducing many new, untested, bug possibilities). So I now believe the right route is again a compromise: I continue to add new features, keeping the runtime idea in mind and contributing to it whenever it is a good time to do so. For v4, of course, we should have a full blow abstraction, because then we finally have the clean object model. So I will spend some limited time on initial creation of a skeleton library, but will continue to work on rsyslog features on a fast pace. I honestly believe this will lead to the overall best result. Feedback is appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Tue Apr 15 15:24:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 15:24:38 +0200 Subject: [rsyslog] rsyslog 3.17.1 (devel) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E1D@grfint2.intern.adiscon.com> Hi all: Rsyslog 3.17.1 has just been released. The primary new feature is full BSD platform support. Rsyslog now not only compiles cleanly on BSD, but also now supports BSD's kernel log natively. In general, kernel logging has been improved and better documented. High-precision timestamps for the kernel log have been enabled. There are also some minor other enhancements as well as a couple of bug fixes. Rsyslog 3.17.1 is a recommended update for all users of the devel branch. Changelog: http://www.rsyslog.com/Article213.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-98.phtml As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 16 17:27:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Apr 2008 17:27:04 +0200 Subject: [rsyslog] Syslog Traffic Data & Alerting Message-ID: <1208359624.22738.353.camel@localhost.localdomain> Hi all, I am currently thinking about a traffic-volume based approach to detect unusual situations and issue alerts. This is in the context of advanced analysis output plugins (which will later also be able to run standalone). I have some ideas, but in order to refine them, I need some real-world syslog traffic data. I am in need of the volume of syslog messages per second, as it evolves over multiple days (preferably over a period of one or two month - and to get started a week worth of this data). I wonder if anyone of you would be interested in providing me such a traffic sample. The data I need would be totally anonymous. All I need is a file with one line per second: timestamp and number of messages per second (no information about message content or any other message property). However, these two fields must be correct and not be any further processed. After all, the root idea is to detect something from the patterns and mangled ones would be very counter-productive. If there is interest (multiple contributors would be happily accepted), I would write an output plugin for rsyslog that gathers this data. It would be very useful to have at least one high-volume environment inside the sample. Please let me know if you could contribute to this effort. Your help would be deeply appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 18 18:38:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 18 Apr 2008 18:38:06 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon Message-ID: <1208536686.22738.360.camel@localhost.localdomain> Hi folks, of course, rsyslog can work with any web interface. Nevertheless, we have begun to rewrite our phpLogCon in the hope that it will be an even useful tool. From Andre, the main project author, I have heard that we will probably see an initial v2 release next week. He has also set up a demo server, based on today's codebase: http://demo.phplogcon.org/ I would appreciate if you could drop by and provide us some feedback. It's the first time phpLogCon v2 is visible online and I we are very interested to learn what people like and what not. Of course, the project will expand much in the next few month, and you can help drive its direction. However, I think it is quite useful as it currently is. We have pushed some limited test data into the demo. If someone would like to try it out on his/her own system, please let us know. We will help you get started for sure :) Thanks, Rainer From bakers at web-ster.com Fri Apr 18 18:46:37 2008 From: bakers at web-ster.com (Scott Baker) Date: Fri, 18 Apr 2008 09:46:37 -0700 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808D06D.5000309@web-ster.com> Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer My first response is... WOW! I've never seen that web interface before but that's pretty impressive. I'm much more of a CLI/grep kinda guy but I can see a lot of PHBs loving this. It's very impressive and seems to work quite well. Only complaint is that the date column wraps to two lines, but that's just cosmetic. Kick ass. Keep up the good work. -- Scott Baker - Canby Telcom RHCE - System Administrator - 503.266.8253 From r.bhatia at ipax.at Fri Apr 18 21:18:47 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 18 Apr 2008 21:18:47 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808F417.2000805@ipax.at> hi rainer, looks pretty decent. i pulled the git repository and will have a further look ;) as far as i can tell by now, i like modular design where you can fetch data from different streams. cheers, raoul Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From lanas at securenet.net Wed Apr 23 01:31:00 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 19:31:00 -0400 Subject: [rsyslog] rsyslog in general Message-ID: <20080422193100.2b2a09a1@mistral.stie> Hello, I'd be interested to use rsyslog in a production environment. On one side I have the impression that it is a stable product but then if I look at the releases, they are quite numerous over a small amount of time. I would like to know what is the average count of bug fixes in general and to which severity. Thanks, Al From mbiebl at gmail.com Wed Apr 23 02:37:15 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 02:37:15 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422193100.2b2a09a1@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: 2008/4/23, lanas : > Hello, > > I'd be interested to use rsyslog in a production environment. On one > side I have the impression that it is a stable product but then if I > look at the releases, they are quite numerous over a small amount of > time. I would like to know what is the average count of bug fixes in > general and to which severity. There are frequent releases of devel "snapshots". For stable releases you should check out http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml (not sure if you are already aware of that) Or do you also have concerns about the frequency of stable releases? Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Wed Apr 23 02:54:47 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 20:54:47 -0400 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: <20080422205447.0772e890@mistral.stie> Le Mercredi, 23 Avril 2008 02:37:15 +0200, "Michael Biebl" a ?crit : >> I'd be interested to use rsyslog in a production environment. On >> one side I have the impression that it is a stable product but then >> if I look at the releases, they are quite numerous over a small >> amount of time. I would like to know what is the average count of >> bug fixes in general and to which severity. > There are frequent releases of devel "snapshots". > For stable releases you should check out > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > (not sure if you are already aware of that) Thanks. The package I ended up with was version 3.17.1. What about this version ? It is not tagged as beta, although it is development. What is the schedule for a stable release of that version ? I see that the page you gave lists 3.14 as stable. > Or do you also have concerns about the frequency of stable releases? Well, to have a grasp at how the development is planned and one would be great. Cheers, Al From mbiebl at gmail.com Wed Apr 23 03:16:27 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 03:16:27 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422205447.0772e890@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> <20080422205447.0772e890@mistral.stie> Message-ID: 2008/4/23, lanas : > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > "Michael Biebl" a ?crit : > > > >> I'd be interested to use rsyslog in a production environment. On > >> one side I have the impression that it is a stable product but then > >> if I look at the releases, they are quite numerous over a small > >> amount of time. I would like to know what is the average count of > >> bug fixes in general and to which severity. > > > There are frequent releases of devel "snapshots". > > > For stable releases you should check out > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > (not sure if you are already aware of that) > > > Thanks. The package I ended up with was version 3.17.1. What about > this version ? It is not tagged as beta, although it is development. > What is the schedule for a stable release of that version ? I see that > the page you gave lists 3.14 as stable. 3.17.1 is from the devel branch, so I wouldn't recommend it for production use. The version numbering has been in discussion not long ago, so take the following with a grain of salt (Rainer, the lead author, will probably have additional information). 2.0.x is the old-stable branch (the current version being 2.0.4). If you want something ultra-stable, without the need for some of the nice feature of the v3 branch, take this one. The first stable release of the v3 branch was 3.14.1 (the current version being 3.14.2). If you need some of the features of the v3 branch, go with 3.14.2. Stable versions are even numbered (second digit). See also http://www.rsyslog.com/doc-status.html The devel branch is usually for bleeding edge stuff and has a odd numbered second digit. The beta branch is for stabilizing features, resulting in the next stable release. I also has a odd numbered second digit. > > > > Or do you also have concerns about the frequency of stable releases? > > > Well, to have a grasp at how the development is planned and one would > be great. > http://www.rsyslog.com/doc-status.html http://www.rsyslog.com/doc-features.html http://www.rsyslog.com/doc-v3compatibility.html http://www.rsyslog.com/doc HTH, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Apr 23 08:20:58 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Apr 2008 08:20:58 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie><20080422205447.0772e890@mistral.stie> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EB0@grfint2.intern.adiscon.com> Hi there, Michael has done an excellent job of summing up things :) Just let me add a few thoughts. First of all, the versioning document needs to be updated and I'll do that soon ;) The frequent releases are a result of rapid development. There is lots to do and we do it quickly. This, of course, introduces bugs. To develop rapidly and still provide stable releases, we have settled on this for now: The v2-stable is a feature-dead end. It receives patches only. Ultra-stable, quite featureless. V3-stable primarily receives patches but from time to time new functionality is integrated. This functionality ripens in the beta branch. The devel version has the newest code and most bugs. It is NOT recommended for production use. At some point in time, beta is branched off devel. Beta than receives bug fixes, but no new features. After some time (between 3 and 10 weeks, depending on bug reports), beta becomes v3-stable. Even inside stable or beta or devel there are several levels of maturity - some code is unmodified for more than a year, while other just recently made it into stable. I have recently begun to add some information about when features have been introduced into a version. The general rule is that the newer a feature is, the more likely it contains bugs. In general, rsyslog is rock-solid. However, some so far seldomly-used features (expression based filters are a good example) has seen limited exposure to practice. Such features are more likely to have problems. If you try judge project and feature stability, it may be a good idea to look at the bugzilla and the code commits: http://bugzilla.adiscon.com/buglist.cgi?cmdtype=runnamed&namedcmd=rsyslog-all http://git.adiscon.com/?p=rsyslog.git;a=summary You may also want to have a look at my blog, where I post anything of note. It's also a good place to see what is going on and actually even to see what is giving me a hard time every now and then: http://rgerhards.blogspot.com Finally, if you let me know what you intend to do, I can provide you my personal view of how stable the required features are. But as I said - it's pretty stable now. Rsyslog also recently made it into Red Hat Enterprise Linux (5.2) and you probably know that these folks are quite conservative when it comes to stability ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, April 23, 2008 3:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in general > > 2008/4/23, lanas : > > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > > "Michael Biebl" a ?crit : > > > > > > >> I'd be interested to use rsyslog in a production environment. > On > > >> one side I have the impression that it is a stable product but > then > > >> if I look at the releases, they are quite numerous over a small > > >> amount of time. I would like to know what is the average count > of > > >> bug fixes in general and to which severity. > > > > > There are frequent releases of devel "snapshots". > > > > > For stable releases you should check out > > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > > (not sure if you are already aware of that) > > > > > > Thanks. The package I ended up with was version 3.17.1. What about > > this version ? It is not tagged as beta, although it is > development. > > What is the schedule for a stable release of that version ? I see > that > > the page you gave lists 3.14 as stable. > > 3.17.1 is from the devel branch, so I wouldn't recommend it for > production use. > The version numbering has been in discussion not long ago, so take the > following with a grain of salt (Rainer, the lead author, will probably > have additional information). > > 2.0.x is the old-stable branch (the current version being 2.0.4). If > you want something ultra-stable, without the need for some of the nice > feature of the v3 branch, take this one. > > The first stable release of the v3 branch was 3.14.1 (the current > version being 3.14.2). If you need some of the features of the v3 > branch, go with 3.14.2. > > Stable versions are even numbered (second digit). > > See also > http://www.rsyslog.com/doc-status.html > > The devel branch is usually for bleeding edge stuff and has a odd > numbered second digit. > The beta branch is for stabilizing features, resulting in the next > stable release. I also has a odd numbered second digit. > > > > > > > > Or do you also have concerns about the frequency of stable > releases? > > > > > > Well, to have a grasp at how the development is planned and one would > > be great. > > > > http://www.rsyslog.com/doc-status.html > http://www.rsyslog.com/doc-features.html > http://www.rsyslog.com/doc-v3compatibility.html > http://www.rsyslog.com/doc > > HTH, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From friedl at hq.adiscon.com Thu Apr 24 14:36:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 24 Apr 2008 14:36:20 +0200 Subject: [rsyslog] rsyslog 3.16.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Hi all, I have just released rsyslog 3.16.0, a v3-stable release. This release brings a major new feature, RELP support, to the stable branch. RELP enables lossless transmission of syslog messages, in cases where even plain tcp syslog fails. Other than that, the release contains a few bugfixes. Users of the v3-stable branch are advised to update to the new release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-99.phtml Changelog: http://www.rsyslog.com/Article215.phtml Florian Riedl From rgerhards at hq.adiscon.com Fri Apr 25 07:49:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 07:49:53 +0200 Subject: [rsyslog] phpLogCon v2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ED7@grfint2.intern.adiscon.com> Hi all, I'd like to pass the info that phpLogCon v2's initial release is now available. I thought a while if I should use this list to announce it, but came to the conclusion that it is justified for now. I will refrain from doing frequent phpLogCon announcements here. We are currently setting up the infrastructure (mailing list et al) for phpLogCon. I'll do one more announcement here when this is completed. In the meantime, I suggest subscribing to freshmeat's announcements, which we maintain in a timely way. You can do so at: http://freshmeat.net/projects/phplogcon/ We have just released version 2.1.0 which is the first official beta version of the v2 branch. PhpLogCon has been completely rewritten from scratch. It now offers a state-of-the art modern user interface and also is able to work with log files and not just databases. For example, it can be used to view a remote server's log files over the web (proper authentication settings highly recommended). It will evolve to a very capable search, reporting and analysis frontend for syslog data. Let me stress the point that it can work with log files directly. For example, we have set it up on one of our mail relays so that we can review mail logs without the need to login onto that machine. Obviously, this functionality should only be available to authenticated users, but then it is quite useful. I would appreciate to learn about any more thought of how this tool can be put to good use. Download: http://www.phplogcon.org/Downloads-req-viewdownloaddetails-lid-7.phtml Many thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 25 18:18:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:18:23 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EEA@grfint2.intern.adiscon.com> Folks, mailman does currently not accept this message, so I forward it to the list. A reply follows. Should someone else also have problems sending list mail, please let me know. Rainer > -----Original Message----- > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Friday, April 25, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Cc: Rainer Gerhards > Subject: FW: RSYSLOG "Best Practices" & General Questions > > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all data > from the remote sites. I am configuring it to store all SYSLOG data > with in the database, but I have followed the recommendations made to > "buffer" it to a spool first. My question is this, I do not want to > just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format on > the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this... It looked like it should able to be done, > but I am just not sure the best way to accomplish this. > > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site... This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to just > send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, and > let the server process and insert like it would local logs? > > 3) Within the scenario listed in question 2, how can I, 1) preserve > all the original IP addresses of the machines that are transmitting > information, and 2) tag that information with a specific account code > identifying the site that the information was sent from. Within the > database, I have created a column called "customerid" that I would like > to do this with. In this, I would like to designate a name or integer > like "1" for site A, "2" for site B, etc. The reason for this is that > I will run into situations where multiple customers will have the same > IP addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this...? > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based machines. > I would like to use the same table that is used for storage of the rest > of SYSLOG data, and I have the associated columns already built. I > just want to make sure that what I setup now will be completely > compatible and able to process NT Event Log information. > > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > From rgerhards at hq.adiscon.com Fri Apr 25 18:48:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:48:43 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions In-Reply-To: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> References: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> Message-ID: <1209142123.22738.461.camel@localhost.localdomain> On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > ?central? server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to ?buffer? it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in ?standard? log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this? It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. > 2) At each customer site, there will be a server called a > ?collector? that will accept all SYSLOG related information for that > site? This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, ?I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.html rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called ?customerid? that I would like to do this > with. In this, I would like to designate a name or integer like ?1? > for site A, ?2? for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site?s > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more ?elegant? method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this?? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > > > From mic at npgx.com.au Sat Apr 26 03:11:24 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sat, 26 Apr 2008 12:11:24 +1100 Subject: [rsyslog] rsyslog 3.16.0 released In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Message-ID: <20080426011021.M97077@npgx.com.au> Hi, > Hi all, > > I have just released rsyslog 3.16.0, a v3-stable release. This > release brings a major new feature, RELP support, to the stable > branch. RELP enables lossless transmission of syslog messages, in > cases where even plain tcp syslog fails. Other than that, the > release contains a few bugfixes. > > Users of the v3-stable branch are advised to update to the new release. Where are the RPM spec files for these releases so I can build the RPM's? Thanks. Michael. From mic at npgx.com.au Sun Apr 27 13:56:13 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sun, 27 Apr 2008 22:56:13 +1100 Subject: [rsyslog] rsyslog 3.16.0 on el4 Message-ID: <20080427115258.M2863@npgx.com.au> Hi, I just created a spec file and compiled an RPM for this version. In the source is the ./redhat/rsyslog.init file. This file references klogd through, so when I ended up installing the RPM, I couldn't start rsyslog using the script as it would exit. If klogd is no longer used in this rsyslog version, could you modify/correct the rsyslog.init file so that it no longer makes reference to it? I've already commented out the relevant entries in the file so I could start up rsyslogd. Thanks. Michael. From mic at npgx.com.au Sun Apr 27 18:12:26 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 03:12:26 +1100 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <20080427160908.M29132@npgx.com.au> Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. From rgerhards at hq.adiscon.com Sun Apr 27 18:31:48 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 27 Apr 2008 18:31:48 +0200 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> I am on a cell phone... Check out the v3 compatibility document. Its available right from the doc main page. rainer ----- Urspr?ngliche Nachricht ----- Von: "Michael Mansour" An: "rsyslog-users" Gesendet: 27.04.08 18:13 Betreff: [rsyslog] Date format on rsyslog 3.16.0 Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From mic at npgx.com.au Mon Apr 28 01:15:49 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 10:15:49 +1100 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> Message-ID: <20080427231354.M42594@npgx.com.au> Hi Rainer, > I am on a cell phone... > > Check out the v3 compatibility document. Its available right from > the doc main page. Yes, from that document I found the option: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat Which made it go back to the older time format. While reading it more though, I think I will actually keep it on the new format as the precision time isn't actually too bad. It may break some of my scripts which analyse daily log files and if it does, I'll enable the traditional format again, but for now I'll wait and see if anything breaks. Thanks. Michael. > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Michael Mansour" > An: "rsyslog-users" > Gesendet: 27.04.08 18:13 > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > Hi, > > The date format logged to the messages/maillog/etc has changed from: > > Apr 27 21:36:58 > > to: > > 2008-04-27T21:44:35.788251+10:00 > > I'm new to v3 so how can I change this format back to the original? > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog ------- End of Original Message ------- From rgerhards at hq.adiscon.com Mon Apr 28 08:01:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 08:01:40 +0200 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <20080427231354.M42594@npgx.com.au> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> <20080427231354.M42594@npgx.com.au> Message-ID: <1209362500.22738.466.camel@localhost.localdomain> Hi Michael, On Mon, 2008-04-28 at 10:15 +1100, Michael Mansour wrote: > Hi Rainer, > > > I am on a cell phone... > > > > Check out the v3 compatibility document. Its available right from > > the doc main page. > > Yes, from that document I found the option: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > Which made it go back to the older time format. While reading it more though, > I think I will actually keep it on the new format as the precision time isn't > actually too bad. It may break some of my scripts which analyse daily log > files and if it does, I'll enable the traditional format again, but for now > I'll wait and see if anything breaks. > Yes, scripts is the big Problem. Rsyslog actually had high-precision timestamps right from the beginning, but nobody ever used them. So I decided to change the default ;) On the scripts: I am thinking about a simple converter that takes high precision log files and converts them to old-style low precision ones. Such a converter could be run before the actual script. Do you (or someone else) think this is useful and worth the effort? Rainer > Thanks. > > Michael. > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Michael Mansour" > > An: "rsyslog-users" > > Gesendet: 27.04.08 18:13 > > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > > > Hi, > > > > The date format logged to the messages/maillog/etc has changed from: > > > > Apr 27 21:36:58 > > > > to: > > > > 2008-04-27T21:44:35.788251+10:00 > > > > I'm new to v3 so how can I change this format back to the original? > > > > Thanks. > > > > Michael. > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > ------- End of Original Message ------- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Mon Apr 28 14:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 14:34:52 +0200 Subject: [rsyslog] rsyslog 3.16.0 on el4 In-Reply-To: <20080427115258.M2863@npgx.com.au> References: <20080427115258.M2863@npgx.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F01@grfint2.intern.adiscon.com> Michael, this was contributed code. We even thought about dropping these files altogether. But as it looks, they seem to be of some value. If you have a corrected version, I'd appreciate if you could send me these, so that I can include them into the tarball- I have very limited platform know-how, so I am depending on other folks to send me startup scripts and such... Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Mansour > Sent: Sunday, April 27, 2008 1:56 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog 3.16.0 on el4 > > Hi, > > I just created a spec file and compiled an RPM for this version. > > In the source is the ./redhat/rsyslog.init file. This file references > klogd > through, so when I ended up installing the RPM, I couldn't start > rsyslog > using the script as it would exit. > > If klogd is no longer used in this rsyslog version, could you > modify/correct > the rsyslog.init file so that it no longer makes reference to it? > > I've already commented out the relevant entries in the file so I could > start > up rsyslogd. > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From smalenshek at skyline-ats.com Mon Apr 28 17:01:36 2008 From: smalenshek at skyline-ats.com (Stephen Malenshek) Date: Mon, 28 Apr 2008 09:01:36 -0600 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions Message-ID: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] Sent: Monday, April 28, 2008 8:55 AM To: rsyslog at lists.adiscon.com; Rainer Gerhards Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to "buffer" it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this. It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. In my present configuration, I am doing an if/then like the following: if $source == 'somemachine' \ and $syslogfacility-text == 'authpriv' \ then ?LocalSecure $template LocalSecure,"/var/log/secure" With this, how can I define multiple directions within this template? > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site. This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.h tml rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. At each site, there will be a single server tat is aggregating all SYSLOG data before transmitting it to the central repository. Are you talking about the machine sending the aggregate information or the hosts, I.E. router, server, switch, etc., that we will be monitoring? > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called "customerid" that I would like to do this > with. In this, I would like to designate a name or integer like "1" > for site A, "2" for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. I need a little more direction on this. I believe I understand, but I want to make absolutely sure. Could you provide me an example of what would be within the remote server's config and what the central server config would look like? > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this.? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html I want to look for things like the following: Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, terminal=ssh res=success)' With this, I want to be notified with the host information, address, etc. via e-mail if someone logs into a particular host as "root". > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. I have taken a good look at it and will integrate it within our administration interface. It looks like an excellent product and will save me a lot of code to write. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer From rgerhards at hq.adiscon.com Mon Apr 28 22:28:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 22:28:26 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F0F@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General > Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would > like to use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like > you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have > them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? I do not fully get you - do you mean how to write to different files based on some message property? If so, you can use the property replacer to do so: $template LocalSecure,"/var/log/%HOSTNAME%/secure" > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* > @{IP Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to > firewalls). Also, you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plai n-tcp-syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is > aggregating all SYSLOG > data before transmitting it to the central repository. Are > you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? Ah, ok. The host is taken from the message, and as such depends on the system that formatted it. You can define a template to consistently define the hostname on the relay. However, that requires that no further relay is between the aggregator and the original senders. > > > > > and 2) tag that information with a specific account code > identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like > to do this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, > if I have > > > to, with a subquery within the insert query to another > table to lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being > separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property > replace is > > part of the template system. It's quite powerful, but you > need to play a > > bit with it. > > > > I need a little more direction on this. I believe I > understand, but I want > to make absolutely sure. Could you provide me an example of > what would be > within the remote server's config and what the central server > config would > look like? > I can't do a lab right now (sorry, busy), but you can try along these lines. On the server's config, use a template along those lines: $template tpl,"%TIMESTAMP% %HOSTNAME% %syslogtag%,%msg:::drop-last-lf%\n" Where sitename is to be replaced by a site name (e.g. 1, SiteA ...). On the central server, you can extract that field inside a template by such a property specification: %msg:F,44:1% Again, all of this is untested but should either work or is pretty close to working ;) > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples > of efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably > has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, > address, etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our > clients use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used > for storage > > > of the rest of SYSLOG data, and I have the associated > columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare > schema", which is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will > define a special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, > so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the > parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product > and will save > me a lot of code to write. If you have any feedback, comments, suggestions or feature request, please let us know. It is very much in its infancy and Andre, the lead developer, is working very hard on it. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but > disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From rgerhards at hq.adiscon.com Tue Apr 29 16:27:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Apr 2008 16:27:33 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F1D@grfint2.intern.adiscon.com> I created a small wiki entry that has details on how to use a site id inside the message. I hope it is useful: http://wiki.rsyslog.com/index.php/Splitting_messages_based_on_a_site_ID Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would like to > use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* @{IP > Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to firewalls). Also, > you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is aggregating all > SYSLOG > data before transmitting it to the central repository. Are you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? > > > > > and 2) tag that information with a specific account code identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like to do > this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, if I have > > > to, with a subquery within the insert query to another table to > lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property replace > is > > part of the template system. It's quite powerful, but you need to play > a > > bit with it. > > > > I need a little more direction on this. I believe I understand, but I > want > to make absolutely sure. Could you provide me an example of what would > be > within the remote server's config and what the central server config > would > look like? > > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples of > efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, address, > etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our clients > use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used for > storage > > > of the rest of SYSLOG data, and I have the associated columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare schema", which > is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will define a > special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product and will > save > me a lot of code to write. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From infofarmer at FreeBSD.org Tue Apr 1 08:36:16 2008 From: infofarmer at FreeBSD.org (Andrew Pantyukhin) Date: Tue, 1 Apr 2008 10:36:16 +0400 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <20080401063614.GD4692@amilo.cenkes.org> On Mon, Mar 31, 2008 at 07:52:37PM +0200, Michael Biebl wrote: > I don't see the use case yet. Why would someone want to build > the plugins without the rsyslogd binary? Does it actually work > if you mix different versions of plugins and rsyslogd? If > someone is interested in only a specific plugin, why not use > "make -C plugins/foo"? It's just a convenience feature. In FreeBSD, we have separate ports for core and each plugin. At present, each plugin has to build rsyslogd all over again. From now on, it can just require the core port and save all the wasted build-time. I'm sure there are other ways to do it (like the one you mentioned), but is there a merit in arguing which one's the best? From rgerhards at hq.adiscon.com Tue Apr 1 08:50:30 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:50:30 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: <20080401063614.GD4692@amilo.cenkes.org> References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> <20080401063614.GD4692@amilo.cenkes.org> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CC9@grfint2.intern.adiscon.com> As a side-note: > Does it actually work > if you mix different versions of plugins and rsyslogd? The answer is: it depends. Rsyslog uses versioning in the plugin interface and also uses versioning in all internal objects. But there are still some non-object accesses. As long as a plugin is of the same version as the plugin interface that rsyslogd is compiled to, it works - provided that all objects it needs are implemented in the core (for v3, this must not necessarily be the case, especially if you use a new plugin with an older rsylogd build). The bottom line, of course, is that mixing versions should be avoided. But the good news is that rsyslog and the plugins detect incompatible versions and, if found, the plugin is simply not activated. So it is a predictable failure case (again, not 100% safe for the v3 case as it is not purely object based so far). Rainer From rgerhards at hq.adiscon.com Tue Apr 1 08:56:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 08:56:45 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com><1206988699.9240.38.camel@cutter><1206991339.9240.45.camel@cutter> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Related question: does anybody know of a (sufficiently good) library that abstracts differences between TLS implementation. I mean in the way libdbi does for databases? It's obviously not important to have more than one driver available at runtime, but it would be necessary to be able to link against different TLS implementation. If anybody has a suggestion, please let me know. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, March 31, 2008 9:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] openssl vs rsyslog > > Whatever I do - I'll try to create a small driver shim so that > hopefully > different libraries could be used together with rsyslog. It depends on > the APIs, but in general that should not be too hard (but practice > often > tells the difference). > > On FIPS I made I comment, where I think the mails crossed. > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Monday, March 31, 2008 9:27 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] openssl vs rsyslog > > > > 2008/3/31, seth vidal : > > > On Mon, 2008-03-31 at 21:18 +0200, Michael Biebl wrote: > > > > Interesting to know. Do you know any technical > > advantages of NSS over > > > > GnuTLS (stability, features, nicer API, etc)? > > > > > > > > > openssl transition library > > > fips certification. > > > > > > http://www.mozilla.org/projects/security/pki/nss/fips/ > > > > > > the second one being the most important, I think. > > > > Yeah, the openssl transition library doesn't concern us. > > > > Why is the fips certification important for rsyslog? > > > > Cheers, > > Michael > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 1 09:19:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 09:19:28 +0200 Subject: [rsyslog] rsyslog 3.13.0-dev0 released In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CB6@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CCB@grfint2.intern.adiscon.com> > > > It fixes two bugs, one is a potential segfault in the > syslog/plain tcp > > > receiver. The other one is removal of some debug instrumentation > that > > > accidently made it into 3.12.5. There is also a new ./configure > option > > > (--enable/disable-rsyslogd) which permits to build just specific > plugins > > > > > > Hm, the default of enable_rsyslogd should imho be "yes" That was a bug, which is now fixed. > Forgot to add: If you intend to build rsyslogd and rfc3195d > conditionally, the corresponding man pages should also be installed > conditionally. This is now done. Thanks for pointing both out. Rainer From pvrabec at redhat.com Tue Apr 1 11:12:43 2008 From: pvrabec at redhat.com (Peter Vrabec) Date: Tue, 1 Apr 2008 11:12:43 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> Message-ID: <200804011112.43694.pvrabec@redhat.com> Hi, On Tuesday 01 April 2008 08:56:45 am Rainer Gerhards wrote: > Related question: does anybody know of a (sufficiently good) library > that abstracts differences between TLS implementation. I mean in the way > libdbi does for databases? It's obviously not important to have more > than one driver available at runtime, but it would be necessary to be > able to link against different TLS implementation. > > If anybody has a suggestion, please let me know. you can look at nss_compat_ossl http://rcritten.fedorapeople.org/nss_compat_ossl.html http://fedoraproject.org/wiki/nss_compat_ossl but when I talked to guy who work on Fedora Crypto Consolidation it was suggested not to use the abstract library, since all these TLS implementations have kind of different "philosophy". It would be much cleaner to use separate plugins for different TLS. Note, that we will appreciate a lot going with NSS. http://fedoraproject.org/wiki/FedoraCryptoConsolidation http://fedoraproject.org/wiki/CryptoConsolidationEval > Thanks, > Rainer From rgerhards at hq.adiscon.com Tue Apr 1 13:50:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 13:50:53 +0200 Subject: [rsyslog] openssl vs rsyslog In-Reply-To: <200804011112.43694.pvrabec@redhat.com> References: <002201c8935b$0c6287e8$060013ac@intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CC8@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CCA@grfint2.intern.adiscon.com> <200804011112.43694.pvrabec@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD1@grfint2.intern.adiscon.com> Hi all, > http://fedoraproject.org/wiki/FedoraCryptoConsolidation > http://fedoraproject.org/wiki/CryptoConsolidationEval This material (the big picture) is quite convincing. Are there similar efforts in other distros? To make a point, I'll probably start developing an as slim as possible abstraction layer as a side-project (much like librelp) and see what I can do with it. I'll probably start with NSS and then see what it take to move the other libs in. My initial focus will be on "get it going", so it will not be a very secure solution at first. All comments are still very welcome and will be considered. I just thought it would be good to do a sum-up of my current thinking. Rainer From rgerhards at hq.adiscon.com Tue Apr 1 19:00:19 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Apr 2008 19:00:19 +0200 Subject: [rsyslog] rsyslog 3.15.0 released - with RELP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD7@grfint2.intern.adiscon.com> Hi all, yes, I know it's April, 1st. But this release is not related to it ;) I have just released rsyslog 3.15.0, the release that finally provides initial RELP support. RELP provides superior reliability over plain tcp syslog, ensuring that no messages are lost. With plain TCP, this can happen if the remote server goes down. RELP protects from this by utilizing a full-duplex protocol where each message is acknowledged. RELP should also be safe to use with stunnel, where plain tcp syslog sometimes gets into trouble. The core RELP protocol support is NOT part of rsyslog. It comes in the form of librelp, available at http://www.librelp.com . You need to install librelp before you try to compile rsyslog with RELP support. That should be fairly simple. If you run into any troubles, please let me know - I am more than happy to help. I plan to do some more in-depth doc on the use cases soon. Change Log: http://www.rsyslog.com/Article203.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-93.phtml I hope this release is useful. Feedback is appreciated. Rainer From BLentz at channing-bete.com Tue Apr 1 21:53:16 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 15:53:16 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem Message-ID: <47F292AC.8040307@channing-bete.com> Greetings, list! I'm having trouble compiling 3.15.0. I get the following error on one machine (CentOS 5): rsyslogd-msg.o: In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch_4' And slightly different error on another machine (Fedora Core 3): rsyslogd-debug.o(.text+0x183d): In function `dbgEntrFunc': /usr/src/redhat/BUILD/rsyslog-3.15.0/debug.c:996: undefined reference to `__sync_fetch_and_add' rsyslogd-msg.o(.text+0x127): In function `msgDestruct': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to `__sync_sub_and_fetch' rsyslogd-msg.o(.text+0xc55): In function `MsgAddRef': /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:449: undefined reference to `__sync_fetch_and_add' Has anyone else seen this or know of a workaround or fix? I can't seem to find any reference in the rsyslog install documentation which says that I need additional libraries on my system. Thanks in advance From mbiebl at gmail.com Tue Apr 1 23:10:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:10:56 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F292AC.8040307@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Ben Lentz : > Greetings, list! > I'm having trouble compiling 3.15.0. > > I get the following error on one machine (CentOS 5): > rsyslogd-msg.o: In function `msgDestruct': > /usr/src/redhat/BUILD/rsyslog-3.15.0/msg.c:249: undefined reference to > `__sync_sub_and_fetch_4' See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 #undef DO_HAVE_ATOMICS 1 should help Obviously, ./configure should detect this automatically. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Tue Apr 1 23:23:47 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Tue, 1 Apr 2008 23:23:47 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: 2008/4/1, Michael Biebl : > > Obviously, ./configure should detect this automatically. We could steal from here: http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From BLentz at channing-bete.com Tue Apr 1 23:25:08 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Tue, 01 Apr 2008 17:25:08 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <47F2A834.9000509@channing-bete.com> > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > #undef DO_HAVE_ATOMICS 1 should help > > Obviously, ./configure should detect this automatically. > > Thanks! I'm still getting errors with debug.c It seems that debug.c has a reference to ATOMIC_INC (line 996) without a set of ifdef DO_HAVE_ATOMICS around it, so the build still dies. If I comment that line out, it will compile, but the binary is unusable "rsyslogd run failed with error -3000." On a whim, I tried ./configure --disable-debug, no luck. Ugh! From rgerhards at hq.adiscon.com Wed Apr 2 08:38:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Apr 2008 08:38:40 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F2A834.9000509@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> Message-ID: <1207118320.22738.3.camel@localhost.localdomain> It's a bug, obviously. I have now fixed it. Please go to atomic.h and change the ATOMIC_INC macro to be #define ATOMIC_INC(data) (++(data)) That'll solve the problems. Doing so does not introduce any real weakness - the code has been tested always with this setting. The atomics are a recent addition to make it ultra-reliable on all CPU architectures ... but obviously we need to do more work on the feature selection than I initially thought... Thanks for bringing this up, it was just at the right time. I'll pull that code from today's stable release (that happens when you think you can add something laaate in the process because it is "unintrusive" ;)). Rainer On Tue, 2008-04-01 at 17:25 -0400, Ben Lentz wrote: > > > > See http://git.michaelbiebl.de/?p=rsyslog.git;a=commitdiff;h=26aa8b09dbc2de1c7bdd97921a273511d585e043 > > > > #undef DO_HAVE_ATOMICS 1 should help > > > > Obviously, ./configure should detect this automatically. > > > > > Thanks! > > I'm still getting errors with debug.c It seems that debug.c has a > reference to ATOMIC_INC (line 996) without a set of ifdef > DO_HAVE_ATOMICS around it, so the build still dies. > > If I comment that line out, it will compile, but the binary is unusable > "rsyslogd run failed with error -3000." > > On a whim, I tried ./configure --disable-debug, no luck. Ugh! > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 08:40:27 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 08:40:27 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: References: <47F292AC.8040307@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CD9@grfint2.intern.adiscon.com> Ummm ;) Could you lend me a helping hand? That looks god, but it also looks like it goes above my autotools horizon ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Tuesday, April 01, 2008 11:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > 2008/4/1, Michael Biebl : > > > > Obviously, ./configure should detect this automatically. > > We could steal from here: > http://gcc.gnu.org/ml/libstdc++/2006-07/txt00001.txt > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 10:35:29 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 04:35:29 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <1207118320.22738.3.camel@localhost.localdomain> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> Message-ID: <47F34551.4010902@channing-bete.com> > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > change the ATOMIC_INC macro to be > > #define ATOMIC_INC(data) (++(data)) > > That'll solve the problems. Hey Rainer, Thanks for the help. I still continue to have problems, now at runtime. I'm patching the source like so: --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 @@ -36,8 +36,8 @@ #define INCLUDED_ATOMIC_H /* set the following to 1 if we have atomic operations (and #undef it otherwise) */ -#define DO_HAVE_ATOMICS 1 -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) +#undef DO_HAVE_ATOMICS 1 +#define ATOMIC_INC(data) (++(data)) #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) #endif /* #ifndef INCLUDED_ATOMIC_H */ then: rsyslogd run failed with error -3000 I've compiled with ggdb and run through the debugger, but I'm not sure where to set the break points... tried a couple things but nothing worked. I've also run it with strace, but I can't seem to find any problems. I believe there's a ton of reasons that it would generate this message and I'm not sure where to start. From rgerhards at hq.adiscon.com Wed Apr 2 10:40:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:40:35 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com><1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CDF@grfint2.intern.adiscon.com> Hi, > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 10:35 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > > > > It's a bug, obviously. I have now fixed it. Please go to atomic.h and > > change the ATOMIC_INC macro to be > > > > #define ATOMIC_INC(data) (++(data)) > > > > That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 - > 0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > That's fine.. > then: > rsyslogd run failed with error -3000 Nice - error codes are defined in rsyslog.h. But of course, -3000 means "an error occurred". In any case, it's something that fails cleanly... > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate > this > message and I'm not sure where to start. ... that's why you don't see anything in the debugger. So let's get a bit down on where it really happens. There is debug support inside rsyslog: http://www.rsyslog.com/doc-debug.html It would be best if you do ./configure --enable-rtinst but that's not strictly necessary. Set RSYSLOG_DEBUG to LogFuncFlow and run rsyslog interactively with -d -n. You'll get some debug output. That should tell us something (well, you could even start without seting RSYSLOG_DEBUG, I think that'll be sufficient and less output). Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From theinric at redhat.com Wed Apr 2 10:46:34 2008 From: theinric at redhat.com (theinric at redhat.com) Date: Wed, 02 Apr 2008 10:46:34 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34551.4010902@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain> <47F34551.4010902@channing-bete.com> Message-ID: <47F347EA.7000501@redhat.com> On 04/02/2008 10:35 AM, Ben Lentz wrote: > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h and >> change the ATOMIC_INC macro to be >> >> #define ATOMIC_INC(data) (++(data)) >> >> That'll solve the problems. > > Hey Rainer, > Thanks for the help. I still continue to have problems, now at runtime. > I'm patching the source like so: > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 -0400 > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > @@ -36,8 +36,8 @@ > #define INCLUDED_ATOMIC_H > > /* set the following to 1 if we have atomic operations (and #undef it > otherwise) */ > -#define DO_HAVE_ATOMICS 1 > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > +#undef DO_HAVE_ATOMICS 1 > +#define ATOMIC_INC(data) (++(data)) > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > then: > rsyslogd run failed with error -3000 > > I've compiled with ggdb and run through the debugger, but I'm not sure > where to set the break points... tried a couple things but nothing > worked. I've also run it with strace, but I can't seem to find any > problems. I believe there's a ton of reasons that it would generate this > message and I'm not sure where to start. Hi Ben, my guess is that rsyslog can't find its libraries (e.g. /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" or something like "ln -s $(find /path/to/rsyslog/sources/ -name "*.so") /usr/[local/]lib/rsyslog". You can run "strace ./rsyslogd -v" to see where it looks for its libraries. From rgerhards at hq.adiscon.com Wed Apr 2 10:48:28 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 10:48:28 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F347EA.7000501@redhat.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> You are probably right. After your message I checked the module loader and it just returns the generic error code. I'll fix that... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of theinric at redhat.com > Sent: Wednesday, April 02, 2008 10:47 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog-3.15.0 compile problem > > On 04/02/2008 10:35 AM, Ben Lentz wrote: > > > >> It's a bug, obviously. I have now fixed it. Please go to atomic.h > and > >> change the ATOMIC_INC macro to be > >> > >> #define ATOMIC_INC(data) (++(data)) > >> > >> That'll solve the problems. > > > > Hey Rainer, > > Thanks for the help. I still continue to have problems, now at > runtime. > > I'm patching the source like so: > > > > --- rsyslog-3.15.0/atomic.h.orig 2008-03-31 05:07:24.000000000 > -0400 > > +++ rsyslog-3.15.0/atomic.h 2008-04-02 02:50:54.000000000 -0400 > > @@ -36,8 +36,8 @@ > > #define INCLUDED_ATOMIC_H > > > > /* set the following to 1 if we have atomic operations (and #undef > it > > otherwise) */ > > -#define DO_HAVE_ATOMICS 1 > > -#define ATOMIC_INC(data) ((void) __sync_fetch_and_add(&data, 1)) > > +#undef DO_HAVE_ATOMICS 1 > > +#define ATOMIC_INC(data) (++(data)) > > #define ATOMIC_DEC_AND_FETCH(data) __sync_sub_and_fetch(&data, 1) > > > > #endif /* #ifndef INCLUDED_ATOMIC_H */ > > > > then: > > rsyslogd run failed with error -3000 > > > > I've compiled with ggdb and run through the debugger, but I'm not > sure > > where to set the break points... tried a couple things but nothing > > worked. I've also run it with strace, but I can't seem to find any > > problems. I believe there's a ton of reasons that it would generate > this > > message and I'm not sure where to start. > > Hi Ben, > > my guess is that rsyslog can't find its libraries (e.g. > /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" > or something like "ln -s $(find /path/to/rsyslog/sources/ -name > "*.so") /usr/[local/]lib/rsyslog". > You can run "strace ./rsyslogd -v" to see where it looks for its > libraries. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 11:03:12 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 05:03:12 -0400 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com> <577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> Message-ID: <47F34BD0.8060606@channing-bete.com> > You are probably right. After your message I checked the module loader > and it just returns the generic error code. I'll fix that... > > Rainer > > >> Hi Ben, >> >> my guess is that rsyslog can't find its libraries (e.g. >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make install" >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name >> "*.so") /usr/[local/]lib/rsyslog". >> You can run "strace ./rsyslogd -v" to see where it looks for its >> libraries. >> Yep, you're right. I was expecting: - Shared library issues at runtime to be visible using ldd(1) - To at least be able to get command-line help or version (-v) information from rsyslogd (without loadable modules) - A nicer error message ;-) strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file or directory) Can the module path (/usr/lib/rsyslog) be specified? Is there a ./configure option that I'm missing at compile time or a command line parameter to rsyslogd or config file parameter in rsyslog.conf? I've been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and can't seem to find anything? It's very early in the US and I might just be missing something obvious. From rgerhards at hq.adiscon.com Wed Apr 2 11:07:03 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 11:07:03 +0200 Subject: [rsyslog] rsyslog-3.15.0 compile problem In-Reply-To: <47F34BD0.8060606@channing-bete.com> References: <47F292AC.8040307@channing-bete.com> <47F2A834.9000509@channing-bete.com> <1207118320.22738.3.camel@localhost.localdomain><47F34551.4010902@channing-bete.com> <47F347EA.7000501@redhat.com><577465F99B41C842AAFBE9ED71E70ABA308CE1@grfint2.intern.adiscon.com> <47F34BD0.8060606@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CE2@grfint2.intern.adiscon.com> > >> my guess is that rsyslog can't find its libraries (e.g. > >> /usr/local/lib/rsyslog/lmregexp.so). You'll need to do "make > install" > >> or something like "ln -s $(find /path/to/rsyslog/sources/ -name > >> "*.so") /usr/[local/]lib/rsyslog". > >> You can run "strace ./rsyslogd -v" to see where it looks for its > >> libraries. > >> > Yep, you're right. I was expecting: > - Shared library issues at runtime to be visible using ldd(1) > - To at least be able to get command-line help or version (-v) > information from rsyslogd (without loadable modules) > - A nicer error message ;-) > > strace ./rsyslogd -v -b -n 2>&1 | grep ENOENT > access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or > directory) > open("/usr/lib/rsyslog/lmnet.so", O_RDONLY) = -1 ENOENT (No such file > or > directory) > > Can the module path (/usr/lib/rsyslog) be specified? Is there a > ./configure option that I'm missing at compile time or a command line > parameter to rsyslogd or config file parameter in rsyslog.conf? I've > been through ./configure --help, rsyslogd.8 and rsyslog.conf.5 and > can't > seem to find anything? That's mostly distro-specific and this is where I have very limited knowledge. But you can set the module path via the environment, use RSYSLOG_MODDIR=/path/to/modules/ . There is also a -m option to do the same, but it currently is read too late to help with the core modules. On my 64Bit system adding --libdir=/lib64 often helped with such problems when I encountered them. Not sure if that's the right cure... > > It's very early in the US and I might just be missing something > obvious. You are *really* brave! From rgerhards at hq.adiscon.com Wed Apr 2 16:34:37 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 16:34:37 +0200 Subject: [rsyslog] RELP vs plain tcp syslog Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Hi folks, I just blogged about the unreliability of plain tcp syslog. This may be interesting for at least some of you: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl og.html Rainer From Gerrard.Geldenhuis at datacash.com Wed Apr 2 17:24:57 2008 From: Gerrard.Geldenhuis at datacash.com (Gerrard Geldenhuis) Date: Wed, 2 Apr 2008 16:24:57 +0100 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: Hi Thanks, that is good to know. Worth keeping in mind. I was under the impression tcp was the miracle cure for udp unreliability but it seems not. There is very expensive software out there that can do guarenteed packet deliver over tcp... tibco is one I have seen before. Regards > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: 02 April 2008 15:35 > To: rsyslog-users > Subject: [rsyslog] RELP vs plain tcp syslog > > Hi folks, > > I just blogged about the unreliability of plain tcp syslog. This may be > interesting for at least some of you: > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-sysl > og.html > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Wed Apr 2 17:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Apr 2008 17:34:56 +0200 Subject: [rsyslog] RELP vs plain tcp syslog In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308CF1@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308CFA@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Gerrard Geldenhuis > Sent: Wednesday, April 02, 2008 5:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] RELP vs plain tcp syslog > > Hi > Thanks, that is good to know. Worth keeping in mind. I was under the > impression tcp was the miracle cure for udp unreliability but it seems > not. > > There is very expensive software out there that can do guarenteed > packet > deliver over tcp... Well... RELP can *really* do the trick. It needs some help in the rsyslog engine to work under all cases, that is when the relp stack is terminated (when the client rsyslog is shut down). But that isn't rocket science and has just been pushed back by more urgent work (securing the channel). So far, relp does single ack, that is the server ack's every packet back to the client. Client discards packets only after ack. So when a session breaks, we know what the server has processed. If, however, some of the acks get lost, we resend packets that the server already processed, resulting in some mild message *duplication* (we currently have a 128 packet app-layer max window and typically acks come in rather quickly). To work around this, the client must ack the server's acks (double-acking). That sounds scary, but is not. It's also not performance intense and the protocol is modeled that those with ultra-slim bandwidth can turn it off and live with the message duplication scenario. If double-acks are active, relp client and server will go into a recovery phase after reconnect negotiation, in which mutally acked package numbers are exchanged. To do so, the client must provide a session cookie back to the server. This is a potential attack vector and thus I'd like to have secure transport in place before I do it. Once the recovery negotiation is done, client and server know, and have discarded, what each other peer has processed. The remaining packets are re-sent, but under the same reliability settings. So if the connection breaks at this point again (or during the renegotiation), we'll simply go into another recovery phase until we finally succeed. It is of course important that session caches be persisted to disk when an engine stops - that will be part of rsyslog. The recovery produced itself is librelp. Once this is done, and with proper rsyslog queue settings, sufficiently stable hardware and sufficient disk space, I guarantee (not in a lawyers sense, though ;)) that you'll never lose a message nor get a duplicate. This is when I think we have achieved our reliability goal. If all goes well... summer? ;) Rainer > > tibco is one I have seen before. > > Regards > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: 02 April 2008 15:35 > > To: rsyslog-users > > Subject: [rsyslog] RELP vs plain tcp syslog > > > > Hi folks, > > > > I just blogged about the unreliability of plain tcp syslog. This may > be > > interesting for at least some of you: > > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > sysl > > og.html > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From BLentz at channing-bete.com Wed Apr 2 23:10:39 2008 From: BLentz at channing-bete.com (Ben Lentz) Date: Wed, 02 Apr 2008 17:10:39 -0400 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter Message-ID: <47F3F64F.5090600@channing-bete.com> When I try to use $HOSTNAME in an expression-based filter, I get "INVALID PROPERTY" errors logged by the system. From the documentation, I'm lead to believe that expression-based filters use the same properties as expression-based filters, which are documented in the property replacer documentation. Okay. But it seems that maybe $HOSTNAME has been changed to $source in the expression-based filters, but I can't seem to find any evidence of this in the man or HTML documentation. From rgerhards at hq.adiscon.com Thu Apr 3 08:52:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Apr 2008 08:52:31 +0200 Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter In-Reply-To: <47F3F64F.5090600@channing-bete.com> References: <47F3F64F.5090600@channing-bete.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D04@grfint2.intern.adiscon.com> Ah, good report. I see where the problem is stemming from. We use the same code for property access no matter which method is used (expression-based, property-based, ...). That's good. The bad thing is that property names were case-sensitive (a really bad idea). We were in the discussion to remove that. The RainerScript engine already is case-insensitive and consequently converts all variable names to lower case. However, the message object property access method has not yet been converted, so it expects it in upper case. The result is the mess that you see... I'll initiate a quick discussion, but I think we'll now drop case-sensitivity at all. That shouldn't hurt any existing installations but instead make life easier. Thanks for bringing this up. For the same reason "$source" is the solution and I have seen in the forum that it works for you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ben Lentz > Sent: Wednesday, April 02, 2008 11:11 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Use of $HOSTNAME in Expression-based filter > > When I try to use $HOSTNAME in an expression-based filter, I get > "INVALID PROPERTY" errors logged by the system. From the documentation, > I'm lead to believe that expression-based filters use the same > properties as expression-based filters, which are documented in the > property replacer documentation. Okay. But it seems that maybe > $HOSTNAME > has been changed to $source in the expression-based filters, but I > can't > seem to find any evidence of this in the man or HTML documentation. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 10:06:55 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:06:55 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: 2008/3/31, Rainer Gerhards : > Well... First thing is that I have urgent need to release -- there are a > couple of things in the queue. If I don't release them this week, I'll > probably don't have a chance to get to a stable any time soon. So I will > release, even at the price that the version number scheme may be less > than optimum this time. > > But now on to the real meat (and thanks for sticking around on this > topic! ;))... I like the idea of the odd/even numbering (of the minor number) to distinguish unstable/stable release. E.g. GNOME uses it and it seems to work fine for them. E.g. the latest stable release was 2.22.0 (and minor point/bug fix releases usually follow as 2.22.1, 2.22.2...) The major development is going on in 2.23. The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... As soon as the code base stabilises (reaching beta/rc quality), they will release 2.23.90,2.23.91,... and the final stable release will be 2.24.0. (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From r.bhatia at ipax.at Fri Apr 4 10:16:01 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 04 Apr 2008 10:16:01 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> Message-ID: <47F5E3C1.9090207@ipax.at> Michael Biebl wrote: > I like the idea of the odd/even numbering (of the minor number) to > distinguish unstable/stable release. E.g. GNOME uses it and it seems > to work fine for them. > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > releases usually follow as 2.22.1, 2.22.2...) > The major development is going on in 2.23. > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > As soon as the code base stabilises (reaching beta/rc quality), they > will release > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). its not new as afairc the linux kernel established this scheme. anyways, what bugs me is, that we then will have something like: 3.20.x - stable 3.21.x - unstable relp no 3.22.x 3.23.x - unstable tls when we then decide to implement yet-another-new-feature-in-a- seperate-tree, we will end up with a scenario like 3.20.x - oldstable 3.21.x - unstable relp - closed 3.23.x - unstable tls 3.25.x - unstable featureX 3.26.x - stable or similar. that is kind of odd ... rainer: what do you think about only having one dev tree and having some patchsets for not-that-much-worked-on features (e.g. tls, featureX). of course, you can arrange your repository as you wish, but i would stick to *one* stable version and *one* dev version for the testers. everything else should be build (as patches) on top of these 2 branches. as a nice effect, we will have only 1 stable core and 1 dev core and have no issues when it comes to merging core-relp with core-tls with core-featureX. cheers, raoul -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From mbiebl at gmail.com Fri Apr 4 10:45:09 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 10:45:09 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: 2008/4/4, Raoul Bhatia [IPAX] : > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered correctly). > > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable > > or similar. > > that is kind of odd ... > git feature branches would be ideal for this kind of development imho. > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 branches. > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. It would definitely make it easier if there was only one stable and one development tree (and maybe a third one, oldstable). Rainer could work on experimental new features in separate git feature branches and when he considers them ready for testing, merge them into the dev branch (usually called master in git). E.g. the relp functionality could be in a branch called feature-relp, tls in feature-tls etc. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:06:35 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:06:35 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <47F5E3C1.9090207@ipax.at> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA4@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CA9@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Ah, the source of the misunderstanding finally hits me... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Raoul Bhatia [IPAX] > Sent: Friday, April 04, 2008 10:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > Michael Biebl wrote: > > I like the idea of the odd/even numbering (of the minor number) to > > distinguish unstable/stable release. E.g. GNOME uses it and it seems > > to work fine for them. > > > > E.g. the latest stable release was 2.22.0 (and minor point/bug fix > > releases usually follow as 2.22.1, 2.22.2...) > > The major development is going on in 2.23. > > The first unstable release will be 2.23.1, followed by 2.23.2, > 2.23.3,... > > As soon as the code base stabilises (reaching beta/rc quality), they > > will release > > 2.23.90,2.23.91,... and the final stable release will be 2.24.0. > > (no -pre,-alpha, -beta suffixes, only numbers, which are ordered > correctly). > > its not new as afairc the linux kernel established this scheme. > > anyways, what bugs me is, that we then will have something like: > > 3.20.x - stable > 3.21.x - unstable relp > no 3.22.x > 3.23.x - unstable tls > > when we then decide to implement yet-another-new-feature-in-a- > seperate-tree, we will end up with a scenario like The trees are NOT independent of each other! There is always just one development tree. In the above scenario, 3.23.x *includes* 3.21.x. Let's look what I currently try: We will finally see 3.14.0 stable today. I have also done a number of improvements since the relp version. So I have (not yet released) 3.14.0 - stable, bug fixes only 3.15.0 - everything that is in stable, plus relp - set to stabilize, bug fixes only 3.17.0 - everything that is in 3.15.x (note the .x not .0) plus new features - devel Development only happens in 3.17.0. When a feature focus has been reached, I now plan to branch off that release and let it stabilize (that is no new features, just bug fixes). I am trialing this currently with the 3.15.x branch. So far, it looks manageable. The plan is to slowly migrate 3.15.x to become 3.16.0, the next stable, deprecating 3.14.x when it comes out. I think that'll on average will be every two month, but I don't know yet. With 3.15.x it may be quicker, as the whole RELP code is in an external library and thus the core engine has not been destabilized. > > 3.20.x - oldstable > 3.21.x - unstable relp - closed > 3.23.x - unstable tls > 3.25.x - unstable featureX > 3.26.x - stable So this picture would actually be: 3.20.x - oldstable 3.21.x - unstable relp - closed 3.11.x - stable 3.23.x - unstable tls 3.25.x - unstable tls+featureX That actually bring us down to three branches (plus old legacy like v2 stable): - stable - stabilizing (branched off the dev tree and in the bugfix wait queue - devel There may be more than one stabilizing tree at a given time (not yet thought out). In recent history, we have one such sample. That is the queue engine. It was an extremely big overhaul of nearly everything. So it needed some extended time to mature. In the mean time, some more focus features could be implemented. With the version numbering system I have now on my mind, that would probably have looked as follows (I am intentionally using major version 999 to avoid confusion with existing versions): 999.2.0 - stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.9.x - devel Actually, we are happy with the feature introduced in 999.7.x. But we won't release a new stable because .5 is not yet ready for prime time. So nothing happens at that point. Then, we are confident in .5. I think the following happens then: 999.2.0 - deprecated stable 999.3.x - big overhaul feature, stabilizing 999.5.x - .3 + next focus feature, stabilizing 999.7.x - .5 + next focus feature, stabilizing 999.8.0 - stable 999.9.x - devel We will never see a 999.6. I personally think this is acceptable, especially as I think it won't happen very often. Does that make more sense? > or similar. > > that is kind of odd ... > > rainer: what do you think about only having one dev tree and > having some patchsets for not-that-much-worked-on features > (e.g. tls, featureX). of course, you can arrange your repository as > you wish, but i would stick to *one* stable version and *one* dev > version for the testers. > > everything else should be build (as patches) on top of these 2 > branches. I am a bit concerned that I need to be too careful which patch I apply where. As you know, rsyslog is developed on a very fast pace and I'd rather prefer to keep coding instead of thinking where I add a "by-feature" (that maybe takes an hour to implement). The scheme I am currently trialing (outlined above) works pretty OK. It is some additional overhead, but I mostly do everything to the trunk, create a patch for bugfixes and apply that to the "old" version in question. Some more work than before, but quite bearable. Of course things are different if I work on a feature, abandon it, and go back after a few weeks. But currently I implement everything up to a usable state (maybe not 100%, but useful in what is there) before I take the next step. Again... comments please ;) Rainer > as a nice effect, we will have only 1 stable core and 1 dev core and > have no issues when it comes to merging core-relp with core-tls with > core-featureX. > > cheers, > raoul > -- > ____________________________________________________________________ > DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at > Technischer Leiter > > IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at > Barawitzkagasse 10/2/2/11 email. office at ipax.at > 1190 Wien tel. +43 1 3670030 > FN 277995t HG Wien fax. +43 1 3670030 15 > ____________________________________________________________________ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 11:19:59 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 11:19:59 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 999.2.0 - stable > 999.3.x - big overhaul feature, stabilizing > 999.5.x - .3 + next focus feature, stabilizing > 999.7.x - .5 + next focus feature, stabilizing > 999.9.x - devel > > Again... comments please ;) I think you really should try to use git feature branches for that. Have a stable and master (development) branch, and develop the features in separate topic branches feature-A, feature-B etc. Whenever one feature is ready, merge it into master. This way, it doesn't matter which feature you have to concentrate on is released first (no skipped version numbers!). The strong merge suppport in git would also allow to cherrypick easily from the different feature branches or merge between them. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 11:42:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 11:42:42 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308CAB@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> > 2008/4/4, Rainer Gerhards : > > > > 999.2.0 - stable > > 999.3.x - big overhaul feature, stabilizing > > 999.5.x - .3 + next focus feature, stabilizing > > 999.7.x - .5 + next focus feature, stabilizing > > 999.9.x - devel > > > > Again... comments please ;) > > I think you really should try to use git feature branches for that. > Have a stable and master (development) branch, and develop the > features in separate topic branches feature-A, feature-B etc. > Whenever one feature is ready, merge it into master. > This way, it doesn't matter which feature you have to concentrate on > is released first (no skipped version numbers!). > The strong merge suppport in git would also allow to cherrypick easily > from the different feature branches or merge between them. Sounds good, but a honest question (NOT trying to give a bias, just a problem description): While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and implement them - all while FocusFeatureX is being developed. Where do I apply these? And don't I get into trouble if that interferes with things that I do in FocusFeatureX? Remember, I change a couple of hundered lines all over the project on a typical day... Rainer > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Fri Apr 4 12:02:20 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Fri, 4 Apr 2008 12:02:20 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com> <47F0E4A8.50207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com> <47F0FBFA.3020802@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com> <47F5E3C1.9090207@ipax.at> <577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: 2008/4/4, Rainer Gerhards : > > 2008/4/4, Rainer Gerhards : > > > > > > 999.2.0 - stable > > > 999.3.x - big overhaul feature, stabilizing > > > 999.5.x - .3 + next focus feature, stabilizing > > > 999.7.x - .5 + next focus feature, stabilizing > > > 999.9.x - devel > > > > > > Again... comments please ;) > > > > I think you really should try to use git feature branches for that. > > Have a stable and master (development) branch, and develop the > > features in separate topic branches feature-A, feature-B etc. > > Whenever one feature is ready, merge it into master. > > This way, it doesn't matter which feature you have to concentrate on > > is released first (no skipped version numbers!). > > The strong merge suppport in git would also allow to cherrypick easily > > from the different feature branches or merge between them. > > > Sounds good, but a honest question (NOT trying to give a bias, just a > problem description): > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > implement them - all while FocusFeatureX is being developed. Where do I > apply these? And don't I get into trouble if that interferes with things > that I do in FocusFeatureX? Remember, I change a couple of hundered > lines all over the project on a typical day... Say you work on a featureA branch. Now you get an unrelated feature request for featureB. You'd switch back to current master, and branch of featureB, starting to work on that. By the end of the day, say featureB is ready, you'd merge those branch back into master (and delete branch featureB if no longer required). If featureC is dependend on featureA, you can branch from there. If you now work again on featureA, and later on featureC, you can merge the new commits from featureA back into featureC again. Later, when featureA and C are ready, you merge them into master again. For small changes, I'd directly work on master and commit there. There is also a nice feature called git-stash, which allows to put uncommitted changes away, work temporarily on other stuff, an get back to the uncommited stuff later. I'd say, just test git and try to get a "feeling" for it. That probably helps to make a better decision. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Fri Apr 4 12:17:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 12:17:33 +0200 Subject: [rsyslog] rsyslog version numbering In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> OK, let me come to a conclusion ;) What Michael writes and Raoul suggested makes an awful lot of sense. Right now, I have two problems: a) we are still having a bit of trouble with the git transistion of rsyslog - I hope to have that sorted out soon b) I need to invest some time to fully understand how git branches. The bigger problem is probably b). Thankfully, I have started to work with git on librelp and it was a very good experience. It still looks like I need to invest at least a day or two more into getting fully involved with it. That doesn't sound much, but there is a lot I can do in rsyslog in this time. I think I will proceed as follows: For the next few weeks, I'll use the scheme that I outlined this morning. It works and it is sufficiently clean for the time being. Especially as I don't see any reason for gaps, there is no such major overhaul in sight. While I do so, I'll get more acquainted to git and see how I can make utilize its branching capabilities. At some point in time (and if everything works as well as advertised, what I assume ;)), I'll switch to the git feature branch strategy. My hope is all this can be done in the next 10 weeks or so. I hope I don't disappoint anyone - but the problem is things to do. All needs to go by priorities and, quite honestly, TLS or the new config file format is higher on my priority scale than the branching strategy. And, yes, I know good knowledge with git will save in the long run. But I need to start somewhere ;) I someone has serious concerns on the route I am taking, please scream now ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Friday, April 04, 2008 12:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > 2008/4/4, Rainer Gerhards : > > > 2008/4/4, Rainer Gerhards : > > > > > > > > 999.2.0 - stable > > > > 999.3.x - big overhaul feature, stabilizing > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > 999.9.x - devel > > > > > > > > Again... comments please ;) > > > > > > I think you really should try to use git feature branches for > that. > > > Have a stable and master (development) branch, and develop the > > > features in separate topic branches feature-A, feature-B etc. > > > Whenever one feature is ready, merge it into master. > > > This way, it doesn't matter which feature you have to concentrate > on > > > is released first (no skipped version numbers!). > > > The strong merge suppport in git would also allow to cherrypick > easily > > > from the different feature branches or merge between them. > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just a > > problem description): > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests and > > implement them - all while FocusFeatureX is being developed. Where > do I > > apply these? And don't I get into trouble if that interferes with > things > > that I do in FocusFeatureX? Remember, I change a couple of hundered > > lines all over the project on a typical day... > > Say you work on a featureA branch. Now you get an unrelated feature > request for featureB. > You'd switch back to current master, and branch of featureB, starting > to work on that. > By the end of the day, say featureB is ready, you'd merge those branch > back into master (and delete branch featureB if no longer required). > If featureC is dependend on featureA, you can branch from there. If > you now work again on featureA, and later on featureC, you can merge > the new commits from featureA back into featureC again. > Later, when featureA and C are ready, you merge them into master again. > For small changes, I'd directly work on master and commit there. There > is also a nice feature called git-stash, which allows to put > uncommitted changes away, work temporarily on other stuff, an get back > to the uncommited stuff later. > > I'd say, just test git and try to get a "feeling" for it. That > probably helps to make a better decision. > > Cheers, > Michael > > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Fri Apr 4 17:21:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Apr 2008 17:21:38 +0200 Subject: [rsyslog] First rsyslog v3 stable released - 3.14.1 Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D3C@grfint2.intern.adiscon.com> Hi all, I am glad to finally announce rsyslog 3.14.1, the first version of the v3 stable branch. This version offers almost all features of the current development version to those in need of a stable branch. Among others, this includes support for multiple database backends, queued and offline operations, SNMP and text file support. It is a big step compared to the v2 stable branch. Users are advised to check the compatibility notes before the update. It's not strictly necessary, but will enable to use rsyslog in the most efficient and problem-free way. Please note that development continues inside the v3 branch. The v3 stable branch will see future feature enhancements after they have sufficiently matured. The v2 stable branch is still supported. It is quite featureless (compared to v3) but extremely solid. So if you are (or need to be) ultra-conservative, you can still take the v2 route. Feature-wise v2 is a dead end and only bug fixes will be provided. The general recommendation is that the v3 stable branch be used for regular production machines. The Fedora project will feature rsyslog v3 stable in its upcoming release 9. Please note that I made a mistake two days ago: I accidently released 3.14.0 to the web, without it being actually ready. For this reason, I have renamed the release to 3.14.1. There will never be an official 3.14.0 release. If you happen to have it downloaded and installed, please accept my apologies. You should get 3.14.1 whenever you are ready. Change Log: http://www.rsyslog.com/Article205.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-94.phtml I hope this release is useful. As always, feedback is appreciated. Best regards, Rainer Gerhards From rgerhards at hq.adiscon.com Mon Apr 7 11:21:31 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Apr 2008 11:21:31 +0200 Subject: [rsyslog] rsyslog on GIT now - RE: rsyslog version numbering In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308C4B@grfint2.intern.adiscon.com><47F0E4A8.50207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB1@grfint2.intern.adiscon.com><47F0FBFA.3020802@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308CB8@grfint2.intern.adiscon.com><47F5E3C1.9090207@ipax.at><577465F99B41C842AAFBE9ED71E70ABA308D21@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308D25@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308D28@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D51@grfint2.intern.adiscon.com> Hi folks, co-incidentally, CVS has just begun to give me some pain last Friday, when it came to actually working with the different branches I now have. And a rainy Sunday made my have a deep look at the git manual. End result: I won't wait 10 weeks from now but have started to finally convert. I have pulled Michael Biebl's git repository, which he thankfully kept synchronized with the CVS. I am doing some final checks right now, but as it looks we will be using git from now on. Rsyslog is available from Adiscon's gitweb: http://git.adiscon.com/ It can also be pulled via git protocol. Please keep in mind that it may receive some more finishing touches. I will now have four active branches: - v2-stable - v3-stable - beta -- what becomes the next release of v3-stable - master -- the current (b)leading development edge There is also v1-stable, which is deprecated and a few legacy branches. As suggested by Raoul and Michael, I'll now begin to work with feature-branches (as soon as I have finished current work in progress). I am new to git and I may mess up things. Bear with me if I do ;) Please note that I am still working on the initial setup, so if you pull the repository now, you may need to pull it again some time later ;) If I mess up, I'll let you know via the mailing list. I hope this will be a good move. Special thanks to Michael and Raoul, who were persistent enough to finally made me move (and, well, to CVS which provided the final bit of motivation ;)). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Friday, April 04, 2008 12:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog version numbering > > OK, let me come to a conclusion ;) > > What Michael writes and Raoul suggested makes an awful lot of sense. > Right now, I have two problems: > > a) we are still having a bit of trouble with the git transistion of > rsyslog - I hope to have that sorted out soon > b) I need to invest some time to fully understand how git branches. > > The bigger problem is probably b). Thankfully, I have started to work > with git on librelp and it was a very good experience. It still looks > like I need to invest at least a day or two more into getting fully > involved with it. That doesn't sound much, but there is a lot I can do > in rsyslog in this time. > > I think I will proceed as follows: > > For the next few weeks, I'll use the scheme that I outlined this > morning. It works and it is sufficiently clean for the time being. > Especially as I don't see any reason for gaps, there is no such major > overhaul in sight. > > While I do so, I'll get more acquainted to git and see how I can make > utilize its branching capabilities. At some point in time (and if > everything works as well as advertised, what I assume ;)), I'll switch > to the git feature branch strategy. My hope is all this can be done in > the next 10 weeks or so. > > I hope I don't disappoint anyone - but the problem is things to do. All > needs to go by priorities and, quite honestly, TLS or the new config > file format is higher on my priority scale than the branching strategy. > And, yes, I know good knowledge with git will save in the long run. But > I need to start somewhere ;) > > I someone has serious concerns on the route I am taking, please scream > now ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Friday, April 04, 2008 12:02 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] rsyslog version numbering > > > > 2008/4/4, Rainer Gerhards : > > > > 2008/4/4, Rainer Gerhards : > > > > > > > > > > 999.2.0 - stable > > > > > 999.3.x - big overhaul feature, stabilizing > > > > > 999.5.x - .3 + next focus feature, stabilizing > > > > > 999.7.x - .5 + next focus feature, stabilizing > > > > > 999.9.x - devel > > > > > > > > > > Again... comments please ;) > > > > > > > > I think you really should try to use git feature branches for > > that. > > > > Have a stable and master (development) branch, and develop the > > > > features in separate topic branches feature-A, feature-B etc. > > > > Whenever one feature is ready, merge it into master. > > > > This way, it doesn't matter which feature you have to > concentrate > > on > > > > is released first (no skipped version numbers!). > > > > The strong merge suppport in git would also allow to cherrypick > > easily > > > > from the different feature branches or merge between them. > > > > > > > > > Sounds good, but a honest question (NOT trying to give a bias, just > a > > > problem description): > > > > > > While I implement FocusFeatureX, I get Feature 1, 2, 3 requests > and > > > implement them - all while FocusFeatureX is being developed. Where > > do I > > > apply these? And don't I get into trouble if that interferes with > > things > > > that I do in FocusFeatureX? Remember, I change a couple of > hundered > > > lines all over the project on a typical day... > > > > Say you work on a featureA branch. Now you get an unrelated feature > > request for featureB. > > You'd switch back to current master, and branch of featureB, starting > > to work on that. > > By the end of the day, say featureB is ready, you'd merge those > branch > > back into master (and delete branch featureB if no longer required). > > If featureC is dependend on featureA, you can branch from there. If > > you now work again on featureA, and later on featureC, you can merge > > the new commits from featureA back into featureC again. > > Later, when featureA and C are ready, you merge them into master > again. > > For small changes, I'd directly work on master and commit there. > There > > is also a nice feature called git-stash, which allows to put > > uncommitted changes away, work temporarily on other stuff, an get > back > > to the uncommited stuff later. > > > > I'd say, just test git and try to get a "feeling" for it. That > > probably helps to make a better decision. > > > > Cheers, > > Michael > > > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 8 18:42:39 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Apr 2008 18:42:39 +0200 Subject: [rsyslog] rsyslog 3.17.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D85@grfint2.intern.adiscon.com> Hi all, rsyslog 3.17.0 has just been released. It is part of the development branch. The primary new feature is the ability to send email alerts based on syslog or file data. The action engine now has the ability to carry out an action only once within a configured interval and/or only during specific time frames. Option processing has been improved and a number of stability updates have been included. This is a recommended update for all users of the development version. More information on the major new feature can be found here: http://www.rsyslog.com/doc-ommail.html The following is a sample code snippet that alerts the operator when disk problems are detected: $ModLoad ommail $ActionMailSMTPServer mail.example.net $ActionMailFrom rsyslog at example.net $ActionMailTo operator at example.net $template mailSubject,"disk problem on %hostname%" $template mailBody,"RSYSLOG Alert\r\nmsg='%msg%'" $ActionMailSubject mailSubject # make sure we receive a mail only once in six # hours (21,600 seconds ;)) $ActionExecOnlyOnceEveryInterval 21600 # the if ... then ... mailBody mus be on one line! if $msg contains 'hard disk fatal failure' then :ommail:;mailBody Note that we now have the ability to limit an action to be executed only once inside a specific period. In the above sample, the email alert happens only if there was no other such alert within the past 6 hours - this is absolutely vital to prevent an accidental DoS on your mailbox ;) ... but it may also be handy with other actions (e.g. SNMP trap notification etc etc). It's implemented at the action engine level, so it will work with any action, even file or database writes. Change Log: http://www.rsyslog.com/Article207.phtml Download: http://www.rsyslog.com/Downloads-req-getit-lid-95.phtml I would be most interested in feedback on the new email feature, including clever use cases. I am sure it can be quite useful (especially if you think about imfile...), but would really appreciate to hear from you (and prove this in practice)! Thanks, Rainer Gerhards From aoz.syn at gmail.com Wed Apr 9 01:47:51 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:47:51 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 Message-ID: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Please forgive me if this is addressed in later versions; I'm just working against the version that was pulled into Gentoo yesterday. The atomic operations used in debug.c and msg.c were not introduced until gcc-4.1.0; as such, systems compiling with an earlier version (i.e. hardened-gentoo) fail compilation with undefined references to __sync_fetch_and_add and __sync_sub_and_fetch. I'm not sure of the right way to check for this in atomic.h, but it would be helpful if a check was made before setting DO_HAVE_ATOMICS. In addition, although msg.c makes appropriate checks for DO_HAVE_ATOMICS, debug.c does not and fails to compile under such systems. I would supply a patch, but being completely new to your codebase thought I'd start with reporting. Thanks for your time! RB From aoz.syn at gmail.com Wed Apr 9 01:53:23 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:53:23 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> Message-ID: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> > The atomic operations used in debug.c and msg.c were not introduced > until gcc-4.1.0; Forgot to give specifics: the atomic operations as defined in atomic.h. The change made to bring debug.c to 1.49 in particular. From aoz.syn at gmail.com Wed Apr 9 01:55:49 2008 From: aoz.syn at gmail.com (RB) Date: Tue, 8 Apr 2008 17:55:49 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> Message-ID: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Strike that. Even better if I just read the ML before posting... *sigh* Long day, guys, sorry for the noise. On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > The atomic operations used in debug.c and msg.c were not introduced > > until gcc-4.1.0; > > Forgot to give specifics: the atomic operations as defined in > atomic.h. The change made to bring debug.c to 1.49 in particular. > From rgerhards at hq.adiscon.com Wed Apr 9 09:10:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:10:23 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com><4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> No problem at all, I think it's well-hidden in the ML. A good fix (via autoconf is still not done, so I should probably add a bug tracker ;). Please keep reporting. All others please note that I will release updates for v3-stable and beta soon. Its brewing ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, April 09, 2008 1:56 AM > To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] Atomic operations in 3.15.0 > > Strike that. Even better if I just read the ML before posting... > *sigh* > > Long day, guys, sorry for the noise. > > On Tue, Apr 8, 2008 at 5:53 PM, RB wrote: > > > The atomic operations used in debug.c and msg.c were not > introduced > > > until gcc-4.1.0; > > > > Forgot to give specifics: the atomic operations as defined in > > atomic.h. The change made to bring debug.c to 1.49 in particular. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From janfrode at tanso.net Wed Apr 9 09:33:02 2008 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 9 Apr 2008 09:33:02 +0200 Subject: [rsyslog] git repo ? Message-ID: I just noticed "- converted to git" on http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html Is there a repo available online somewhere ? If not, maybe you could get http://github.com/ to host it so that there's a nice web interface to it and is easily forkable for others to work on it ? -jf From rgerhards at hq.adiscon.com Wed Apr 9 09:34:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 09:34:56 +0200 Subject: [rsyslog] git repo ? In-Reply-To: References: Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308D8D@grfint2.intern.adiscon.com> Oops... too much going on? I thought I had posted it to the ML. Anyhow... it's at http://git.adiscon.com - git protocol is enabled Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Jan-Frode Myklebust > Sent: Wednesday, April 09, 2008 9:33 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] git repo ? > > I just noticed "- converted to git" on > http://rgerhards.blogspot.com/2008/04/rsyslog-work-log_08.html > > Is there a repo available online somewhere ? If not, maybe you > could get http://github.com/ to host it so that there's a nice > web interface to it and is easily forkable for others to work > on it ? > > > > -jf > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From aoz.syn at gmail.com Wed Apr 9 17:08:22 2008 From: aoz.syn at gmail.com (RB) Date: Wed, 9 Apr 2008 09:08:22 -0600 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> Message-ID: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> On 4/9/08, Rainer Gerhards wrote: > No problem at all, I think it's well-hidden in the ML. A good fix (via > autoconf is still not done, so I should probably add a bug tracker ;). I don't know about the remainder of the atomic changes you've made, but a check for >=gcc-4.1.0 will suffice for the stuff that's in atomic.h. I must say - I can't express how excited I was to find this project. It's been on my long-term plate for quite some time to fork syslog-ng and add all the nice features the Pro license grants; now I don't have to! I'm partial to the syslog-ng configuration syntax, but can most definitely live with this for not having to re-implement all that stuff. Tell me, are you also working on UDF-WORM? ;) From rgerhards at hq.adiscon.com Wed Apr 9 18:17:42 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Apr 2008 18:17:42 +0200 Subject: [rsyslog] rsyslog v3-stable/3.14.2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DA8@grfint2.intern.adiscon.com> Hi all, I have just released 3.14.2, an update to v3-stable. It is a purely bug-fixing release. Most importantly, it fixes a problem that caused imklog not to pull module symbols correctly from recent kernels. Also, a segfault caused by using expression-based filters is fixed. There are also some other fixes, see ChangeLog for details. This is a recommended update for all v3-stable branch users. Changelog: http://www.rsyslog.com/Article209.phtml Download (now pointing to the info page so that you see the md5sum ;)): http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-96.phtml I hope this release is useful. As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 9 18:51:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 09 Apr 2008 18:51:04 +0200 Subject: [rsyslog] Atomic operations in 3.15.0 In-Reply-To: <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> References: <4255c2570804081647y6f5c8253q5547147370e3ecb6@mail.gmail.com> <4255c2570804081653s5b1c51cere9270ec774f44a53@mail.gmail.com> <4255c2570804081655m73097e3lc0323196eb54cb01@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA308D8C@grfint2.intern.adiscon.com> <4255c2570804090808l4b5b165bo22117786268c4a2c@mail.gmail.com> Message-ID: <1207759864.22738.94.camel@localhost.localdomain> On Wed, 2008-04-09 at 09:08 -0600, RB wrote: > On 4/9/08, Rainer Gerhards wrote: > > No problem at all, I think it's well-hidden in the ML. A good fix (via > > autoconf is still not done, so I should probably add a bug tracker ;). > > I don't know about the remainder of the atomic changes you've made, > but a check for >=gcc-4.1.0 will suffice for the stuff that's in > atomic.h. I just began, more outstanding... But it looks like that easy solution is the right one - thanks! > I must say - I can't express how excited I was to find this project. > It's been on my long-term plate for quite some time to fork syslog-ng > and add all the nice features the Pro license grants; now I don't have > to! Good to hear ;) And, believe it or not, I never ever installed syslog-ng. But,of course, it's the primary competitor in this space. IMHO we are now far ahead of it (I guess you already know that chart, but...): http://www.rsyslog.com/doc-rsyslog_ng_comparison.html Bazsi seem currently to brew something in regard to the log store, which is not (yet ;)) my priority. Also I don't think it helps to cryptographically sign the store (at least for court), the message itself must be signed (that's my route on that). So we will probably see a few features in the *paid* edition of syslog-ng that rsyslog does not yet have. Even with that, and even compared to the paid edition, I think we are now far ahead. > I'm partial to the syslog-ng configuration syntax, but can most > definitely live with this for not having to re-implement all that > stuff. rsyslog's config file syntax is just plain ugly. No way around. But wait a bit, I am working on a scripting engine: http://rgerhards.blogspot.com/2008/02/introducing-rainerscript-and-some.html http://www.rsyslog.com/doc-rscript_abnf.html > Tell me, are you also working on UDF-WORM? ;) Neither of them - just the boring stuff ;) Rainer From rgerhards at hq.adiscon.com Thu Apr 10 14:53:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 14:53:40 +0200 Subject: [rsyslog] Status of TLS development Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Hi folks, I just wanted to give you an update on my native TLS plans. I've been searching around and trying things the past days. As it looks now, I will use a wrapper class to cover nss and gnutls. Contrary to what I initially thought, I'll now probably start with GNUTls. The reason is that it seems to be much easier to start with it. For some insight, please follow this discussion on the NSS newsgroup: http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread /45e418ca0a556a42 That doesn't mean I settle for just GNUTls. Even if I start with it, NSS support will follow. I hope it will be easier to implement when I can check the NSS manuals for specific functionality. I thought I let you know. Rainer From mbiebl at gmail.com Thu Apr 10 15:29:06 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:29:06 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Hi folks, > > I just wanted to give you an update on my native TLS plans. I've been > searching around and trying things the past days. As it looks now, I > will use a wrapper class to cover nss and gnutls. Contrary to what I > initially thought, I'll now probably start with GNUTls. The reason is > that it seems to be much easier to start with it. For some insight, > please follow this discussion on the NSS newsgroup: > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/thread > /45e418ca0a556a42 > > That doesn't mean I settle for just GNUTls. Even if I start with it, NSS > support will follow. I hope it will be easier to implement when I can > check the NSS manuals for specific functionality. > > I thought I let you know. Is it technically necessary to implement the layer via a separate library (libsci), or could the layer also be implemented directly within rsyslog as (libtool) convenience lib? The point is, that a separate library means more works for packagers and people who want to compile rsyslog themselves. If you think, that libsci might be useful for other projects, then implementing it as a system library makes sense (or due to licensing matters). If libsci will only be used within rsyslog though, I think implementing it as a libtool convenience lib within rsyslog makes more sense. Just my 2? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:31:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:31:38 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Michael, I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Depending on how it works out, it may be useful for others, too. But that's not the primary focus. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:29 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Hi folks, > > > > I just wanted to give you an update on my native TLS plans. I've > been > > searching around and trying things the past days. As it looks now, I > > will use a wrapper class to cover nss and gnutls. Contrary to what I > > initially thought, I'll now probably start with GNUTls. The reason > is > > that it seems to be much easier to start with it. For some insight, > > please follow this discussion on the NSS newsgroup: > > > > > http://groups.google.com/group/mozilla.dev.tech.crypto/browse_frm/threa > d > > /45e418ca0a556a42 > > > > That doesn't mean I settle for just GNUTls. Even if I start with it, > NSS > > support will follow. I hope it will be easier to implement when I > can > > check the NSS manuals for specific functionality. > > > > I thought I let you know. > > Is it technically necessary to implement the layer via a separate > library (libsci), or could the layer also be implemented directly > within rsyslog as (libtool) convenience lib? > > The point is, that a separate library means more works for packagers > and people who want to compile rsyslog themselves. > > If you think, that libsci might be useful for other projects, then > implementing it as a system library makes sense (or due to licensing > matters). > If libsci will only be used within rsyslog though, I think > implementing it as a libtool convenience lib within rsyslog makes more > sense. > > Just my 2? > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:38:18 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:38:18 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Michael, > > I fully agree, but licensing is the culprit. I think it'll be either LGPL or BSD. I'd really prefer to plug it directly into librelp and rsyslog, but that will create license conflicts. Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be fine with rsyslogs GPLv3). Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:40:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:40:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go to link to GPLv3. Sorry... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Thursday, April 10, 2008 3:38 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > 2008/4/10, Rainer Gerhards : > > Michael, > > > > I fully agree, but licensing is the culprit. I think it'll be either > LGPL or BSD. I'd really prefer to plug it directly into librelp and > rsyslog, but that will create license conflicts. > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > fine with rsyslogs GPLv3). > > Cheers, > Michael > > > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Thu Apr 10 15:43:01 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:43:01 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com><577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Bad typo. Should have been: librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go to link to GPLv3. Sorry... > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, April 10, 2008 3:40 PM > To: rsyslog-users > Subject: Re: [rsyslog] Status of TLS development > > librelp is dual-licensed - GPLv2 + commercial license. So it is a no-go > to link to GPLv3. Sorry... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > > Sent: Thursday, April 10, 2008 3:38 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Status of TLS development > > > > 2008/4/10, Rainer Gerhards : > > > Michael, > > > > > > I fully agree, but licensing is the culprit. I think it'll be > either > > LGPL or BSD. I'd really prefer to plug it directly into librelp and > > rsyslog, but that will create license conflicts. > > > > Where do you see the conflict? GnuTLS is LGPLv2.1+ (compatible with > > GPLv3) and NSS is either MPL1.1, GPLv2+ or LGPLv2.1+ (which should be > > fine with rsyslogs GPLv3). > > > > Cheers, > > Michael > > > > > > -- > > Why is it that all of the instruments seeking intelligent life in the > > universe are pointed away from Earth? > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From mbiebl at gmail.com Thu Apr 10 15:50:34 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:50:34 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Rainer Gerhards : > Bad typo. Should have been: > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > to link to GPLv3. Sorry... GPLv3+/commercial is not incompatible with LGPLv2.1+ Am I missing something? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 15:56:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 15:56:12 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: <1207835772.22738.132.camel@localhost.localdomain> On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > 2008/4/10, Rainer Gerhards : > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ If I put the crypto wrapper (code named so far libsci) into *rsyslog* the crypto wrapper will be under GPLv3. If librelp needs TLS functionality, it must link to the crypto-wrapper *inside* rsyslog, which is GPLv3. librelp can not do that if it is not used inside a GPLv3 program (because, in essence, I would need to detouch those parts from rsyslog that are needed by librelp and then add that to the non-GPLv3 application). Or am I overlooking something? A prominent example of what will use librelp functionality and is not under GPLv3 is Adiscon's MonitorWare products under Windows (and keep in mind that they currently sponsor rsyslog development ;)). Rainer From mbiebl at gmail.com Thu Apr 10 15:55:45 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 15:55:45 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> Message-ID: 2008/4/10, Michael Biebl : > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > to link to GPLv3. Sorry... > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > Am I missing something? The whole point of the LGPL is, to be useful for commercial/closed source applications. And GPLv3+ and LGPLv2.1+ is no problem either. I'm not a lawyer though, so take this with a grain of salt. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From mbiebl at gmail.com Thu Apr 10 16:01:56 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Thu, 10 Apr 2008 16:01:56 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: <1207835772.22738.132.camel@localhost.localdomain> References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: 2008/4/10, Rainer Gerhards : > > On Thu, 2008-04-10 at 15:50 +0200, Michael Biebl wrote: > > 2008/4/10, Rainer Gerhards : > > > Bad typo. Should have been: > > > > > > librelp is dual-licensed - GPLv3 + commercial license. So it is a no-go > > > > > > to link to GPLv3. Sorry... > > > > > > GPLv3+/commercial is not incompatible with LGPLv2.1+ > > > If I put the crypto wrapper (code named so far libsci) into *rsyslog* > the crypto wrapper will be under GPLv3. > > If librelp needs TLS functionality, it must link to the crypto-wrapper Ok, this was probably my misconception. I thought, only rsyslog would need the TLS functionality. It wasn't clear to me, that *both* rsyslog and librelp would link against libsci. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Thu Apr 10 16:34:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Apr 2008 16:34:33 +0200 Subject: [rsyslog] Status of TLS development In-Reply-To: References: <577465F99B41C842AAFBE9ED71E70ABA308DBA@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DBF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC0@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA308DC1@grfint2.intern.adiscon.com> <1207835772.22738.132.camel@localhost.localdomain> Message-ID: <1207838073.22738.144.camel@localhost.localdomain> On Thu, 2008-04-10 at 16:01 +0200, Michael Biebl wrote: > Ok, this was probably my misconception. > I thought, only rsyslog would need the TLS functionality. > It wasn't clear to me, that *both* rsyslog and librelp would link > against libsci. Sorry, probably I've been to brief. Let me sort out things at least now: Both rsyslog and librelp need native TLS functionality. For rsyslog, this is to support plain TCP syslog via TLS (hopefully compatible with the like syslog-ng and MonitorWare feature) as well as the upcoming IETF syslog-transport-tls standard. Librelps needs to do native TLS because it can not rely on an external entity (like stunnel) and claim to be reliable and pretty tamperproof. So it must talk to the remote peer directly. In RELP protocol, there will be a STARTTLS command verb, just like in SMTP. Negotiation on the availability of this verb is similar to SMTP's EHLO message or BEEP's greeting message. Both the RELP protocol as well as librelp already contain the necessary provisioning. My intention is to make libsci available to anyone who may have a need for it, but it will be focused on what I actually need. Please note that I have intentionally called it libsci - simple *crypto* interface - and not something like libsti - simple *tls* interface - because I intend to place all crypto functions into a single library (thus relieving at least a bit of the burden from an extra library). Other than TLS, rsyslog will need to support digital signatures as well as generic hashes (not only for cryptographic needs, but also for some other nice things). If JF finally convinces me to implement some of the advanced output plugin functionality as completely separate projects [he is seeding that thought quite well ;)], these projects will most probably also need to link to it. My intention is to put libsci under a very liberal license, probably either a BSD style one or at least LGPL. I have no idea yet when we will see any of libsci functionality in practice ;) I hope this finally clarifies. Sorry for the confusion. Please drop me any questions that are still around. Rainer From rgerhards at hq.adiscon.com Fri Apr 11 10:04:12 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Apr 2008 10:04:12 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... Message-ID: <1207901052.22738.230.camel@localhost.localdomain> Hi folks, I am sorry, this will be a long mail. But I would appreciate if you'd read it in full and comment on it. This mail covers a really important decision for rsyslog and will probably even influence if the project succeeds in the long term. Package maintainers and code contributors are especially requested to *really* read it. Though I t try hard to provide all relevant facts (that's why it is getting long), I will probably miss some and not properly convey others. Please feel free to ask. Let me start with explaining that the rsyslog project conceptually consists of three parts: - the modules - "helper" functions - rsyslog-specific functionality Modules are actually projects in their own right, just being distributed with the rsyslog tarball for convenience. A module may be released under any license. Note that modules call both rsyslog-specific functionality (e.g. to submit a message) as well as helper function (e.g. to handle tcp sessions). The "helper" functions are a growing set of generic objects. Examples are the module loader, the queue engine, networking support, the script engine and virtual machine, ... - you get the idea: Things that are used inside rsyslog but are not necessarily of use only for rsyslog. Actually, this could be called a "rsyslog runtime library". Rsyslog-specific functionality is primarily rsyslogd and everything it takes to glue together helpers and plugins to build the working syslog subsystem. Let's stick with that for a brief while. Now let me explain the idea of loosely coupled modules. This stems back to JF's effort to convince me to the Unix philosophy of "small tools that work well together". We had another good discussion yesterday (on the blog) and it made me change my mind a bit (well, probably, not 100% convinced yet, but it he managed to seed the thought ;)). While I still think that there are things that need to be really tightly coupled to the rsyslog core, there are others which not necessarily need to. Let me call the later "loosely coupled modules", in contrast to the (tightly coupled) plugins that actually become part of the rsyslogd process during runtime. The analysis plugins I have on my mind could become such loosely coupled modules. As an interface, the usual Unix "send it and forget it" pipe could be used, and it would probably be acceptable to allow for minor message loss during shutdown and plugin failure (anything else would require a pipe application protocol, e.g. relp over pipe, which sounds scary). The plus in doing so would be the ability to use those plugins in configurations where rsyslog is not present (e.g. driven by syslog-ng or a detached message generator [fed from a log file]). Done right, one could even select the (sligly lossy) pipe interface or the full blown plugin interface as a compile time switch. If you think it out, we may even end with an abstraction layer where each module can be compiled for either the plugin interface or the pipe (no promises, though). One problem with this approach is that modules call into the rsyslog helpers. For example, rsyslog's network support need to be available for all those modules that do something over the net. That's not a problem if I have a tightly coupled plugin as today (the rsyslog core makes the necessary bindings). It would become more problematic if I move the module to a pipe interface, because I now need to find a way to use the rsyslog objects. But that's still doable (though pretty ugly). It becomes really problematic if the same module, using a pipe interface, is to be used with e.g. syslog-ng. I don't think that syslog-ng will be able to provide it with an emulated rsyslog "net" object. Let's stick with this problem for a moment. Coincidentally, we had another discussion on the mailing list yesterday - on the TLS support wrapper for rsyslog and librelp. That discussion centered around licenses. Technically, there are also a number of issues. I have now involved myself enough with GnuTLS and a bit of NSS so that I am able to try draft a first abstraction layer. I thought hard and the "right solution" involves encapsulation stream network access. So the right thing to do is to have one object that handles network streams. That object then is configured to use either plain tcp, TLS (via whatever library) or even GSS-API. Nice and clean. It gets dirty if I think about the details. If I do it that way, it makes rsyslog depend on this object (so-far codenamed libsci). However, that would mean that any rsyslog installation would need to pull in libsci. Not a big deal, except, right, except if the crypto libraries are also pulled in by libsci. So would every desktop system running rsyslog need to have the crypto libraries installed? Scary... unacceptable. In rsyslog, we had the same problem a few month ago (at that time the mysql client libraries were the problem). The solution was the rsyslog loader, which dynamically loads other libraries (and their dependencies) on demand. The loader is what enables rsyslogd to be installed everywhere, but only with minimal core requirements (and have everything else in separate packages). So if libsci would be part of rsyslog, we would not have any problem at all. After all, the necessary plumbing is ready at hand in form of the rsyslog helper objects. This is where we come back to loosely coupled modules. You notice it is the same problem? Both them as well as libsci would need to call the rsyslog helpers. Now let's come a bit to licensing. In order to understand that, we need to talk about rsyslog funding first. Obviously, I am spending full time (and a bit more than that) on rsyslog for quite a while now. I even intend to do that for some more months as rsyslog is currently mabye 55% of what I would like it to be. Somehow I must get funding - for the time, for the hardware and for all the other things ;) What made the rsyslog project possible, and still 99.9% funds it is Adiscon, the company that provides (of course ;)) the best-ever logging solutions on Windows. Actually the Windows closed source pays for the rsyslog project. While we hope to find other sources of funding in the future, I can not ignore the fact. Once thing we would like to do at Adiscon is include select parts of the technology I am now developing into the closed source applications, too. The most prominent example is the RELP protocol. I obviously find this a fair policy - after all, the alternative would be to do it in closed source only and I was able to convince my folks at Adiscon that it is far better to contribute to the open source world. There is one drawback in this requirement: licensing. Of course, we could pick a BSD style license and every problem would be solved. But, I have to admit, we do not like to give everything to our competitors in the *closed source* space. We have made very bad experiences with folks building on our technology and even turning it against us. I won't get agreement from Adiscon to use a BSD license for everything (plus I personally, too, don't like to see that effect). We already discussed this on the mailing list here as part of dual-licensing in the past. The solution was that the technology in question was created as its own, dual-licensed, project. This lead to the creation of librelp. Rsyslog itself was left under GPLv3 (which I sincerely believe in because of its anti-patent, anti-drm clauses - even though the license gives myself obviously some troubles). Dual-licensing librelp lead to some duplicate code and made me not use some features which I could have used if I had access to all rsyslog helper objects. For librelp, that is not yet a big deal, because it is quite unique, very few needs to access the rsyslog helpers. With TLS, however, the situation changes and we get the dangling issue of rsyslog helpers in librelp, too. LETS TRY TO WRAP-UP If I put all of this together, I think I have taken a (slightly? ;)) wrong path. The core problem is monolithic design from a very high point of view. I have to admit I think this is what JF and some others were pointing out, but I didn't realize it quickly enough. Sure, rsyslog is quite modular by now. But rsyslog always requires rsyslog to do everything. It is very hard to do any rsyslog-related work without the rsyslog core. While rsyslog has a carefully crafted set of helper objects, these are not exposed to the outside world. And the licensing issues associated with that design begin to screw up everything in the long run. I think we need change. The obvious solution seems to be extracting the rsyslog helpers out of the rsyslog core project and create a "rsyslog runtime". That runtime than could individually be installed and be put under a different license (bear with me, explanation follows below). Let's consider a complicated case with the runtime. Assume we have a plugin "NeverBeforeSeenAnalysis". Let's say someone wants to use it with syslog-ng (!). With the runtime, all needed would be to compile it for the pipe interface and install rsyslog-runtime and the module onto the system. Now let's consider Adiscon's MonitorWare products on Windows. When they implement RELP, they need librelp and can pull in the rsyslog-runtime (for network access including TLS). For rsyslogd itself nothing really happens, the runtime is now just its own library - linking to it needs not to be modified. So for rsyslogd, the change would be transparent. Technically, this indeed solves the issues. Let me stress the point that it leads to code reuse, where I currently need to rewrite things (which increasingly concerns me, especially from a maintenance point of view). Now, on to the licensing. Obviously, the MonitorWare use case above would be totally incompatible with GPLv3. So the rsyslog-runtime would need to be under a different license. It could be dual-licensed, but I think that would probably do more bad than good. I think I can convince Adiscon to go with LGPL for the runtime part. Granted, it introduces risk of closed source competitors pulling it in, but the advantages should outweigh this risk. >From the ability to put this work under a different license, I think I am in good shape: most of the helper objects are freshly written and have only received limited patches (if at all) from contributors. I can contact them and ask for permission to change the license. Where I don't get permission, I think I can re-implement the contribution. Again, most of the code in question has been written in the past 4 month and is 99.999% non-contributed. There may be some few runtime objects which stem back to sysklogd. There, a license change is impractical. I'll have to life with the fact that those can not go into a re-licensed runtime. Depending on how important the functionality is, I either need to rewrite or drop it (for non-rsyslogd use). In any case, this looks (pending detail analysis) quite possible. Big question number one is what you think of this runtime approach? Have I overlooked something? Do you object it for some reason? If so, which? The next question is how to package this inside the source tree. Remember, currently rsyslog and the plugins (considered separate projects) are all packed together inside a single tarball. This is very convenient, both for me as well as for package maintainers and users. The question is if we split rsyslog into the rsyslogd and the rsyslog-runtime, will we continue to deliver the runtime as part of the rsyslog package? Or would it be better to move it to a librsyslog project? Other than with the plugins, we actually would have two different licenses, so it may be confusing to have both of it in the same project (but I have seen that GnuTLS uses exactly this approach, with the main library being LGPL and the - included - extras library GPL only). So that's the next question, obviously depending on the first: how to pack projects if we do a runtime split? I know this is a long and dense mail. My apologies for this. But I think the discussion is needed. I honestly believe that a number of discussions in the past weeks actually circled around this theme, we just didn't actually get down to the point. Please note that I will hold TLS development until we have reached consensus on the runtime/licensing topic. The reason is if we don't do a runtime split, I need to do things considerable different than when we do one (much more code, probably yet another external library). So, obviously, I have a current bias towards the split. However, experience shows that I (as everyone ;)) tend to overlook or misunderstand things. Thus your feedback is so important. I don't like the idea of jiggling back and forth on such an important topic as licensing and high-level modularization, so I would like to get it now done in "the right way" and keep it stable for at least the foreseeable future. Given the fact that the decision somehow affects rsyslog's development as whole, I would even appreciate quick feedback. In this spirit: please let the comments flow ;) Thanks, Rainer From friedl at hq.adiscon.com Fri Apr 11 11:24:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Fri, 11 Apr 2008 11:24:20 +0200 Subject: [rsyslog] rsyslog 3.15.1 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308DD6@grfint2.intern.adiscon.com> Hi all, rsyslog 3.15.1 has been released. This is a refresh of the beta branch, providing new bug fixes. The beta branch currently features the RELP protocol and will be the next v3-stable once it has sufficiently matured. Changelog: http://www.rsyslog.com/Article210.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-97.phtml As always, feedback is appreciated. Florian Riedl From aoz.syn at gmail.com Sat Apr 12 00:10:04 2008 From: aoz.syn at gmail.com (RB) Date: Fri, 11 Apr 2008 16:10:04 -0600 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <1207901052.22738.230.camel@localhost.localdomain> References: <1207901052.22738.230.camel@localhost.localdomain> Message-ID: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Read once, digested for a few hours, read again. *whew* A monolithic message about a monolithic problem... ;) Also please understand I'm new here and am not completely familiar with your codebase, so may well miss the mark. Do I understand you correctly that part of your concern with looser coupling is the higher potential for lossiness? I don't think I can afford a line-by-line analysis at the moment, but got a persistent thought both times through: if you're considering pursuing a loosely-coupled model, why not consider going all the way and treat rsyslogd as a dispatch/routing agent among the various modules? That's by no means a simple solution, but I wonder why certain pieces of functionality must be tightly coupled, either to each other (network/crypt) or to the core itself (network). I *think* it becomes even more interesting when you start thinking about ways to 'stack' modules. Don't feel qualified to even start answering the licensing/packaging stuff, so will leave that be. From rgerhards at hq.adiscon.com Mon Apr 14 08:58:50 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 08:58:50 +0200 Subject: [rsyslog] TLS, loosely coupled modules, a runtime and licensing... In-Reply-To: <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> References: <1207901052.22738.230.camel@localhost.localdomain> <4255c2570804111510l548a4c6fof03eadfee9e2e08d@mail.gmail.com> Message-ID: <1208156330.22738.252.camel@localhost.localdomain> On Fri, 2008-04-11 at 16:10 -0600, RB wrote: > Read once, digested for a few hours, read again. *whew* A monolithic > message about a monolithic problem... ;) Also please understand I'm > new here and am not completely familiar with your codebase, so may > well miss the mark. > > Do I understand you correctly that part of your concern with looser > coupling is the higher potential for lossiness? Yes, there are two issues with that: increased lossiness (due to less knowledge inside rsyslog core of what happens & the same problem plain tcp has) and performance. A pipe interface is much slower than the native in-process plugin interface - this can turn out to become a problem in high performance big enterprise deployments. HOWEVER, all of this can be solved if the module can be compiled to utilize either one of both interfaces. I have also thought more about this and it looks like the absolutely perfect solution to all needs (the bottom line is: the user decides what to use and has all options at hand). > > I don't think I can afford a line-by-line analysis at the moment, but > got a persistent thought both times through: if you're considering > pursuing a loosely-coupled model, why not consider going all the way > and treat rsyslogd as a dispatch/routing agent among the various > modules? Actually, this is the approach rsyslog takes. Though this page is a bit old (and generic), it still covers the base ideas: http://www.rsyslog.com/doc-generic_design.html This document was originally written for an IETF discussion, so it is not rsyslog-specific (but given the same authorship... ;)). > That's by no means a simple solution, but I wonder why > certain pieces of functionality must be tightly coupled, either to > each other (network/crypt) or to the core itself (network). I *think* > it becomes even more interesting when you start thinking about ways to > 'stack' modules. The most recent rsyslog loader supports stacking. We now have a call where an object can request it to find other objects, which can lead to an object load request if the object is not yet in core. Of course, if a new object is loaded, it can request the loader to find other objects it needs... Run a recent build of rsyslogd with -d -n and you'll see these calls in action ;) However, not all of the rsyslog modules are full blown objects as of now. Converting them is part of the ongoing modularization effort. But this also touches the core of my long message: the loader and all helpers currently live in rsyslogd core ONLY. They are not usable by any entity that is not either part of rsyslog or being *called by rsyslog* (plugins). So it is currently impossible to use any of rsyslog's technology without rsyslogd. Among others, this prevents loosely coupling modules (because the module would need access to the runtime objects without rsyslog). It also prevents cases where someone wants to use parts that depend on rsyslog technology, but without using rsyslogd itself. While, of course, I do not find that smart, it may be just the right thing for the user in question. I intend to no longer limit the user in his options, freeing the code base even more. This is both a technical issue as well as a licensing one. The solution, the more I think about it, is really obvious: create a runtime system and place it under LGPL. That'll solve all issues and make rsyslog modular not only on a detail level but also from the high-level perspective. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 12:46:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 12:46:23 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux Message-ID: <1208169983.22738.290.camel@localhost.localdomain> Hi all, I am currently working on the imklog module. I noticed that - for years now - internally generated messages of this module are also emitted with the "kernel" facility. For example, the startup message: imklog 3.17.1-bsd, log source = /proc/kmsg started. rsyslog has taken over that behavior from the sysklogd project, so it is really ages old. HOWEVER, I personally think this is the wrong facility. IMHO the syslogd facility would be the right one, as this message is not from the kernel log but instead locally generated by the syslogd (the syslog subsystem, to be precise). On BSD, no similar problem exists so far, because there no kernel message was added. At the current snapshot, I add few messages if things go wrong. Actually, this triggered the question on how to handle internally generated messages... Question now: shall I change that to LOG_SYSLOG? Or should it remain as is? Final option: a config switch where the user can configure which facility to use (that would then default to LOG_SYSLOG IMO). Feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Mon Apr 14 14:20:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Apr 2008 14:20:52 +0200 Subject: [rsyslog] Facility of imklog-generated messages on linux In-Reply-To: <1208169983.22738.290.camel@localhost.localdomain> References: <1208169983.22738.290.camel@localhost.localdomain> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E00@grfint2.intern.adiscon.com> Hi all, I have thought a bit more about it. I ended up with a configuration variable, but the default is depending on the klog driver. So on Linux, we continue to log those messages as kernel, while on BSD we continue to log them as syslogd. The default can be overwritten by the user. However, I still wonder if it really is desirable to log the messages as kernel on Linux. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Monday, April 14, 2008 12:46 PM > To: rsyslog-users > Subject: [rsyslog] Facility of imklog-generated messages on linux > > Hi all, > > I am currently working on the imklog module. I noticed that - for years > now - internally generated messages of this module are also emitted > with > the "kernel" facility. For example, the startup message: > > imklog 3.17.1-bsd, log source = /proc/kmsg started. > > rsyslog has taken over that behavior from the sysklogd project, so it > is > really ages old. HOWEVER, I personally think this is the wrong > facility. > IMHO the syslogd facility would be the right one, as this message is > not > from the kernel log but instead locally generated by the syslogd (the > syslog subsystem, to be precise). > > On BSD, no similar problem exists so far, because there no kernel > message was added. At the current snapshot, I add few messages if > things > go wrong. Actually, this triggered the question on how to handle > internally generated messages... > > Question now: shall I change that to LOG_SYSLOG? Or should it remain as > is? Final option: a config switch where the user can configure which > facility to use (that would then default to LOG_SYSLOG IMO). > > Feedback is appreciated. > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Tue Apr 15 09:41:56 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 09:41:56 +0200 Subject: [rsyslog] some rsyslog files under LGLP Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E12@grfint2.intern.adiscon.com> Hi all, I will now begin to change the license for some rsyslog files from GPLv3 only to LGPLv3. I have contacted the major contributors and will contact others on an as-needed basis. The license change will probably take some time. Rainer From rgerhards at hq.adiscon.com Tue Apr 15 10:51:45 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 10:51:45 +0200 Subject: [rsyslog] rsyslog runtime library Message-ID: <1208249505.22738.309.camel@localhost.localdomain> Hi all, I have begun to lay the foundation for a well-defined rsyslog runtime library. I had a couple of discussions and the current (still open to change ;)) strategy is to build gradually build the runtime library as part of the main rsyslog project. This resembles the way plugins, also individual projects, are shipped together in the main tarball. What convinced me to work on this route, at least for now, is some technical advantages. Using this approach, I can incrementally work on the runtime library. While it is useful to have, it requires quite some effort to build a clean, abstracted runtime library. Remember the main effort for v3 (which will lead to v4) is to create a fully modular rsyslog subsystem. I started from the quite monolithic v2 source based and have already extracted a great deal of objects. But this is not yet completed - I do this work in parallel to introducing new features and when it best fits. Similarly, splitting off the runtime library right now would mean I stopping feature development for a few weeks, finally cleaning up the object model and putting it all nicely together. While it obviously sound like the right thing from the software engineering point of view, it sounds pretty bad from a feature perspective (bringing everything to a halt plus introducing many new, untested, bug possibilities). So I now believe the right route is again a compromise: I continue to add new features, keeping the runtime idea in mind and contributing to it whenever it is a good time to do so. For v4, of course, we should have a full blow abstraction, because then we finally have the clean object model. So I will spend some limited time on initial creation of a skeleton library, but will continue to work on rsyslog features on a fast pace. I honestly believe this will lead to the overall best result. Feedback is appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Tue Apr 15 15:24:38 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Apr 2008 15:24:38 +0200 Subject: [rsyslog] rsyslog 3.17.1 (devel) released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308E1D@grfint2.intern.adiscon.com> Hi all: Rsyslog 3.17.1 has just been released. The primary new feature is full BSD platform support. Rsyslog now not only compiles cleanly on BSD, but also now supports BSD's kernel log natively. In general, kernel logging has been improved and better documented. High-precision timestamps for the kernel log have been enabled. There are also some minor other enhancements as well as a couple of bug fixes. Rsyslog 3.17.1 is a recommended update for all users of the devel branch. Changelog: http://www.rsyslog.com/Article213.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-98.phtml As always, feedback is appreciated. Rainer From rgerhards at hq.adiscon.com Wed Apr 16 17:27:04 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 16 Apr 2008 17:27:04 +0200 Subject: [rsyslog] Syslog Traffic Data & Alerting Message-ID: <1208359624.22738.353.camel@localhost.localdomain> Hi all, I am currently thinking about a traffic-volume based approach to detect unusual situations and issue alerts. This is in the context of advanced analysis output plugins (which will later also be able to run standalone). I have some ideas, but in order to refine them, I need some real-world syslog traffic data. I am in need of the volume of syslog messages per second, as it evolves over multiple days (preferably over a period of one or two month - and to get started a week worth of this data). I wonder if anyone of you would be interested in providing me such a traffic sample. The data I need would be totally anonymous. All I need is a file with one line per second: timestamp and number of messages per second (no information about message content or any other message property). However, these two fields must be correct and not be any further processed. After all, the root idea is to detect something from the patterns and mangled ones would be very counter-productive. If there is interest (multiple contributors would be happily accepted), I would write an output plugin for rsyslog that gathers this data. It would be very useful to have at least one high-volume environment inside the sample. Please let me know if you could contribute to this effort. Your help would be deeply appreciated. Thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 18 18:38:06 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 18 Apr 2008 18:38:06 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon Message-ID: <1208536686.22738.360.camel@localhost.localdomain> Hi folks, of course, rsyslog can work with any web interface. Nevertheless, we have begun to rewrite our phpLogCon in the hope that it will be an even useful tool. From Andre, the main project author, I have heard that we will probably see an initial v2 release next week. He has also set up a demo server, based on today's codebase: http://demo.phplogcon.org/ I would appreciate if you could drop by and provide us some feedback. It's the first time phpLogCon v2 is visible online and I we are very interested to learn what people like and what not. Of course, the project will expand much in the next few month, and you can help drive its direction. However, I think it is quite useful as it currently is. We have pushed some limited test data into the demo. If someone would like to try it out on his/her own system, please let us know. We will help you get started for sure :) Thanks, Rainer From bakers at web-ster.com Fri Apr 18 18:46:37 2008 From: bakers at web-ster.com (Scott Baker) Date: Fri, 18 Apr 2008 09:46:37 -0700 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808D06D.5000309@web-ster.com> Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer My first response is... WOW! I've never seen that web interface before but that's pretty impressive. I'm much more of a CLI/grep kinda guy but I can see a lot of PHBs loving this. It's very impressive and seems to work quite well. Only complaint is that the date column wraps to two lines, but that's just cosmetic. Kick ass. Keep up the good work. -- Scott Baker - Canby Telcom RHCE - System Administrator - 503.266.8253 From r.bhatia at ipax.at Fri Apr 18 21:18:47 2008 From: r.bhatia at ipax.at (Raoul Bhatia [IPAX]) Date: Fri, 18 Apr 2008 21:18:47 +0200 Subject: [rsyslog] web interface for rsyslog - phpLogCon In-Reply-To: <1208536686.22738.360.camel@localhost.localdomain> References: <1208536686.22738.360.camel@localhost.localdomain> Message-ID: <4808F417.2000805@ipax.at> hi rainer, looks pretty decent. i pulled the git repository and will have a further look ;) as far as i can tell by now, i like modular design where you can fetch data from different streams. cheers, raoul Rainer Gerhards wrote: > Hi folks, > > of course, rsyslog can work with any web interface. Nevertheless, we > have begun to rewrite our phpLogCon in the hope that it will be an even > useful tool. From Andre, the main project author, I have heard that we > will probably see an initial v2 release next week. He has also set up a > demo server, based on today's codebase: > > http://demo.phplogcon.org/ > > I would appreciate if you could drop by and provide us some feedback. > It's the first time phpLogCon v2 is visible online and I we are very > interested to learn what people like and what not. Of course, the > project will expand much in the next few month, and you can help drive > its direction. However, I think it is quite useful as it currently is. > > We have pushed some limited test data into the demo. If someone would > like to try it out on his/her own system, please let us know. We will > help you get started for sure :) > > Thanks, > Rainer > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog -- ____________________________________________________________________ DI (FH) Raoul Bhatia M.Sc. email. r.bhatia at ipax.at Technischer Leiter IPAX - Aloy Bhatia Hava OEG web. http://www.ipax.at Barawitzkagasse 10/2/2/11 email. office at ipax.at 1190 Wien tel. +43 1 3670030 FN 277995t HG Wien fax. +43 1 3670030 15 ____________________________________________________________________ From lanas at securenet.net Wed Apr 23 01:31:00 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 19:31:00 -0400 Subject: [rsyslog] rsyslog in general Message-ID: <20080422193100.2b2a09a1@mistral.stie> Hello, I'd be interested to use rsyslog in a production environment. On one side I have the impression that it is a stable product but then if I look at the releases, they are quite numerous over a small amount of time. I would like to know what is the average count of bug fixes in general and to which severity. Thanks, Al From mbiebl at gmail.com Wed Apr 23 02:37:15 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 02:37:15 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422193100.2b2a09a1@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: 2008/4/23, lanas : > Hello, > > I'd be interested to use rsyslog in a production environment. On one > side I have the impression that it is a stable product but then if I > look at the releases, they are quite numerous over a small amount of > time. I would like to know what is the average count of bug fixes in > general and to which severity. There are frequent releases of devel "snapshots". For stable releases you should check out http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml (not sure if you are already aware of that) Or do you also have concerns about the frequency of stable releases? Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From lanas at securenet.net Wed Apr 23 02:54:47 2008 From: lanas at securenet.net (lanas) Date: Tue, 22 Apr 2008 20:54:47 -0400 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie> Message-ID: <20080422205447.0772e890@mistral.stie> Le Mercredi, 23 Avril 2008 02:37:15 +0200, "Michael Biebl" a ?crit : >> I'd be interested to use rsyslog in a production environment. On >> one side I have the impression that it is a stable product but then >> if I look at the releases, they are quite numerous over a small >> amount of time. I would like to know what is the average count of >> bug fixes in general and to which severity. > There are frequent releases of devel "snapshots". > For stable releases you should check out > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > (not sure if you are already aware of that) Thanks. The package I ended up with was version 3.17.1. What about this version ? It is not tagged as beta, although it is development. What is the schedule for a stable release of that version ? I see that the page you gave lists 3.14 as stable. > Or do you also have concerns about the frequency of stable releases? Well, to have a grasp at how the development is planned and one would be great. Cheers, Al From mbiebl at gmail.com Wed Apr 23 03:16:27 2008 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 23 Apr 2008 03:16:27 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: <20080422205447.0772e890@mistral.stie> References: <20080422193100.2b2a09a1@mistral.stie> <20080422205447.0772e890@mistral.stie> Message-ID: 2008/4/23, lanas : > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > "Michael Biebl" a ?crit : > > > >> I'd be interested to use rsyslog in a production environment. On > >> one side I have the impression that it is a stable product but then > >> if I look at the releases, they are quite numerous over a small > >> amount of time. I would like to know what is the average count of > >> bug fixes in general and to which severity. > > > There are frequent releases of devel "snapshots". > > > For stable releases you should check out > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > (not sure if you are already aware of that) > > > Thanks. The package I ended up with was version 3.17.1. What about > this version ? It is not tagged as beta, although it is development. > What is the schedule for a stable release of that version ? I see that > the page you gave lists 3.14 as stable. 3.17.1 is from the devel branch, so I wouldn't recommend it for production use. The version numbering has been in discussion not long ago, so take the following with a grain of salt (Rainer, the lead author, will probably have additional information). 2.0.x is the old-stable branch (the current version being 2.0.4). If you want something ultra-stable, without the need for some of the nice feature of the v3 branch, take this one. The first stable release of the v3 branch was 3.14.1 (the current version being 3.14.2). If you need some of the features of the v3 branch, go with 3.14.2. Stable versions are even numbered (second digit). See also http://www.rsyslog.com/doc-status.html The devel branch is usually for bleeding edge stuff and has a odd numbered second digit. The beta branch is for stabilizing features, resulting in the next stable release. I also has a odd numbered second digit. > > > > Or do you also have concerns about the frequency of stable releases? > > > Well, to have a grasp at how the development is planned and one would > be great. > http://www.rsyslog.com/doc-status.html http://www.rsyslog.com/doc-features.html http://www.rsyslog.com/doc-v3compatibility.html http://www.rsyslog.com/doc HTH, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rgerhards at hq.adiscon.com Wed Apr 23 08:20:58 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 23 Apr 2008 08:20:58 +0200 Subject: [rsyslog] rsyslog in general In-Reply-To: References: <20080422193100.2b2a09a1@mistral.stie><20080422205447.0772e890@mistral.stie> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EB0@grfint2.intern.adiscon.com> Hi there, Michael has done an excellent job of summing up things :) Just let me add a few thoughts. First of all, the versioning document needs to be updated and I'll do that soon ;) The frequent releases are a result of rapid development. There is lots to do and we do it quickly. This, of course, introduces bugs. To develop rapidly and still provide stable releases, we have settled on this for now: The v2-stable is a feature-dead end. It receives patches only. Ultra-stable, quite featureless. V3-stable primarily receives patches but from time to time new functionality is integrated. This functionality ripens in the beta branch. The devel version has the newest code and most bugs. It is NOT recommended for production use. At some point in time, beta is branched off devel. Beta than receives bug fixes, but no new features. After some time (between 3 and 10 weeks, depending on bug reports), beta becomes v3-stable. Even inside stable or beta or devel there are several levels of maturity - some code is unmodified for more than a year, while other just recently made it into stable. I have recently begun to add some information about when features have been introduced into a version. The general rule is that the newer a feature is, the more likely it contains bugs. In general, rsyslog is rock-solid. However, some so far seldomly-used features (expression based filters are a good example) has seen limited exposure to practice. Such features are more likely to have problems. If you try judge project and feature stability, it may be a good idea to look at the bugzilla and the code commits: http://bugzilla.adiscon.com/buglist.cgi?cmdtype=runnamed&namedcmd=rsyslog-all http://git.adiscon.com/?p=rsyslog.git;a=summary You may also want to have a look at my blog, where I post anything of note. It's also a good place to see what is going on and actually even to see what is giving me a hard time every now and then: http://rgerhards.blogspot.com Finally, if you let me know what you intend to do, I can provide you my personal view of how stable the required features are. But as I said - it's pretty stable now. Rsyslog also recently made it into Red Hat Enterprise Linux (5.2) and you probably know that these folks are quite conservative when it comes to stability ;) HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Biebl > Sent: Wednesday, April 23, 2008 3:16 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog in general > > 2008/4/23, lanas : > > Le Mercredi, 23 Avril 2008 02:37:15 +0200, > > "Michael Biebl" a ?crit : > > > > > > >> I'd be interested to use rsyslog in a production environment. > On > > >> one side I have the impression that it is a stable product but > then > > >> if I look at the releases, they are quite numerous over a small > > >> amount of time. I would like to know what is the average count > of > > >> bug fixes in general and to which severity. > > > > > There are frequent releases of devel "snapshots". > > > > > For stable releases you should check out > > > http://www.rsyslog.com/Downloads-req-viewsdownload-sid-1.phtml > > > (not sure if you are already aware of that) > > > > > > Thanks. The package I ended up with was version 3.17.1. What about > > this version ? It is not tagged as beta, although it is > development. > > What is the schedule for a stable release of that version ? I see > that > > the page you gave lists 3.14 as stable. > > 3.17.1 is from the devel branch, so I wouldn't recommend it for > production use. > The version numbering has been in discussion not long ago, so take the > following with a grain of salt (Rainer, the lead author, will probably > have additional information). > > 2.0.x is the old-stable branch (the current version being 2.0.4). If > you want something ultra-stable, without the need for some of the nice > feature of the v3 branch, take this one. > > The first stable release of the v3 branch was 3.14.1 (the current > version being 3.14.2). If you need some of the features of the v3 > branch, go with 3.14.2. > > Stable versions are even numbered (second digit). > > See also > http://www.rsyslog.com/doc-status.html > > The devel branch is usually for bleeding edge stuff and has a odd > numbered second digit. > The beta branch is for stabilizing features, resulting in the next > stable release. I also has a odd numbered second digit. > > > > > > > > Or do you also have concerns about the frequency of stable > releases? > > > > > > Well, to have a grasp at how the development is planned and one would > > be great. > > > > http://www.rsyslog.com/doc-status.html > http://www.rsyslog.com/doc-features.html > http://www.rsyslog.com/doc-v3compatibility.html > http://www.rsyslog.com/doc > > HTH, > Michael > -- > Why is it that all of the instruments seeking intelligent life in the > universe are pointed away from Earth? > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From friedl at hq.adiscon.com Thu Apr 24 14:36:20 2008 From: friedl at hq.adiscon.com (Florian Riedl) Date: Thu, 24 Apr 2008 14:36:20 +0200 Subject: [rsyslog] rsyslog 3.16.0 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Hi all, I have just released rsyslog 3.16.0, a v3-stable release. This release brings a major new feature, RELP support, to the stable branch. RELP enables lossless transmission of syslog messages, in cases where even plain tcp syslog fails. Other than that, the release contains a few bugfixes. Users of the v3-stable branch are advised to update to the new release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-99.phtml Changelog: http://www.rsyslog.com/Article215.phtml Florian Riedl From rgerhards at hq.adiscon.com Fri Apr 25 07:49:53 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 07:49:53 +0200 Subject: [rsyslog] phpLogCon v2 released Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308ED7@grfint2.intern.adiscon.com> Hi all, I'd like to pass the info that phpLogCon v2's initial release is now available. I thought a while if I should use this list to announce it, but came to the conclusion that it is justified for now. I will refrain from doing frequent phpLogCon announcements here. We are currently setting up the infrastructure (mailing list et al) for phpLogCon. I'll do one more announcement here when this is completed. In the meantime, I suggest subscribing to freshmeat's announcements, which we maintain in a timely way. You can do so at: http://freshmeat.net/projects/phplogcon/ We have just released version 2.1.0 which is the first official beta version of the v2 branch. PhpLogCon has been completely rewritten from scratch. It now offers a state-of-the art modern user interface and also is able to work with log files and not just databases. For example, it can be used to view a remote server's log files over the web (proper authentication settings highly recommended). It will evolve to a very capable search, reporting and analysis frontend for syslog data. Let me stress the point that it can work with log files directly. For example, we have set it up on one of our mail relays so that we can review mail logs without the need to login onto that machine. Obviously, this functionality should only be available to authenticated users, but then it is quite useful. I would appreciate to learn about any more thought of how this tool can be put to good use. Download: http://www.phplogcon.org/Downloads-req-viewdownloaddetails-lid-7.phtml Many thanks, Rainer From rgerhards at hq.adiscon.com Fri Apr 25 18:18:23 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:18:23 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308EEA@grfint2.intern.adiscon.com> Folks, mailman does currently not accept this message, so I forward it to the list. A reply follows. Should someone else also have problems sending list mail, please let me know. Rainer > -----Original Message----- > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Friday, April 25, 2008 6:05 PM > To: rsyslog at lists.adiscon.com > Cc: Rainer Gerhards > Subject: FW: RSYSLOG "Best Practices" & General Questions > > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all data > from the remote sites. I am configuring it to store all SYSLOG data > with in the database, but I have followed the recommendations made to > "buffer" it to a spool first. My question is this, I do not want to > just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format on > the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this... It looked like it should able to be done, > but I am just not sure the best way to accomplish this. > > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site... This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to just > send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, and > let the server process and insert like it would local logs? > > 3) Within the scenario listed in question 2, how can I, 1) preserve > all the original IP addresses of the machines that are transmitting > information, and 2) tag that information with a specific account code > identifying the site that the information was sent from. Within the > database, I have created a column called "customerid" that I would like > to do this with. In this, I would like to designate a name or integer > like "1" for site A, "2" for site B, etc. The reason for this is that > I will run into situations where multiple customers will have the same > IP addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this...? > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based machines. > I would like to use the same table that is used for storage of the rest > of SYSLOG data, and I have the associated columns already built. I > just want to make sure that what I setup now will be completely > compatible and able to process NT Event Log information. > > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > From rgerhards at hq.adiscon.com Fri Apr 25 18:48:43 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Apr 2008 18:48:43 +0200 Subject: [rsyslog] FW: RSYSLOG "Best Practices" & General Questions In-Reply-To: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> References: <15B9428F923A4D3BAB3D4BCBDEBEA52B@devel.skylinesmms.com> Message-ID: <1209142123.22738.461.camel@localhost.localdomain> On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > ?central? server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to ?buffer? it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in ?standard? log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this? It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. > 2) At each customer site, there will be a server called a > ?collector? that will accept all SYSLOG related information for that > site? This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, ?I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.html rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called ?customerid? that I would like to do this > with. In this, I would like to designate a name or integer like ?1? > for site A, ?2? for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site?s > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more ?elegant? method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this?? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer > > > > > Stephen Malenshek > > Manager, Managed Services Group > > Skyline Advanced Technology Services > > Bozeman, Montana > > smalenshek at skyline-ats.com > > > > Phone: (406) 587-1047 x106 > > Cell: (406) 599-9569 > > Fax: (406) 587-1035 > > > > > > > > From mic at npgx.com.au Sat Apr 26 03:11:24 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sat, 26 Apr 2008 12:11:24 +1100 Subject: [rsyslog] rsyslog 3.16.0 released In-Reply-To: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA308ECA@grfint2.intern.adiscon.com> Message-ID: <20080426011021.M97077@npgx.com.au> Hi, > Hi all, > > I have just released rsyslog 3.16.0, a v3-stable release. This > release brings a major new feature, RELP support, to the stable > branch. RELP enables lossless transmission of syslog messages, in > cases where even plain tcp syslog fails. Other than that, the > release contains a few bugfixes. > > Users of the v3-stable branch are advised to update to the new release. Where are the RPM spec files for these releases so I can build the RPM's? Thanks. Michael. From mic at npgx.com.au Sun Apr 27 13:56:13 2008 From: mic at npgx.com.au (Michael Mansour) Date: Sun, 27 Apr 2008 22:56:13 +1100 Subject: [rsyslog] rsyslog 3.16.0 on el4 Message-ID: <20080427115258.M2863@npgx.com.au> Hi, I just created a spec file and compiled an RPM for this version. In the source is the ./redhat/rsyslog.init file. This file references klogd through, so when I ended up installing the RPM, I couldn't start rsyslog using the script as it would exit. If klogd is no longer used in this rsyslog version, could you modify/correct the rsyslog.init file so that it no longer makes reference to it? I've already commented out the relevant entries in the file so I could start up rsyslogd. Thanks. Michael. From mic at npgx.com.au Sun Apr 27 18:12:26 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 03:12:26 +1100 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <20080427160908.M29132@npgx.com.au> Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. From rgerhards at hq.adiscon.com Sun Apr 27 18:31:48 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 27 Apr 2008 18:31:48 +0200 Subject: [rsyslog] Date format on rsyslog 3.16.0 Message-ID: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> I am on a cell phone... Check out the v3 compatibility document. Its available right from the doc main page. rainer ----- Urspr?ngliche Nachricht ----- Von: "Michael Mansour" An: "rsyslog-users" Gesendet: 27.04.08 18:13 Betreff: [rsyslog] Date format on rsyslog 3.16.0 Hi, The date format logged to the messages/maillog/etc has changed from: Apr 27 21:36:58 to: 2008-04-27T21:44:35.788251+10:00 I'm new to v3 so how can I change this format back to the original? Thanks. Michael. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog From mic at npgx.com.au Mon Apr 28 01:15:49 2008 From: mic at npgx.com.au (Michael Mansour) Date: Mon, 28 Apr 2008 10:15:49 +1100 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> Message-ID: <20080427231354.M42594@npgx.com.au> Hi Rainer, > I am on a cell phone... > > Check out the v3 compatibility document. Its available right from > the doc main page. Yes, from that document I found the option: $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat Which made it go back to the older time format. While reading it more though, I think I will actually keep it on the new format as the precision time isn't actually too bad. It may break some of my scripts which analyse daily log files and if it does, I'll enable the traditional format again, but for now I'll wait and see if anything breaks. Thanks. Michael. > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Michael Mansour" > An: "rsyslog-users" > Gesendet: 27.04.08 18:13 > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > Hi, > > The date format logged to the messages/maillog/etc has changed from: > > Apr 27 21:36:58 > > to: > > 2008-04-27T21:44:35.788251+10:00 > > I'm new to v3 so how can I change this format back to the original? > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog ------- End of Original Message ------- From rgerhards at hq.adiscon.com Mon Apr 28 08:01:40 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 08:01:40 +0200 Subject: [rsyslog] [Spam?BadBits] Re: Date format on rsyslog 3.16.0 In-Reply-To: <20080427231354.M42594@npgx.com.au> References: <001701c8a884$39b9f6e1$060013ac@intern.adiscon.com> <20080427231354.M42594@npgx.com.au> Message-ID: <1209362500.22738.466.camel@localhost.localdomain> Hi Michael, On Mon, 2008-04-28 at 10:15 +1100, Michael Mansour wrote: > Hi Rainer, > > > I am on a cell phone... > > > > Check out the v3 compatibility document. Its available right from > > the doc main page. > > Yes, from that document I found the option: > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > Which made it go back to the older time format. While reading it more though, > I think I will actually keep it on the new format as the precision time isn't > actually too bad. It may break some of my scripts which analyse daily log > files and if it does, I'll enable the traditional format again, but for now > I'll wait and see if anything breaks. > Yes, scripts is the big Problem. Rsyslog actually had high-precision timestamps right from the beginning, but nobody ever used them. So I decided to change the default ;) On the scripts: I am thinking about a simple converter that takes high precision log files and converts them to old-style low precision ones. Such a converter could be run before the actual script. Do you (or someone else) think this is useful and worth the effort? Rainer > Thanks. > > Michael. > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Michael Mansour" > > An: "rsyslog-users" > > Gesendet: 27.04.08 18:13 > > Betreff: [rsyslog] Date format on rsyslog 3.16.0 > > > > Hi, > > > > The date format logged to the messages/maillog/etc has changed from: > > > > Apr 27 21:36:58 > > > > to: > > > > 2008-04-27T21:44:35.788251+10:00 > > > > I'm new to v3 so how can I change this format back to the original? > > > > Thanks. > > > > Michael. > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > ------- End of Original Message ------- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From rgerhards at hq.adiscon.com Mon Apr 28 14:34:52 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 14:34:52 +0200 Subject: [rsyslog] rsyslog 3.16.0 on el4 In-Reply-To: <20080427115258.M2863@npgx.com.au> References: <20080427115258.M2863@npgx.com.au> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F01@grfint2.intern.adiscon.com> Michael, this was contributed code. We even thought about dropping these files altogether. But as it looks, they seem to be of some value. If you have a corrected version, I'd appreciate if you could send me these, so that I can include them into the tarball- I have very limited platform know-how, so I am depending on other folks to send me startup scripts and such... Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Michael Mansour > Sent: Sunday, April 27, 2008 1:56 PM > To: rsyslog-users > Subject: [rsyslog] rsyslog 3.16.0 on el4 > > Hi, > > I just created a spec file and compiled an RPM for this version. > > In the source is the ./redhat/rsyslog.init file. This file references > klogd > through, so when I ended up installing the RPM, I couldn't start > rsyslog > using the script as it would exit. > > If klogd is no longer used in this rsyslog version, could you > modify/correct > the rsyslog.init file so that it no longer makes reference to it? > > I've already commented out the relevant entries in the file so I could > start > up rsyslogd. > > Thanks. > > Michael. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog From smalenshek at skyline-ats.com Mon Apr 28 17:01:36 2008 From: smalenshek at skyline-ats.com (Stephen Malenshek) Date: Mon, 28 Apr 2008 09:01:36 -0600 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions Message-ID: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] Sent: Monday, April 28, 2008 8:55 AM To: rsyslog at lists.adiscon.com; Rainer Gerhards Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > I am currently setting up creating a managed service platform from > various open source products out on the market and I would like to use > your product as the standard SYSLOG replacement on all our sites. I > have a couple of questions related to this and would like you provide > some input on the best ways to achieve specific objectives. > > > > 1) At the present time, I have started the configuration on the > "central" server, which will act as the central repository for all > data from the remote sites. I am configuring it to store all SYSLOG > data with in the database, but I have followed the recommendations > made to "buffer" it to a spool first. My question is this, I do not > want to just write the information to the database, for governmental > compliance, I need to keep a duplicate copy in "standard" log format > on the drive, which I will rotate and gzip daily, for long term log > retention. I have looked around and did not find anything that > specifically addresses this. It looked like it should able to be > done, but I am just not sure the best way to accomplish this. > Storing to the database is in no way special. So you can just add as many other selector lines as you like. For example: *.* :ommysql*... *.* /var/log/soxstore works perfectly. What you need to be aware is that the buffer parameters ($Action...) work on the NEXT action, so you want to have them directly in front of the action in question. In my present configuration, I am doing an if/then like the following: if $source == 'somemachine' \ and $syslogfacility-text == 'authpriv' \ then ?LocalSecure $template LocalSecure,"/var/log/secure" With this, how can I define multiple directions within this template? > 2) At each customer site, there will be a server called a > "collector" that will accept all SYSLOG related information for that > site. This server will store a copy of the log files for the local > network as a repository, but it also needs to send it to the central > server for processing. My question is whether it will be more > efficient to write the information directly to the database, or to > just send it using normal SYSLOG directives, 'I.E. *.* @{IP Address}, > and let the server process and insert like it would local logs? > that's a very good question. I'd say it depends on a lot of factors, probablky most importantly from the clients ability to talk to the database directly (this is often impossible due to firewalls). Also, you may consider where the plain text files need to be written. If they should be on a central system, I'd opt for moving everything to the central server and writing it to database and/or text file there. Please note that UDP (@IP) is a really bad protocol choice. You will lose a lot of messages and it is definitely not "compliance-compliant" ;). TCP based syslog is much better but not ideal, see: http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp-syslog.h tml rsyslog's solution is the relp protocol, which prevents message loss. > 3) Within the scenario listed in question 2, how can I, 1) > preserve all the original IP addresses of the machines that are > transmitting information, If you use rsyslog on the clients, too, this is automatically done for you. At each site, there will be a single server tat is aggregating all SYSLOG data before transmitting it to the central repository. Are you talking about the machine sending the aggregate information or the hosts, I.E. router, server, switch, etc., that we will be monitoring? > and 2) tag that information with a specific account code identifying > the site that the information was sent from. Within the database, I > have created a column called "customerid" that I would like to do this > with. In this, I would like to designate a name or integer like "1" > for site A, "2" for site B, etc. The reason for this is that I will > run into situations where multiple customers will have the same IP > addressing scheme. I figure this could be passed from the site's > collector as a site identifier, but I am not sure how to accomplish > this. I think I can accomplish this on the central server, if I have > to, with a subquery within the insert query to another table to lookup > this value, but I am looking for a much more "elegant" method. You should look into template definitions. On the client, create a template that contains the customer ID somewhere AFTER the tag. Maybe immediately behind it with the rest of the message being separated by a comma. Then, on the server, look into the property replace. Use field-based extraction to pull out that value and insert it. The property replace is part of the template system. It's quite powerful, but you need to play a bit with it. I need a little more direction on this. I believe I understand, but I want to make absolutely sure. Could you provide me an example of what would be within the remote server's config and what the central server config would look like? > > 4) During the processing of this information, whether it is the > logs or the database inserts, we need to be able to parse this > information, attempt to match using defined regular expressions and > generate an email with the information matched. I saw an example of > this somewhere, but after looking some, not a lot, I just have not > found it again. Would you provide me with a few examples of efficient > ways to accomplish this.? I have none at hand, but the http://wiki.rsyslog.com probably has some. If not, post one or two actual cases and I'll create a few samples of property replacer statenets. You friend is this doc page: http://www.rsyslog.com/doc-property_replacer.html I want to look for things like the following: Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, terminal=ssh res=success)' With this, I want to be notified with the host information, address, etc. via e-mail if someone logs into a particular host as "root". > > 5) Lastly, I am going to strongly recommend to all our clients use > the products related to SYSLOG from Adiscon for us to be able to > process information within this environment for Windows based > machines. I would like to use the same table that is used for storage > of the rest of SYSLOG data, and I have the associated columns already > built. I just want to make sure that what I setup now will be > completely compatible and able to process NT Event Log information. > The schema that comes with rsyslog is the "MonitorWare schema", which is the same for the commercial softwares too. I also suggest to have a look at the recently announced http://www.phplogcon.org. As part of phpLogCon, we will define a special Windows Event Format (based on our existing techology) that will be understood by phpLogCon. I will make sure it is documented, so that you can also subparse it. phpLogCon (also GPLv3) will contain the parser in php, so you should be quite easy be able to adopt it. I have taken a good look at it and will integrate it within our administration interface. It looks like an excellent product and will save me a lot of code to write. > > > Once again, thanks for your time and look forward to hearing your > thoughts related to this implementation. I have used SYSLOG-NG for > years and found that it was great in some respects, but disappointing > being able to do storage to MySQL. You have a great product here. > thanks, appreciated. Our long-term vision specifically includes compliance, so I would be most interested in any requirement you have. For example, I am currently implementing TLS, first for plain tcp syslog, then for RELP. Digital message signatures are also on my list. So any ideas/actual requirements are most welcome and I will happily work together with you to get things done. My vision is *much* broader than syslog-ng (at least as far as I know that project). Rainer From rgerhards at hq.adiscon.com Mon Apr 28 22:28:26 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 28 Apr 2008 22:28:26 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F0F@grfint2.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General > Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would > like to use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like > you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have > them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? I do not fully get you - do you mean how to write to different files based on some message property? If so, you can use the property replacer to do so: $template LocalSecure,"/var/log/%HOSTNAME%/secure" > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* > @{IP Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to > firewalls). Also, you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plai n-tcp-syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is > aggregating all SYSLOG > data before transmitting it to the central repository. Are > you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? Ah, ok. The host is taken from the message, and as such depends on the system that formatted it. You can define a template to consistently define the hostname on the relay. However, that requires that no further relay is between the aggregator and the original senders. > > > > > and 2) tag that information with a specific account code > identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like > to do this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, > if I have > > > to, with a subquery within the insert query to another > table to lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being > separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property > replace is > > part of the template system. It's quite powerful, but you > need to play a > > bit with it. > > > > I need a little more direction on this. I believe I > understand, but I want > to make absolutely sure. Could you provide me an example of > what would be > within the remote server's config and what the central server > config would > look like? > I can't do a lab right now (sorry, busy), but you can try along these lines. On the server's config, use a template along those lines: $template tpl,"%TIMESTAMP% %HOSTNAME% %syslogtag%,%msg:::drop-last-lf%\n" Where sitename is to be replaced by a site name (e.g. 1, SiteA ...). On the central server, you can extract that field inside a template by such a property specification: %msg:F,44:1% Again, all of this is untested but should either work or is pretty close to working ;) > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples > of efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably > has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, > address, etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our > clients use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used > for storage > > > of the rest of SYSLOG data, and I have the associated > columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare > schema", which is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will > define a special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, > so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the > parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product > and will save > me a lot of code to write. If you have any feedback, comments, suggestions or feature request, please let us know. It is very much in its infancy and Andre, the lead developer, is working very hard on it. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but > disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > From rgerhards at hq.adiscon.com Tue Apr 29 16:27:33 2008 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Apr 2008 16:27:33 +0200 Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions In-Reply-To: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> References: <80BA91657C124D52A596FEF2D374240D@devel.skylinesmms.com> Message-ID: <577465F99B41C842AAFBE9ED71E70ABA308F1D@grfint2.intern.adiscon.com> I created a small wiki entry that has details on how to use a site id inside the message. I hope it is useful: http://wiki.rsyslog.com/index.php/Splitting_messages_based_on_a_site_ID Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Stephen Malenshek > Sent: Monday, April 28, 2008 5:02 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] FW: RE: RSYSLOG "Best Practices" & General Questions > > From: Stephen Malenshek [mailto:smalenshek at skyline-ats.com] > Sent: Monday, April 28, 2008 8:55 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: [rsyslog] RE: RSYSLOG "Best Practices" & General Questions > > > > On Fri, 2008-04-25 at 10:05 -0600, Stephen Malenshek wrote: > > > I am currently setting up creating a managed service platform from > > > various open source products out on the market and I would like to > use > > > your product as the standard SYSLOG replacement on all our sites. I > > > have a couple of questions related to this and would like you provide > > > some input on the best ways to achieve specific objectives. > > > > > > > > > > > > 1) At the present time, I have started the configuration on the > > > "central" server, which will act as the central repository for all > > > data from the remote sites. I am configuring it to store all SYSLOG > > > data with in the database, but I have followed the recommendations > > > made to "buffer" it to a spool first. My question is this, I do not > > > want to just write the information to the database, for governmental > > > compliance, I need to keep a duplicate copy in "standard" log format > > > on the drive, which I will rotate and gzip daily, for long term log > > > retention. I have looked around and did not find anything that > > > specifically addresses this. It looked like it should able to be > > > done, but I am just not sure the best way to accomplish this. > > > > > Storing to the database is in no way special. So you can just add as > > many other selector lines as you like. For example: > > > > *.* :ommysql*... > > *.* /var/log/soxstore > > > > works perfectly. What you need to be aware is that the buffer > parameters > > ($Action...) work on the NEXT action, so you want to have them directly > > in front of the action in question. > > > > In my present configuration, I am doing an if/then like the following: > > > > if $source == 'somemachine' \ > > and $syslogfacility-text == 'authpriv' \ > > then ?LocalSecure > > > > $template LocalSecure,"/var/log/secure" > > > > With this, how can I define multiple directions within this template? > > > > > 2) At each customer site, there will be a server called a > > > "collector" that will accept all SYSLOG related information for that > > > site. This server will store a copy of the log files for the local > > > network as a repository, but it also needs to send it to the central > > > server for processing. My question is whether it will be more > > > efficient to write the information directly to the database, or to > > > just send it using normal SYSLOG directives, 'I.E. *.* @{IP > Address}, > > > and let the server process and insert like it would local logs? > > > > > that's a very good question. I'd say it depends on a lot of factors, > > probablky most importantly from the clients ability to talk to the > > database directly (this is often impossible due to firewalls). Also, > you > > may consider where the plain text files need to be written. If they > > should be on a central system, I'd opt for moving everything to the > > central server and writing it to database and/or text file there. > > > > Please note that UDP (@IP) is a really bad protocol choice. You will > > lose a lot of messages and it is definitely not > > "compliance-compliant" ;). TCP based syslog is much better but not > > ideal, see: > > > > http://rgerhards.blogspot.com/2008/04/on-unreliability-of-plain-tcp- > syslog.h > tml > > > > rsyslog's solution is the relp protocol, which prevents message loss. > > > > > 3) Within the scenario listed in question 2, how can I, 1) > > > preserve all the original IP addresses of the machines that are > > > transmitting information, > > > > If you use rsyslog on the clients, too, this is automatically done for > > you. > > > > At each site, there will be a single server tat is aggregating all > SYSLOG > data before transmitting it to the central repository. Are you talking > about the machine sending the aggregate information or the hosts, I.E. > router, server, switch, etc., that we will be monitoring? > > > > > and 2) tag that information with a specific account code identifying > > > the site that the information was sent from. Within the database, I > > > have created a column called "customerid" that I would like to do > this > > > with. In this, I would like to designate a name or integer like "1" > > > for site A, "2" for site B, etc. The reason for this is that I will > > > run into situations where multiple customers will have the same IP > > > addressing scheme. I figure this could be passed from the site's > > > collector as a site identifier, but I am not sure how to accomplish > > > this. I think I can accomplish this on the central server, if I have > > > to, with a subquery within the insert query to another table to > lookup > > > this value, but I am looking for a much more "elegant" method. > > > > You should look into template definitions. On the client, create a > > template that contains the customer ID somewhere AFTER the tag. Maybe > > immediately behind it with the rest of the message being separated by a > > comma. > > > > Then, on the server, look into the property replace. Use field-based > > extraction to pull out that value and insert it. The property replace > is > > part of the template system. It's quite powerful, but you need to play > a > > bit with it. > > > > I need a little more direction on this. I believe I understand, but I > want > to make absolutely sure. Could you provide me an example of what would > be > within the remote server's config and what the central server config > would > look like? > > > > > > > > 4) During the processing of this information, whether it is the > > > logs or the database inserts, we need to be able to parse this > > > information, attempt to match using defined regular expressions and > > > generate an email with the information matched. I saw an example of > > > this somewhere, but after looking some, not a lot, I just have not > > > found it again. Would you provide me with a few examples of > efficient > > > ways to accomplish this.? > > > > I have none at hand, but the http://wiki.rsyslog.com probably has some. > > If not, post one or two actual cases and I'll create a few samples of > > property replacer statenets. You friend is this doc page: > > > > http://www.rsyslog.com/doc-property_replacer.html > > > > I want to look for things like the following: > > > > Apr 28 07:31:28 centralca01 kernel: audit(1209393088.758:15174): user > pid=9509 uid=0 auid=0 msg='PAM: authentication acct="root" : > exe="/usr/sbin/sshd" (hostname=XXXXXXXXXXX, addr=XXX.XXX.XXX.XXX, > terminal=ssh res=success)' > > > > With this, I want to be notified with the host information, address, > etc. > via e-mail if someone logs into a particular host as "root". > > > > > > > > 5) Lastly, I am going to strongly recommend to all our clients > use > > > the products related to SYSLOG from Adiscon for us to be able to > > > process information within this environment for Windows based > > > machines. I would like to use the same table that is used for > storage > > > of the rest of SYSLOG data, and I have the associated columns already > > > built. I just want to make sure that what I setup now will be > > > completely compatible and able to process NT Event Log information. > > > > > The schema that comes with rsyslog is the "MonitorWare schema", which > is > > the same for the commercial softwares too. > > > > I also suggest to have a look at the recently announced > > http://www.phplogcon.org. As part of phpLogCon, we will define a > special > > Windows Event Format (based on our existing techology) that will be > > understood by phpLogCon. I will make sure it is documented, so that you > > can also subparse it. phpLogCon (also GPLv3) will contain the parser in > > php, so you should be quite easy be able to adopt it. > > > > I have taken a good look at it and will integrate it within our > administration interface. It looks like an excellent product and will > save > me a lot of code to write. > > > > > > > > > > > Once again, thanks for your time and look forward to hearing your > > > thoughts related to this implementation. I have used SYSLOG-NG for > > > years and found that it was great in some respects, but disappointing > > > being able to do storage to MySQL. You have a great product here. > > > > > thanks, appreciated. Our long-term vision specifically includes > > compliance, so I would be most interested in any requirement you have. > > For example, I am currently implementing TLS, first for plain tcp > > syslog, then for RELP. Digital message signatures are also on my list. > > So any ideas/actual requirements are most welcome and I will happily > > work together with you to get things done. My vision is *much* broader > > than syslog-ng (at least as far as I know that project). > > > > Rainer > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog