From a.piesk at gmx.net Tue Nov 1 10:27:58 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Tue, 01 Nov 2011 10:27:58 +0100 Subject: [rsyslog] rsyslog hangs at startup In-Reply-To: <4EAE9DF0.8080703@gmx.net> References: <4EADBB77.5040801@gmx.net> <4EAE9DF0.8080703@gmx.net> Message-ID: <4EAFBB9E.60805@gmx.net> On 31.10.2011 14:09, Andreas Piesk wrote: > > after looking at runtime/modules.c i suspected that the mutex doesn't get unlocked. > > after inserting some more debug printf: > > 2426.965520000:2ad3d4b29320: cfline: '$ModLoad imfile' > 2426.965545000:2ad3d4b29320: Requested to load module 'imfile' > 2426.965548000:2ad3d4b29320: setting module load/unlock lock > 2426.965551000:2ad3d4b29320: module load/unload lock set > 2426.965555000:2ad3d4b29320: loading module '/lib64/rsyslog/imfile.so' > 2426.965591000:419d5940: strm 0x2ad3e9475220: file 8 read 537 bytes > 2426.965733000:419d5940: MsgSetTAG in: len 9, pszBuf: rsyslogd: > 2426.965740000:419d5940: MsgSetTAG exit: pMsg->iLenTAG 9, pMsg->TAG.szBuf: rsyslogd: > 2426.965776000:419d5940: XXXXX: tryDoAction 0x2ad3e94705a0, pnElem 1, nElem 1 > 2426.965780000:419d5940: Action 0x2ad3e94705a0 transitioned to state: itx > 2426.965783000:419d5940: entering actionCalldoAction(), state: itx > 2426.965797000:419d5940: 10.10.0.254 > 2426.965818000:419d5940: caller requested object 'nsd_ptcp', not found (iRet -3003) > 2426.965843000:419d5940: Requested to load module 'lmnsd_ptcp' > 2426.965846000:419d5940: setting module load/unlock lock > > i don't see "imfile: version %s initializing\n", so the problem seems to be imfile because it > doesn't get initialized and blocks the mutex. > i've debugged the issue a little bit further and it turned out that imfile:modInit() hangs at UseObj(), or to be precise at pthread_mutex_lock(&mutObjGlobalOp). i think it's a deadlock between mutObjGlobalOp and mutLoadUnload: - thread 2ad3d4b29320 calls modules:Load() which acquires lock mutLoadUnload - thread 419d5940 calls obj:UseObj() which aquires lock mutObjGlobalOp - thread 2ad3d4b29320 calls imfile:modInit() which calls obj:UseObj() and hangs at lock mutObjGlobalOp - thread 419d5940 calls modules:Load() and hangs at lock mutLoadUnload Rainer, am i right so far? if yes, do you have an idea how to fix it? regards, -ap BTW: the issue is not easily reproducible because the timing has to be right. in a test-VM i ususally get a hanging rsyslogd every 5 reboots. From rcorsaro at gmail.com Tue Nov 1 21:06:18 2011 From: rcorsaro at gmail.com (Bob Corsaro) Date: Tue, 1 Nov 2011 16:06:18 -0400 Subject: [rsyslog] imfile state file Message-ID: If there is a better way to do what I'm doing, I'm all ears. I just basically need to send varnish logs to a syslog daemon on a remote server. I'm using rsyslog 4.2.0 packaged with Ubuntu lucid to ship varnish logs to a remote server. The related part of my config looks like this: $InputFileName /var/log/varnish/varnishncsa.log $InputFileTag varnish: $InputFileStateFile stat-varnish $InputFileSeverity debug $InputFileFacility local0 $InputRunFileMonitor $InputFilePollInterval 1 $InputFilePersistStateInterval 20000 local0.* @@varnish-log-processor:515 I realize that $InputFilePersistStateInterval is not supported in 4.2.0, I'll get to that later. The problem is that I'm seeing the same logs sent to varnish-log-processor mutliple times. At some point something happens and all the logs, starting at the beginning of the file, are resent. I'm assuming rsyslog is dying and the state file isn't being created on exit. I did see this in my messages log this morning: Nov 1 04:27:16 balancer01 rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="14143" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'. Anyway, I tried upgrading to 4.8.0, which does leave a state file in $WorkDirectory right away, instead of waiting for rsyslog to exit, but it doesn't seem to respect the $InputFilePersistStateInterval . I've received well over 20k logs since starting it, but the state file remains unchanged. If anyone knows if this problem is related to the way varnish logs, or is a known bug, or any other advice, it would be much appreciated. Thanks, Bob Corsaro From a.piesk at gmx.net Tue Nov 1 22:43:19 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Tue, 01 Nov 2011 22:43:19 +0100 Subject: [rsyslog] imfile state file In-Reply-To: References: Message-ID: <4EB067F7.6090508@gmx.net> On 01.11.2011 21:06, Bob Corsaro wrote: > > Nov 1 04:27:16 balancer01 rsyslogd: [origin software="rsyslogd" > swVersion="4.2.0" x-pid="14143" x-info="http://www.rsyslog.com"] rsyslogd > was HUPed, type 'lightweight'. do you use logrotate? i guess rsyslogd got SIGHUP from logrotate. > Anyway, I tried upgrading to 4.8.0, which does leave a state file in > $WorkDirectory right away, instead of waiting for rsyslog to exit, but it > doesn't seem to respect the $InputFilePersistStateInterval . I've received > well over 20k logs since starting it, but the state file remains unchanged. you know that $InputFilePersistStateInterval is given in lines not bytes, right? regards, -ap From rgerhards at hq.adiscon.com Wed Nov 2 08:32:52 2011 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Nov 2011 08:32:52 +0100 Subject: [rsyslog] rsyslog hangs at startup In-Reply-To: <4EAFBB9E.60805@gmx.net> References: <4EADBB77.5040801@gmx.net> <4EAE9DF0.8080703@gmx.net> <4EAFBB9E.60805@gmx.net> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com> Hi Andreas, we had a public holiday over here yesterday. Thanks for your analysis, this sounds very interesting (and probably explains why I haven't seen it so far). I need to review the code, it's quite a while since I looked at that place the last time, seems it has "worked" for a couple of years. I'll do so later today and let you know the findings. We may need to further instrument/try out some patches... rainer > -----Original Message----- > From: Andreas Piesk [mailto:a.piesk at gmx.net] > Sent: Tuesday, November 01, 2011 10:28 AM > To: rsyslog at lists.adiscon.com; Rainer Gerhards > Subject: Re: [rsyslog] rsyslog hangs at startup > > On 31.10.2011 14:09, Andreas Piesk wrote: > > > > after looking at runtime/modules.c i suspected that the mutex doesn't > get unlocked. > > > > after inserting some more debug printf: > > > > 2426.965520000:2ad3d4b29320: cfline: '$ModLoad imfile' > > 2426.965545000:2ad3d4b29320: Requested to load module 'imfile' > > 2426.965548000:2ad3d4b29320: setting module load/unlock lock > > 2426.965551000:2ad3d4b29320: module load/unload lock set > > 2426.965555000:2ad3d4b29320: loading module > '/lib64/rsyslog/imfile.so' > > 2426.965591000:419d5940: strm 0x2ad3e9475220: file 8 read 537 bytes > > 2426.965733000:419d5940: MsgSetTAG in: len 9, pszBuf: rsyslogd: > > 2426.965740000:419d5940: MsgSetTAG exit: pMsg->iLenTAG 9, pMsg- > >TAG.szBuf: rsyslogd: > > 2426.965776000:419d5940: XXXXX: tryDoAction 0x2ad3e94705a0, pnElem > 1, nElem 1 > > 2426.965780000:419d5940: Action 0x2ad3e94705a0 transitioned to state: > itx > > 2426.965783000:419d5940: entering actionCalldoAction(), state: itx > > 2426.965797000:419d5940: 10.10.0.254 > > 2426.965818000:419d5940: caller requested object 'nsd_ptcp', not > found (iRet -3003) > > 2426.965843000:419d5940: Requested to load module 'lmnsd_ptcp' > > 2426.965846000:419d5940: setting module load/unlock lock > > > > i don't see "imfile: version %s initializing\n", so the problem seems > to be imfile because it > > doesn't get initialized and blocks the mutex. > > > > i've debugged the issue a little bit further and it turned out that > imfile:modInit() hangs at > UseObj(), or to be precise at pthread_mutex_lock(&mutObjGlobalOp). > > i think it's a deadlock between mutObjGlobalOp and mutLoadUnload: > > - thread 2ad3d4b29320 calls modules:Load() which acquires lock > mutLoadUnload > - thread 419d5940 calls obj:UseObj() which aquires lock mutObjGlobalOp > - thread 2ad3d4b29320 calls imfile:modInit() which calls obj:UseObj() > and hangs at lock mutObjGlobalOp > - thread 419d5940 calls modules:Load() and hangs at lock mutLoadUnload > > Rainer, am i right so far? if yes, do you have an idea how to fix it? > > regards, > -ap > > BTW: the issue is not easily reproducible because the timing has to be > right. in a test-VM i > ususally get a hanging rsyslogd every 5 reboots. From rcorsaro at gmail.com Wed Nov 2 16:00:32 2011 From: rcorsaro at gmail.com (Bob Corsaro) Date: Wed, 2 Nov 2011 11:00:32 -0400 Subject: [rsyslog] imfile state file In-Reply-To: <4EB067F7.6090508@gmx.net> References: <4EB067F7.6090508@gmx.net> Message-ID: Thanks for the response. Yes. I think it could be logrotate causing the issue. Are there any best practices to avoid duplicate logs? I open to an alternative option for rolling the logs. As far as $InputFilePersistStateInterval is concerned, I'm aware it's by lines. I've sent over 20k lines to my log and the state file has not been modified. On Tue, Nov 1, 2011 at 5:43 PM, Andreas Piesk wrote: > On 01.11.2011 21:06, Bob Corsaro wrote: > > > > Nov 1 04:27:16 balancer01 rsyslogd: [origin software="rsyslogd" > > swVersion="4.2.0" x-pid="14143" x-info="http://www.rsyslog.com"] > rsyslogd > > was HUPed, type 'lightweight'. > > do you use logrotate? i guess rsyslogd got SIGHUP from logrotate. > > > Anyway, I tried upgrading to 4.8.0, which does leave a state file in > > $WorkDirectory right away, instead of waiting for rsyslog to exit, but it > > doesn't seem to respect the $InputFilePersistStateInterval . I've > received > > well over 20k logs since starting it, but the state file remains > unchanged. > > you know that $InputFilePersistStateInterval is given in lines not bytes, > right? > > regards, > -ap > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbiebl at gmail.com Wed Nov 2 16:25:08 2011 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 2 Nov 2011 16:25:08 +0100 Subject: [rsyslog] imfile state file In-Reply-To: References: Message-ID: 2011/11/1 Bob Corsaro : > If there is a better way to do what I'm doing, I'm all ears. I just > basically need to send varnish logs to a syslog daemon on a remote server. > > I'm using rsyslog 4.2.0 packaged with Ubuntu lucid to ship varnish logs to > a remote server. ?The related part of my config looks like this: [..] > Anyway, I tried upgrading to 4.8.0, which does leave a state file in > $WorkDirectory right away, instead of waiting for rsyslog to exit, but it > doesn't seem to respect the $InputFilePersistStateInterval . ?I've received > well over 20k logs since starting it, but the state file remains unchanged. I've been working on a scenario similar to yours. Several Ubuntu Lucid servers with 4.2 which read from a custom log file and forward that to a central rsyslog server collecting the logs. I also noticed, that 4.2 does not create the state file and iirc I've also seen the resend-logs-on-restart issue, you mentioned, but I haven't investigated that further. Rainer generally recommends to *not* use 4.2. I'm suprised though, as you mentioned that 4.8 is showing the same behaviour, unless I misunderstood you. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From rcorsaro at gmail.com Wed Nov 2 16:47:33 2011 From: rcorsaro at gmail.com (Bob Corsaro) Date: Wed, 2 Nov 2011 11:47:33 -0400 Subject: [rsyslog] imfile state file In-Reply-To: References: Message-ID: On Wed, Nov 2, 2011 at 11:25 AM, Michael Biebl wrote: > > 2011/11/1 Bob Corsaro : > > If there is a better way to do what I'm doing, I'm all ears. I just > > basically need to send varnish logs to a syslog daemon on a remote server. > > > > I'm using rsyslog 4.2.0 packaged with Ubuntu lucid to ship varnish logs to > > a remote server. ?The related part of my config looks like this: > > [..] > > > Anyway, I tried upgrading to 4.8.0, which does leave a state file in > > $WorkDirectory right away, instead of waiting for rsyslog to exit, but it > > doesn't seem to respect the $InputFilePersistStateInterval . ?I've received > > well over 20k logs since starting it, but the state file remains unchanged. > > I've been working on a scenario similar to yours. Several Ubuntu Lucid > servers with 4.2 which read from a custom log file and forward that to > a central rsyslog server collecting the logs. > I also noticed, that 4.2 does not create the state file and iirc I've > also seen the resend-logs-on-restart issue, you mentioned, but I > haven't investigated that further. > > Rainer generally recommends to *not* use 4.2. > I'm suprised though, as you mentioned that 4.8 is showing the same > behaviour, unless I misunderstood you. It's not displaying the exact same behavior. It is creating a state file, but it doesn't seem to respect the persist state interval. At any rate I'm going to stick with 4.2.0 for now. I think the problem was logrotate messing around with inodes and filenames and confusing rsyslog. As a workaround, I've added the following configuration to logrotate for varnish: /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { daily rotate 7 missingok compress delaycompress missingok firstaction stop rsyslog rm /var/spool/rsyslog/my-state-file endscript lastaction start rsyslog endscript postrotate for service in varnishlog varnishncsa; do if /usr/bin/pgrep -P 1 $service >/dev/null; then /usr/sbin/invoke-rc.d $service reload > /dev/null fi done endscript } I may lose a few logs at the end of the day with this method, but It's better then what I had before. I'm using this data to charge customers so I'd rather err on the low side of things. It would be better if I could just have varnish log directly to the remote syslog server :P. Varnish does support syslog in vcl so maybe I'll try it at some point. If anyone has any improvement, I'm always open to suggestion. Sorry for top posting earlier, I haven't used a mailing list in a while. Bob Corsaro From david at lang.hm Wed Nov 2 18:55:51 2011 From: david at lang.hm (david at lang.hm) Date: Wed, 2 Nov 2011 10:55:51 -0700 (PDT) Subject: [rsyslog] imfile state file In-Reply-To: References: Message-ID: On Wed, 2 Nov 2011, Michael Biebl wrote: > 2011/11/1 Bob Corsaro : >> If there is a better way to do what I'm doing, I'm all ears. I just >> basically need to send varnish logs to a syslog daemon on a remote server. >> >> I'm using rsyslog 4.2.0 packaged with Ubuntu lucid to ship varnish logs to >> a remote server. ?The related part of my config looks like this: > > [..] > >> Anyway, I tried upgrading to 4.8.0, which does leave a state file in >> $WorkDirectory right away, instead of waiting for rsyslog to exit, but it >> doesn't seem to respect the $InputFilePersistStateInterval . ?I've received >> well over 20k logs since starting it, but the state file remains unchanged. > > I've been working on a scenario similar to yours. Several Ubuntu Lucid > servers with 4.2 which read from a custom log file and forward that to > a central rsyslog server collecting the logs. > I also noticed, that 4.2 does not create the state file and iirc I've > also seen the resend-logs-on-restart issue, you mentioned, but I > haven't investigated that further. > > Rainer generally recommends to *not* use 4.2. > I'm suprised though, as you mentioned that 4.8 is showing the same > behaviour, unless I misunderstood you. I think it was version 5 that changed the behavior of HUP so that it didn't do a full shutdown/restart but instead just closed and reopened files (which is what's needed for log rotation) I would strongly recommend moving to version 5. David Lang From a.piesk at gmx.net Wed Nov 2 20:15:40 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Wed, 02 Nov 2011 20:15:40 +0100 Subject: [rsyslog] imfile state file In-Reply-To: References: <4EB067F7.6090508@gmx.net> Message-ID: <4EB196DC.9050705@gmx.net> On 02.11.2011 16:00, Bob Corsaro wrote: > > Yes. I think it could be logrotate causing the issue. Are there any best > practices to avoid duplicate logs? I open to an alternative option for > rolling the logs. > have you considered sending the logs after they have been rotated to avoid any duplicates? you know, something like 'logger -p local0.debug -f ', probably in logrotates post-script? i have test cases even for 5.8.6 where at least some lines are duplicated but it doesn't matter in my case because i use syslog for monitoring/security only, not for accounting. regards, -ap From a.piesk at gmx.net Wed Nov 2 22:31:42 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Wed, 02 Nov 2011 22:31:42 +0100 Subject: [rsyslog] rsyslog hangs at startup In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com> References: <4EADBB77.5040801@gmx.net> <4EAE9DF0.8080703@gmx.net> <4EAFBB9E.60805@gmx.net> <9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com> Message-ID: <4EB1B6BE.8050201@gmx.net> On 02.11.2011 08:32, Rainer Gerhards wrote: > > we had a public holiday over here yesterday. Thanks for your analysis, this same here :) > sounds very interesting (and probably explains why I haven't seen it so far). > I need to review the code, it's quite a while since I looked at that place > the last time, seems it has "worked" for a couple of years. I'll do so later > today and let you know the findings. We may need to further instrument/try > out some patches... so save you some time: for my tests i already instrumented the code, basically dbgprintf() around the critical pthread_mutex_* calls. the problem here is, the debug statements change the timing and make it a lot harder to get a dead lock. after a lot of runs i managed to get a dead lock with the instrumented rsyslogd (patch and complete log attached). the interesting part: 7592.695797000:2b073f035320: mutLoadUnload locked <- thread A got mutLoadUnload 7592.695806000:2b073f035320: loading module '/lib64/rsyslog/imfile.so' 7592.695840000:420bc940: mutLoadUnload released <- this a strange 7592.695862000:420bc940: source file netstrms.c requested reference for module 'lmnsd_ptcp', reference count now 1 7592.695868000:420bc940: mutObjGlobalOp released 7592.695885000:420bc940: locking mutObjGlobalOp ... 7592.695890000:420bc940: mutObjGlobalOp locked 7592.695901000:420bc940: mutObjGlobalOp released 7592.696880000:420bc940: 10.10.0.254:514/tcp 7592.696989000:420bc940: TCP sent 181 bytes, requested 181 7592.696999000:420bc940: Action 0x2b0757efe5a0 transitioned to state: rdy 7592.697004000:420bc940: action 0x2b0757efe5a0 call returned 0 7592.697015000:420bc940: Action 0x2b0757efe5a0 transitioned to state: itx 7592.697143000:420bc940: entering actionCalldoAction(), state: itx 7592.697147000:420bc940: 10.10.0.254 7592.697152000:420bc940: 10.10.0.254:514/tcp 7592.697159000:420bc940: CheckConnection detected broken connection - closing it 7592.697173000:420bc940: locking mutObjGlobalOp ... 7592.697177000:420bc940: mutObjGlobalOp locked <- thread B got mutObjGlobalOp 7592.697184000:420bc940: file netstrms.c released module 'lmnsd_ptcp', reference count now 0 7592.697188000:420bc940: module 'lmnsd_ptcp' has zero reference count, unloading... 7592.697192000:420bc940: locking mutLoadUnload ... <- thread B wants mutLoadUnload held by A 7592.703794000:2b073f035320: locking mutObjGlobalOp ... <- thread A wants mutObjGlobalOp held by B 7608.246465000:2b073f035320: DoDie called. <- i did it :) getting mutObjGlobalOp before mutLoadUnload and releasing it afterwards might by a solution but there may be better ones. regards, -ap -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog_debug5.log URL: From a.piesk at gmx.net Thu Nov 3 19:22:40 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Thu, 03 Nov 2011 19:22:40 +0100 Subject: [rsyslog] rsyslog hangs at startup In-Reply-To: <4EB1B6BE.8050201@gmx.net> References: <4EADBB77.5040801@gmx.net> <4EAE9DF0.8080703@gmx.net> <4EAFBB9E.60805@gmx.net> <9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com> <4EB1B6BE.8050201@gmx.net> Message-ID: <4EB2DBF0.6070307@gmx.net> Sorry, forgot the patch adding some debug stuff, fixed :) regards, -ap -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-dbgmutex.patch URL: From rcorsaro at gmail.com Thu Nov 3 21:32:41 2011 From: rcorsaro at gmail.com (Bob Corsaro) Date: Thu, 3 Nov 2011 16:32:41 -0400 Subject: [rsyslog] imfile state file In-Reply-To: <4EB196DC.9050705@gmx.net> References: <4EB067F7.6090508@gmx.net> <4EB196DC.9050705@gmx.net> Message-ID: On Wed, Nov 2, 2011 at 3:15 PM, Andreas Piesk wrote: > > On 02.11.2011 16:00, Bob Corsaro wrote: > > > > Yes. I think it could be logrotate causing the issue. Are there any best > > practices to avoid duplicate logs? I open to an alternative option for > > rolling the logs. > > > > have you considered sending the logs after they have been rotated to avoid any duplicates? > you know, something like 'logger -p local0.debug -f ', probably in logrotates post-script? I could but then I don't get realtime reporting, which is super cool. > i have test cases even for 5.8.6 where at least some lines are duplicated but it doesn't matter in > my case because i use syslog for monitoring/security only, not for accounting. :( The disadvantage of going directly from varnish to the remote syslog server is that I lose the ability to store logs if the remote syslog daemon is unavailable. Why do you think you get duplicates? From a.piesk at gmx.net Thu Nov 3 22:47:18 2011 From: a.piesk at gmx.net (Andreas Piesk) Date: Thu, 03 Nov 2011 22:47:18 +0100 Subject: [rsyslog] imfile state file In-Reply-To: References: <4EB067F7.6090508@gmx.net> <4EB196DC.9050705@gmx.net> Message-ID: <4EB30BE6.2050308@gmx.net> On 03.11.2011 21:32, Bob Corsaro wrote: > On Wed, Nov 2, 2011 at 3:15 PM, Andreas Piesk wrote: > >> i have test cases even for 5.8.6 where at least some lines are duplicated but it doesn't matter in >> my case because i use syslog for monitoring/security only, not for accounting. > > :( The disadvantage of going directly from varnish to the remote > syslog server is that I lose the ability to store logs if the remote > syslog daemon is unavailable. Why do you think you get duplicates? i don't know, i haven't looked into it yet. the test cases run different actions in different combinations, so i can't tell where the duplicates come from. there are only a few and i'm not sure i get them every time. but if varnish can log via syslog (as i understood it), why don't you let varnish log to a local rsyslogd directly without the detour via files? regards, -ap From david at lang.hm Fri Nov 4 10:41:01 2011 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Nov 2011 02:41:01 -0700 (PDT) Subject: [rsyslog] imfile state file In-Reply-To: References: <4EB067F7.6090508@gmx.net> <4EB196DC.9050705@gmx.net> Message-ID: On Thu, 3 Nov 2011, Bob Corsaro wrote: > On Wed, Nov 2, 2011 at 3:15 PM, Andreas Piesk wrote: >> >> On 02.11.2011 16:00, Bob Corsaro wrote: >>> >>> Yes. I think it could be logrotate causing the issue. Are there any best >>> practices to avoid duplicate logs? I open to an alternative option for >>> rolling the logs. >>> >> >> have you considered sending the logs after they have been rotated to avoid any duplicates? >> you know, something like 'logger -p local0.debug -f ', probably in logrotates post-script? > > I could but then I don't get realtime reporting, which is super cool. > >> i have test cases even for 5.8.6 where at least some lines are duplicated but it doesn't matter in >> my case because i use syslog for monitoring/security only, not for accounting. > > :( The disadvantage of going directly from varnish to the remote > syslog server is that I lose the ability to store logs if the remote > syslog daemon is unavailable. Why do you think you get duplicates? Two options there. 1. go to a local syslog daemon and let it forward 2. make your remote syslog daemon highly available. writing to a file and then picking it up later is one of the most complicated approaches that you can take to move logs. David Lang From rcorsaro at gmail.com Sun Nov 6 18:50:42 2011 From: rcorsaro at gmail.com (Bob Corsaro) Date: Sun, 6 Nov 2011 12:50:42 -0500 Subject: [rsyslog] imfile state file In-Reply-To: References: <4EB067F7.6090508@gmx.net> <4EB196DC.9050705@gmx.net> Message-ID: On Fri, Nov 4, 2011 at 5:41 AM, wrote: [snip] > On Thu, 3 Nov 2011, Bob Corsaro wrote: >> :( The disadvantage of going directly from varnish to the remote >> syslog server is that I lose the ability to store logs if the remote >> syslog daemon is unavailable. Why do you think you get duplicates? > > Two options there. > > 1. go to a local syslog daemon and let it forward > > 2. make your remote syslog daemon highly available. > > writing to a file and then picking it up later is one of the most > complicated approaches that you can take to move logs. Of course! Thanks for making me see the obvious solution. For future reference, my solution was to run /usr/bin/varnishnsca | /usr/bin/logger -p local0.info -t mirage-varnish Then create a rule in rsyslog to send that to the remote system: local0.* @@remote-system:514 #(use @ for udp and @@ for tcp) I'm not sure yet if rsyslog is dealing with the remote system not accepting logs by storing logs in the $WorkDirectory, but I'm sure it's doable. I'm ok with losing a few logs here and there for the time being. From rodney.mckee at gmail.com Mon Nov 7 06:17:51 2011 From: rodney.mckee at gmail.com (Rodney McKee) Date: Mon, 07 Nov 2011 16:17:51 +1100 (EST) Subject: [rsyslog] system not recovering after network issue In-Reply-To: <099a7cd7-298e-49cf-9175-f9331bef8831@wsrmckee> Message-ID: <0a284d16-60f4-401d-aeea-7d59e8cb9d51@wsrmckee> Hello, With the following config: ... # Send all logs onto the local relay # *.*;syslog.!=info @@log1;RSYSLOG_ForwardFormat $ActionExecOnlyWhenPreviousIsSuspended on & @@log2 & /var/spool/rsyslog-buffer $ActionExecOnlyWhenPreviousIsSuspended off We have systems stuck in a connect state that do not appear to be recovering: # strace -p 32318 Process 32318 attached - interrupt to quit connect(1, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("log2")}, 16 Process 32318 detached Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/rsyslog/lmnsd_ptcp.so...done. Loaded symbols for /lib64/rsyslog/lmnsd_ptcp.so 0x000000319ca0cf2b in connect () from /lib64/libpthread.so.0 #0 0x000000319ca0cf2b in connect () from /lib64/libpthread.so.0 #1 0x00002aaaab0d1d65 in Connect (pNsd=0x2aaaac8abe10, family=, port=, host=) at nsd_ptcp.c:684 #2 0x000000000040ff29 in TCPSendInit () #3 0x0000000000410038 in doTryResume () #4 0x0000000000436d30 in actionTryResume () #5 0x0000000000437393 in submitBatch () #6 0x0000000000437978 in processBatchMain () #7 0x0000000000435896 in doSubmitToActionQBatch () #8 0x00000000004361f9 in doSubmitToActionQNotAllMarkBatch () #9 0x00000000004325b8 in processBatchDoActions () #10 0x000000000041d5d8 in llExecFunc () #11 0x0000000000432933 in processBatch () #12 0x00000000004319de in processBatchDoRules () #13 0x000000000041d5d8 in llExecFunc () #14 0x0000000000431f04 in processBatch () #15 0x000000000040b5cf in msgConsumer () #16 0x0000000000430dcd in ConsumerReg () #17 0x000000000042a51c in wtiWorker () #18 0x000000000042a136 in wtpWorker () #19 0x000000319ca062f7 in start_thread () from /lib64/libpthread.so.0 #20 0x000000319c2d1b6d in clone () from /lib64/libc.so.6 $ sudo /usr/sbin/lsof -p 20064 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME rsyslogd 20064 root cwd DIR 8,7 4096 2 / rsyslogd 20064 root rtd DIR 8,7 4096 2 / rsyslogd 20064 root txt REG 8,7 441537 1310792 /sbin/rsyslogd rsyslogd 20064 root mem REG 8,7 134400 885017 /lib64/ld-2.5.so rsyslogd 20064 root mem REG 8,7 1699912 885018 /lib64/libc-2.5.so rsyslogd 20064 root mem REG 8,7 23360 885019 /lib64/libdl-2.5.so rsyslogd 20064 root mem REG 8,7 141440 885023 /lib64/libpthread-2.5.so rsyslogd 20064 root mem REG 8,7 53448 885024 /lib64/librt-2.5.so rsyslogd 20064 root mem REG 8,6 85928 164425 /usr/lib64/libz.so.1.2.3 rsyslogd 20064 root mem REG 8,7 92736 884796 /lib64/libresolv-2.5.so rsyslogd 20064 root mem REG 8,7 53880 884764 /lib64/libnss_files-2.5.so rsyslogd 20064 root mem REG 8,7 23632 884762 /lib64/libnss_dns-2.5.so rsyslogd 20064 root mem REG 8,7 93320 885005 /lib64/rsyslog/lmnsd_ptcp.so rsyslogd 20064 root mem REG 8,7 75802 884921 /lib64/rsyslog/lmnet.so rsyslogd 20064 root mem REG 8,7 1295631 884806 /lib64/rsyslog/imuxsock.so rsyslogd 20064 root mem REG 8,7 81914 884794 /lib64/rsyslog/imklog.so rsyslogd 20064 root mem REG 8,7 57594 884804 /lib64/rsyslog/imudp.so rsyslogd 20064 root mem REG 8,7 37373 884801 /lib64/rsyslog/impstats.so rsyslogd 20064 root mem REG 8,7 90803 884994 /lib64/rsyslog/lmnetstrms.so rsyslogd 20064 root mem REG 8,7 35770 884873 /lib64/rsyslog/lmtcpclt.so rsyslogd 20064 root 0u unix 0xffff81031e95e0c0 876257540 /dev/log rsyslogd 20064 root 1u IPv4 1000073229 TCP app1.lhr.acx:43293->192.168.132.143:shell (SYN_SENT) rsyslogd 20064 root 2r 0000 0,10 0 876257542 eventpoll rsyslogd 20064 root 3u IPv6 876257538 UDP *:syslog rsyslogd 20064 root 4u IPv4 876257539 UDP *:syslog rsyslogd 20064 root 8r REG 0,3 0 4026531849 /proc/kmsg The suspected code is in nsd_ptcp.c as it does not appear to allow for a timeout with a NODELAY or other mechanism on the connect. if((pThis->sock = socket(res->ai_family, res->ai_socktype, res->ai_protocol)) == -1) { ABORT_FINALIZE(RS_RET_IO_ERROR); } if(connect(pThis->sock, res->ai_addr, res->ai_addrlen) != 0) { ABORT_FINALIZE(RS_RET_IO_ERROR); } Rgds Rodney From djbristow at optonline.net Mon Nov 7 19:11:22 2011 From: djbristow at optonline.net (David Bristow) Date: Mon, 07 Nov 2011 13:11:22 -0500 Subject: [rsyslog] Rsyslog spool files not going away Message-ID: <1320689483.1727.3.camel@localhost> When we have a connectivity problem and clients start to spool to the: $WorkDirectory /var/spool/rsyslog the total amount of files in that directory can grow pretty big. When connectivity resumes, however, it does not seem like those spool files go away. Is this expected? -- David Bristow From Murphy.Brandon at principal.com Mon Nov 7 23:41:52 2011 From: Murphy.Brandon at principal.com (Murphy, Brandon) Date: Mon, 7 Nov 2011 16:41:52 -0600 Subject: [rsyslog] omudpspoof performance Message-ID: I have noticed that using the omudpspoof action with the omudpspoof module enabled results in very slow forwards. I suspect this is due to the action of creating new sockets for each message being spoofed. I have noticed, that even if all the messages being forwarded via a omudpspoof action are from a single ip address, it creates new sockets for each message being forwarded. This results in very slow forwarding times. I believe it would be much quicker, if instead, a single socket is created for each unique source ip address, send all messages via the open socket, keep that session open for a given timeout (perhaps configurable), and if it remains idle for that timeframe, close it. I attempted to speed up the forwarding process by limiting the spoof port start and end configuration options to a single port, but there was no noticeable increase in speed. I am running version 5.8.6 on RHEL 5.6 Brandon Murphy | IT Network Security Assoc-Lead | Information Services - Information Security | The Principal Financial Group(r) | ph 515.247.5161
-----Message Disclaimer-----

This e-mail message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply email to Connect at principal.com and delete or destroy all copies of the original message and attachments thereto. Email sent to or from the Principal Financial Group or any of its member companies may be retained as required by law or regulation.

Nothing in this message is intended to constitute an Electronic signature for purposes of the Uniform Electronic Transactions Act (UETA) or the Electronic Signatures in Global and National Commerce Act ("E-Sign") unless a specific statement to the contrary is included in this message.

While this communication may be used to promote or market a transaction or an idea that is discussed in the publication, it is intended to provide general information about the subject matter covered and is provided with the understanding that The Principal is not rendering legal, accounting, or tax advice. It is not a marketed opinion and may not be used to avoid penalties under the Internal Revenue Code. You should consult with appropriate counsel or other advisors on all matters pertaining to legal, tax, or accounting obligations and requirements.

From david at lang.hm  Tue Nov  8 00:01:12 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 7 Nov 2011 15:01:12 -0800 (PST)
Subject: [rsyslog] omudpspoof performance
In-Reply-To: 
References: 
Message-ID: 

On Mon, 7 Nov 2011, Murphy, Brandon wrote:

> I have noticed that using the omudpspoof action with the omudpspoof 
> module enabled results in very slow forwards.  I suspect this is due to 
> the action of creating new sockets for each message being spoofed.

Yes, that would be the bottleneck.

> I have noticed, that even if all the messages being forwarded via a 
> omudpspoof action are from a single ip address, it creates new sockets 
> for each message being forwarded.  This results in very slow forwarding 
> times.  I believe it would be much quicker, if instead, a single socket 
> is created for each unique source ip address, send all messages via the 
> open socket, keep that session open for a given timeout (perhaps 
> configurable), and if it remains idle for that timeframe, close it.

This approach would solve the problem, however the code to manage the open 
sockets would be significant. A large number of open, but idle sockets 
could cause grief for other processes trying to grab ephermeral ports.

The other approach, which would be faster and safer, would be to craft the 
raw packet directly rather than opening a socket and writing to it. This 
is probably simpler than managing (including expiring) a large population 
of sockets, but is far from trivial.

The current 'bind to a socket for every message' approach was implemented 
because it was the easiest one to do.

> I attempted to speed up the forwarding process by limiting the spoof 
> port start and end configuration options to a single port, but there was 
> no noticeable increase in speed.

This would not make any difference (and would actually risk slowing things 
down if it can't re-use a port already in use)

David Lang

From rgerhards at hq.adiscon.com  Tue Nov  8 08:42:58 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 8 Nov 2011 08:42:58 +0100
Subject: [rsyslog] Rsyslog spool files not going away
In-Reply-To: <1320689483.1727.3.camel@localhost>
References: <1320689483.1727.3.camel@localhost>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728143E@GRFEXC.intern.adiscon.com>

The current file for each disk assisted queue will remain, once the DA thread
has been initiated. Multiple files *for the same queue* should not remain.

Hth rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of David Bristow
> Sent: Monday, November 07, 2011 7:11 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Rsyslog spool files not going away
> 
> When we have a connectivity problem and clients start to spool to the:
> 
> $WorkDirectory /var/spool/rsyslog
> 
> the total amount of files in that directory can grow pretty big.  When
> connectivity resumes, however, it does not seem like those spool files
> go away.
> 
> Is this expected?
> 
> --
> David Bristow 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

From victor.lu at citi.com  Tue Nov  8 17:58:39 2011
From: victor.lu at citi.com (Lu, Victor )
Date: Tue, 8 Nov 2011 10:58:39 -0600
Subject: [rsyslog] Rsyslog spool files not going away
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728143E@GRFEXC.intern.adiscon.com>
References: <1320689483.1727.3.camel@localhost>
	<9B6E2A8877C38245BFB15CC491A11DA728143E@GRFEXC.intern.adiscon.com>
Message-ID: <35B12B7283BF44478AFA717323EE52951CEDECE5DD@extxmb32.nam.nsroot.net>

David,

You did not mention about which version of rsyslog collector you are using and whether tls is used for communication and on what OS platforms. During my testing on RHEL 5 and 6, I found the disk files are not going away only with TLS on a lower version rsyslog collector. However the rest of them are working fine.

Regarding the amount of files in the queue directory, The following two tokens control the queue file size and number of files to be created. When the maximum queue disk space is reached, the new arrived messages will be discarded, so the number of files will not grow.

ActionQueueMaxFileSize
ActionQueueMaxDiskSpace

Hope it helps. 

Victor

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Tuesday, November 08, 2011 2:43 AM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog spool files not going away

The current file for each disk assisted queue will remain, once the DA thread
has been initiated. Multiple files *for the same queue* should not remain.

Hth rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of David Bristow
> Sent: Monday, November 07, 2011 7:11 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Rsyslog spool files not going away
> 
> When we have a connectivity problem and clients start to spool to the:
> 
> $WorkDirectory /var/spool/rsyslog
> 
> the total amount of files in that directory can grow pretty big.  When
> connectivity resumes, however, it does not seem like those spool files
> go away.
> 
> Is this expected?
> 
> --
> David Bristow 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From giacomo.bandoni at gmail.com  Wed Nov  9 14:04:19 2011
From: giacomo.bandoni at gmail.com (Giacomo Bandoni)
Date: Wed, 9 Nov 2011 13:04:19 +0000
Subject: [rsyslog] Apache error logs problem
Message-ID: 

Is there a way for Rsyslog remote logging to coexist with local Apache
error log?

I'm using the following lines on httpd.conf, but I'm getting either
remote or local depending on which line I put as 1st one:

ErrorLog "|/usr/bin/logger -t httpd_error -p local6.err"
ErrorLog logs/error_log

A similar combination works fine with the CustomLog directive

Thanks

From djbristow at optonline.net  Wed Nov  9 15:17:39 2011
From: djbristow at optonline.net (David Bristow)
Date: Wed, 09 Nov 2011 09:17:39 -0500
Subject: [rsyslog] Rsyslog spool files not going away
In-Reply-To: <35B12B7283BF44478AFA717323EE52951CEDECE5DD@extxmb32.nam.nsroot.net>
References: <1320689483.1727.3.camel@localhost>
	<9B6E2A8877C38245BFB15CC491A11DA728143E@GRFEXC.intern.adiscon.com>
	<35B12B7283BF44478AFA717323EE52951CEDECE5DD@extxmb32.nam.nsroot.net>
Message-ID: <1320848260.1727.7.camel@localhost>

The version of rsyslog is 4.2.0-2ubuntu8.1 from Ubuntu Server 10.04 LTS
and I don't believe we are using TLS.

ActionQueueMaxDiskSpace looks promising.

On Tue, 2011-11-08 at 10:58 -0600, Lu, Victor wrote:
> David,
> 
> You did not mention about which version of rsyslog collector you are using and whether tls is used for communication and on what OS platforms. During my testing on RHEL 5 and 6, I found the disk files are not going away only with TLS on a lower version rsyslog collector. However the rest of them are working fine.
> 
> Regarding the amount of files in the queue directory, The following two tokens control the queue file size and number of files to be created. When the maximum queue disk space is reached, the new arrived messages will be discarded, so the number of files will not grow.
> 
> ActionQueueMaxFileSize
> ActionQueueMaxDiskSpace
> 
> Hope it helps. 
> 
> Victor
> 
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, November 08, 2011 2:43 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog spool files not going away
> 
> The current file for each disk assisted queue will remain, once the DA thread
> has been initiated. Multiple files *for the same queue* should not remain.
> 
> Hth rainer
> 
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of David Bristow
> > Sent: Monday, November 07, 2011 7:11 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Rsyslog spool files not going away
> > 
> > When we have a connectivity problem and clients start to spool to the:
> > 
> > $WorkDirectory /var/spool/rsyslog
> > 
> > the total amount of files in that directory can grow pretty big.  When
> > connectivity resumes, however, it does not seem like those spool files
> > go away.
> > 
> > Is this expected?
> > 
> > --
> > David Bristow 
> > 
> > 
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

-- 
David Bristow 


From victor.lu at citi.com  Wed Nov  9 16:34:47 2011
From: victor.lu at citi.com (Lu, Victor )
Date: Wed, 9 Nov 2011 09:34:47 -0600
Subject: [rsyslog] Rsyslog spool files not going away
In-Reply-To: <1320848260.1727.7.camel@localhost>
References: <1320689483.1727.3.camel@localhost>
	<9B6E2A8877C38245BFB15CC491A11DA728143E@GRFEXC.intern.adiscon.com>
	<35B12B7283BF44478AFA717323EE52951CEDECE5DD@extxmb32.nam.nsroot.net>
	<1320848260.1727.7.camel@localhost>
Message-ID: <35B12B7283BF44478AFA717323EE52951CEDFA5F7F@extxmb32.nam.nsroot.net>

I suggest you should try a later version rsyslog package to see if it works.

-----Original Message-----
From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of David Bristow
Sent: Wednesday, November 09, 2011 9:18 AM
To: rsyslog-users
Cc: Brian Gupta
Subject: Re: [rsyslog] Rsyslog spool files not going away

The version of rsyslog is 4.2.0-2ubuntu8.1 from Ubuntu Server 10.04 LTS
and I don't believe we are using TLS.

ActionQueueMaxDiskSpace looks promising.

On Tue, 2011-11-08 at 10:58 -0600, Lu, Victor wrote:
> David,
>
> You did not mention about which version of rsyslog collector you are using and whether tls is used for communication and on what OS platforms. During my testing on RHEL 5 and 6, I found the disk files are not going away only with TLS on a lower version rsyslog collector. However the rest of them are working fine.
>
> Regarding the amount of files in the queue directory, The following two tokens control the queue file size and number of files to be created. When the maximum queue disk space is reached, the new arrived messages will be discarded, so the number of files will not grow.
>
> ActionQueueMaxFileSize
> ActionQueueMaxDiskSpace
>
> Hope it helps.
>
> Victor
>
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Tuesday, November 08, 2011 2:43 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog spool files not going away
>
> The current file for each disk assisted queue will remain, once the DA thread
> has been initiated. Multiple files *for the same queue* should not remain.
>
> Hth rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> > bounces at lists.adiscon.com] On Behalf Of David Bristow
> > Sent: Monday, November 07, 2011 7:11 PM
> > To: rsyslog at lists.adiscon.com
> > Subject: [rsyslog] Rsyslog spool files not going away
> >
> > When we have a connectivity problem and clients start to spool to the:
> >
> > $WorkDirectory /var/spool/rsyslog
> >
> > the total amount of files in that directory can grow pretty big.  When
> > connectivity resumes, however, it does not seem like those spool files
> > go away.
> >
> > Is this expected?
> >
> > --
> > David Bristow 
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

--
David Bristow 

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

From nathans at aconex.com  Thu Nov 10 00:51:32 2011
From: nathans at aconex.com (Nathan Scott)
Date: Thu, 10 Nov 2011 10:51:32 +1100 (EST)
Subject: [rsyslog] Missing changelog for latest stable release
In-Reply-To: <1633925521.56476.1320882455613.JavaMail.root@acxmail-au2.aconex.com>
Message-ID: <751909601.56496.1320882692741.JavaMail.root@acxmail-au2.aconex.com>

Hi all,

There's a link from here:
http://www.rsyslog.com/tag/5-8-6/

To here:
http://www.rsyslog.com/changelog-for-5-8-6-v5-stable/

Which appears to be broken (the latter URL) - gives a 404.

(Just FYI).

cheers.

-- 
Nathan

From smainklh at free.fr  Mon Nov 14 09:54:17 2011
From: smainklh at free.fr (smainklh at free.fr)
Date: Mon, 14 Nov 2011 09:54:17 +0100 (CET)
Subject: [rsyslog] Rsyslog freezes slowing down authentication
In-Reply-To: <1493012091.12372651321260722489.JavaMail.root@zimbra26-e5.priv.proxad.net>
Message-ID: <592013793.12373391321260857341.JavaMail.root@zimbra26-e5.priv.proxad.net>

Hi all,

We are recently facing a very annoying problem in our production environment with the rsyslog daemon.
>From time to time our monitoring system sends us alerts stating that our server in unreachable.
We investigated and figured out that the rsyslog daemon stopped working properly (the /var/log/messages logfile is not filed anymore).
And consequently every connection attempts are slowed down (for our monitoring system that was the nrpe check attempts).

The only way we found in order to solve that is to restart the rsyslog daemon.

Our rsyslog is configured in order to send some logs to 2 remote servers.
We noticed that this problem occurs when we have a pic of activity.
Could it be related to a network issue ?

Could you please give us your support ?

Thanks,
Sam

From john at feurix.com  Mon Nov 14 12:11:35 2011
From: john at feurix.com (John Feuerstein)
Date: Mon, 14 Nov 2011 12:11:35 +0100
Subject: [rsyslog] Apache error logs problem
In-Reply-To: 
References: 
Message-ID: <4EC0F767.9070804@feurix.com>

On 11/09/2011 02:04 PM, Giacomo Bandoni wrote:
> Is there a way for Rsyslog remote logging to coexist with local Apache
> error log?
>
> I'm using the following lines on httpd.conf, but I'm getting either
> remote or local depending on which line I put as 1st one:
>
> ErrorLog "|/usr/bin/logger -t httpd_error -p local6.err"
> ErrorLog logs/error_log

Apache can log to syslog directly:
http://httpd.apache.org/docs/2.2/mod/core.html#errorlog

ErrorLog syslog:local6

The final location (remote or local) depends entirely on your syslogd 
configuration.

From francois.billard at alyseo.com  Thu Nov 17 19:28:43 2011
From: francois.billard at alyseo.com (Francois Billard)
Date: Thu, 17 Nov 2011 19:28:43 +0100
Subject: [rsyslog] Relay distinction
In-Reply-To: <013d01cca4a9$616f63a0$244e2ae0$@alyseo.com>
References: <013d01cca4a9$616f63a0$244e2ae0$@alyseo.com>
Message-ID: <017c01cca556$bdba5ea0$392f1be0$@alyseo.com>

Hi,

 

I have rsyslog relay installed on my customers sites, they all relay syslog
at my datacenter. 

To distinguish my customers (not possible on IP), i would like to configure
the rsyslog customer config file to concatenate a string (the name of the
customer) with the syslog message (no matter if the customer name is
hardcoded in the config file). How could i do that ?

Thanks

 

 

Regards,

Francois


From cosmomill at gmail.com  Thu Nov 17 19:38:10 2011
From: cosmomill at gmail.com (Rene Kanzler)
Date: Thu, 17 Nov 2011 19:38:10 +0100
Subject: [rsyslog] Outchannel is not working
Message-ID: 

Hello

I try to use output channels in rsyslog 5.8.6 and it does not working at
all. This is my config:

$outchannel
logrotation,/var/log/messages,5242880,/etc/rsyslog.d/log_rotation_script/messages.sh
*.* :omfile:$logrotation

After starting rsyslogd in debug mode the output shows me the following
lines:

Outchannel: Name='logrotation'
File Template: '/var/log/messages'
Max Size.....: 0
OnSizeLimtCmd: '5242880,/etc/rsyslog.d/log_rotation_script/messages.sh'

Did I made a mistake in my configuration?
Or is this an error in "outchannel.c"? It seems that "outchannel.c"
did not correctly parse the "Outchannel" statement in "rsyslog.conf".

Thanks
Rene

From tom.teal at noaa.gov  Thu Nov 17 19:43:27 2011
From: tom.teal at noaa.gov (Tom Teal)
Date: Thu, 17 Nov 2011 13:43:27 -0500
Subject: [rsyslog] rsyslog for AIX
Message-ID: <4EC555CF.7000601@noaa.gov>

Hello,

I am posting this question a second time since we had email problems and
my not have received someone's response.  Has anyone used rsyslog on the
AIX platform and if so could you share your information regarding the
install and configuration of it?  Thanks...

Tom

-- 
Tom Teal
Systems Administrator
NOAA's National Climatic Data Center
Veach-Baley Federal Building
151 Patton Avenue, Suite 320
Asheville, NC 28801



From aoz.syn at gmail.com  Thu Nov 17 21:12:11 2011
From: aoz.syn at gmail.com (RB)
Date: Thu, 17 Nov 2011 13:12:11 -0700
Subject: [rsyslog] rsyslog for AIX
In-Reply-To: <4EC555CF.7000601@noaa.gov>
References: <4EC555CF.7000601@noaa.gov>
Message-ID: 

On Thu, Nov 17, 2011 at 11:43, Tom Teal  wrote:
> Hello,
>
> I am posting this question a second time since we had email problems and
> my not have received someone's response. ?Has anyone used rsyslog on the
> AIX platform and if so could you share your information regarding the
> install and configuration of it? ?Thanks...

I have not personally, but there are relatively recent AIX5 RPMs here:

   http://www.oss4aix.org/download/RPMS/ryslog/

I've also been told on good authority that the RPMs on that site tend
to work rather well.

From tom.teal at noaa.gov  Thu Nov 17 21:24:39 2011
From: tom.teal at noaa.gov (Tom Teal)
Date: Thu, 17 Nov 2011 15:24:39 -0500
Subject: [rsyslog] rsyslog for AIX
In-Reply-To: 
References: <4EC555CF.7000601@noaa.gov>
	
Message-ID: <4EC56D87.2010307@noaa.gov>

Thanks RB for that info.  I'm at the website now and will check it out
in the next few days and see how it does.  Thanks again........

Tom

On 11/17/2011 3:12 PM, RB wrote:
> On Thu, Nov 17, 2011 at 11:43, Tom Teal  wrote:
>> Hello,
>>
>> I am posting this question a second time since we had email problems and
>> my not have received someone's response.  Has anyone used rsyslog on the
>> AIX platform and if so could you share your information regarding the
>> install and configuration of it?  Thanks...
> I have not personally, but there are relatively recent AIX5 RPMs here:
>
>    http://www.oss4aix.org/download/RPMS/ryslog/
>
> I've also been told on good authority that the RPMs on that site tend
> to work rather well.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 
Tom Teal
Systems Administrator
NOAA's National Climatic Data Center
Veach-Baley Federal Building
151 Patton Avenue, Suite 320
Asheville, NC 28801
STG Inc. - Government Contractor
Tel: 828-271-4436
Fax: 828-271-4009
tom.teal at noaa.gov


From rgerhards at hq.adiscon.com  Fri Nov 18 07:31:25 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 18 Nov 2011 07:31:25 +0100
Subject: [rsyslog] rsyslog for AIX
In-Reply-To: <4EC56D87.2010307@noaa.gov>
References: <4EC555CF.7000601@noaa.gov>
	<4EC56D87.2010307@noaa.gov>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814B4@GRFEXC.intern.adiscon.com>

Hi Tom,

it would be great if you could post your experience. It if is well, I'll
probably add a link from rsyslog.com, as this would be useful for other users
as well.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Tom Teal
> Sent: Thursday, November 17, 2011 9:25 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] rsyslog for AIX
> 
> Thanks RB for that info.  I'm at the website now and will check it out
> in the next few days and see how it does.  Thanks again........
> 
> Tom
> 
> On 11/17/2011 3:12 PM, RB wrote:
> > On Thu, Nov 17, 2011 at 11:43, Tom Teal  wrote:
> >> Hello,
> >>
> >> I am posting this question a second time since we had email problems
> and
> >> my not have received someone's response.  Has anyone used rsyslog on
> the
> >> AIX platform and if so could you share your information regarding
> the
> >> install and configuration of it?  Thanks...
> > I have not personally, but there are relatively recent AIX5 RPMs
> here:
> >
> >    http://www.oss4aix.org/download/RPMS/ryslog/
> >
> > I've also been told on good authority that the RPMs on that site tend
> > to work rather well.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> 
> --
> Tom Teal
> Systems Administrator
> NOAA's National Climatic Data Center
> Veach-Baley Federal Building
> 151 Patton Avenue, Suite 320
> Asheville, NC 28801
> STG Inc. - Government Contractor
> Tel: 828-271-4436
> Fax: 828-271-4009
> tom.teal at noaa.gov
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From rgerhards at hq.adiscon.com  Fri Nov 18 07:33:51 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 18 Nov 2011 07:33:51 +0100
Subject: [rsyslog] Relay distinction
In-Reply-To: <017c01cca556$bdba5ea0$392f1be0$@alyseo.com>
References: <013d01cca4a9$616f63a0$244e2ae0$@alyseo.com>
	<017c01cca556$bdba5ea0$392f1be0$@alyseo.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814B5@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Francois Billard
> Sent: Thursday, November 17, 2011 7:29 PM
> To: rsyslog at lists.adiscon.com
> Subject: [rsyslog] Relay distinction
> 
> Hi,
> 
> 
> 
> I have rsyslog relay installed on my customers sites, they all relay
> syslog
> at my datacenter.
> 
> To distinguish my customers (not possible on IP), i would like to
> configure
> the rsyslog customer config file to concatenate a string (the name of
> the
> customer) with the syslog message (no matter if the customer name is
> hardcoded in the config file). How could i do that ?


You can do this with a custom template, probably based on the default
templates. That would be along the lines (search doc correct details):

$template mytempl,"...CUSTOMERA..."
@@ h.example.net;mytempl


Hth
rainer
> Thanks
> 
> 
> 
> 
> 
> Regards,
> 
> Francois
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From tom.teal at noaa.gov  Fri Nov 18 13:13:20 2011
From: tom.teal at noaa.gov (Tom Teal)
Date: Fri, 18 Nov 2011 07:13:20 -0500
Subject: [rsyslog] rsyslog for AIX
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72814B4@GRFEXC.intern.adiscon.com>
References: <4EC555CF.7000601@noaa.gov>
	
	<4EC56D87.2010307@noaa.gov>
	<9B6E2A8877C38245BFB15CC491A11DA72814B4@GRFEXC.intern.adiscon.com>
Message-ID: <4EC64BE0.9070400@noaa.gov>

Hey Rainer,

No problem with sharing my findings.  I will be starting the install
today and collecting notes as I go.  Once complete I will share the
notes that I accumulated through out the entire process.  Thanks for
your help.

Tom 
 

On 11/18/2011 1:31 AM, Rainer Gerhards wrote:
> Hi Tom,
>
> it would be great if you could post your experience. It if is well, I'll
> probably add a link from rsyslog.com, as this would be useful for other users
> as well.
>
> Thanks,
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Tom Teal
>> Sent: Thursday, November 17, 2011 9:25 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] rsyslog for AIX
>>
>> Thanks RB for that info.  I'm at the website now and will check it out
>> in the next few days and see how it does.  Thanks again........
>>
>> Tom
>>
>> On 11/17/2011 3:12 PM, RB wrote:
>>> On Thu, Nov 17, 2011 at 11:43, Tom Teal  wrote:
>>>> Hello,
>>>>
>>>> I am posting this question a second time since we had email problems
>> and
>>>> my not have received someone's response.  Has anyone used rsyslog on
>> the
>>>> AIX platform and if so could you share your information regarding
>> the
>>>> install and configuration of it?  Thanks...
>>> I have not personally, but there are relatively recent AIX5 RPMs
>> here:
>>>    http://www.oss4aix.org/download/RPMS/ryslog/
>>>
>>> I've also been told on good authority that the RPMs on that site tend
>>> to work rather well.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>> --
>> Tom Teal
>> Systems Administrator
>> NOAA's National Climatic Data Center
>> Veach-Baley Federal Building
>> 151 Patton Avenue, Suite 320
>> Asheville, NC 28801
>> STG Inc. - Government Contractor
>> Tel: 828-271-4436
>> Fax: 828-271-4009
>> tom.teal at noaa.gov
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 
Tom Teal
Systems Administrator
NOAA's National Climatic Data Center
Veach-Baley Federal Building
151 Patton Avenue, Suite 320
Asheville, NC 28801
STG Inc. - Government Contractor
Tel: 828-271-4436
Fax: 828-271-4009
tom.teal at noaa.gov


From a.piesk at gmx.net  Fri Nov 18 21:21:31 2011
From: a.piesk at gmx.net (Andreas Piesk)
Date: Fri, 18 Nov 2011 21:21:31 +0100
Subject: [rsyslog] Rsyslog freezes slowing down authentication
In-Reply-To: <592013793.12373391321260857341.JavaMail.root@zimbra26-e5.priv.proxad.net>
References: <592013793.12373391321260857341.JavaMail.root@zimbra26-e5.priv.proxad.net>
Message-ID: <4EC6BE4B.50408@gmx.net>

On 14.11.2011 09:54, smainklh at free.fr wrote:
> 
> We are recently facing a very annoying problem in our production environment with the rsyslog daemon.
> From time to time our monitoring system sends us alerts stating that our server in unreachable.
> We investigated and figured out that the rsyslog daemon stopped working properly (the /var/log/messages logfile is not filed anymore).
> And consequently every connection attempts are slowed down (for our monitoring system that was the nrpe check attempts).
> 
> The only way we found in order to solve that is to restart the rsyslog daemon.
> 
> Our rsyslog is configured in order to send some logs to 2 remote servers.
> We noticed that this problem occurs when we have a pic of activity.
> Could it be related to a network issue ?
> 
> Could you please give us your support ?
> 

which rsyslog version on what platform?
do you use by any chance the imfile module?

i encountered a similar problem, search the mailing list archive for
"rsyslog stops local logging and local logging hangsif remote destination is unresponsive".

i solved the problem with a small patch based on Rainer's suggestion. you can find the patch in the
mentioned thread.

regards,
-ap

From a.piesk at gmx.net  Fri Nov 18 21:30:06 2011
From: a.piesk at gmx.net (Andreas Piesk)
Date: Fri, 18 Nov 2011 21:30:06 +0100
Subject: [rsyslog] rsyslog hangs at startup
In-Reply-To: <4EB1B6BE.8050201@gmx.net>
References: <4EADBB77.5040801@gmx.net>
	<4EAE9DF0.8080703@gmx.net>	<4EAFBB9E.60805@gmx.net>	<9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com>
	<4EB1B6BE.8050201@gmx.net>
Message-ID: <4EC6C04E.60301@gmx.net>

On 02.11.2011 22:31, Andreas Piesk wrote:
> 
>> sounds very interesting (and probably explains why I haven't seen it so far).
>> I need to review the code, it's quite a while since I looked at that place
>> the last time, seems it has "worked" for a couple of years. I'll do so later
>> today and let you know the findings. We may need to further instrument/try
>> out some patches... 

Rainer, have you found some time to look at the issue?

what about the idea to use one mutex for both operations? i mean, wouldn't replacing the mutexes
mutLoadUnload and mutObjGlobalOp with a single mutex, type PTHREAD_MUTEX_RECURSIVE, solve the problem?

regards,
-ap

From smainklh at free.fr  Sat Nov 19 15:36:39 2011
From: smainklh at free.fr (smainklh at free.fr)
Date: Sat, 19 Nov 2011 15:36:39 +0100 (CET)
Subject: [rsyslog] Rsyslog freezes slowing down authentication
In-Reply-To: <4EC6BE4B.50408@gmx.net>
Message-ID: <1838760478.13545261321713399418.JavaMail.root@zimbra26-e5.priv.proxad.net>


----- Mail Original -----
De: "Andreas Piesk" 
?: "rsyslog-users" 
Cc: smainklh at free.fr
Envoy?: Vendredi 18 Novembre 2011 21h21:31 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [rsyslog] Rsyslog freezes slowing down authentication

On 14.11.2011 09:54, smainklh at free.fr wrote:
> 
> We are recently facing a very annoying problem in our production environment with the rsyslog daemon.
> From time to time our monitoring system sends us alerts stating that our server in unreachable.
> We investigated and figured out that the rsyslog daemon stopped working properly (the /var/log/messages logfile is not filed anymore).
> And consequently every connection attempts are slowed down (for our monitoring system that was the nrpe check attempts).
> 
> The only way we found in order to solve that is to restart the rsyslog daemon.
> 
> Our rsyslog is configured in order to send some logs to 2 remote servers.
> We noticed that this problem occurs when we have a pic of activity.
> Could it be related to a network issue ?
> 
> Could you please give us your support ?
> 

which rsyslog version on what platform?
do you use by any chance the imfile module?

i encountered a similar problem, search the mailing list archive for
"rsyslog stops local logging and local logging hangsif remote destination is unresponsive".

i solved the problem with a small patch based on Rainer's suggestion. you can find the patch in the
mentioned thread.

regards,
-ap
-----------------------------------

Hi Andreas,

Sorry i indeed forgot to mention the version. The plateform is RHEL. 

"> do you use by any chance the imfile module?"
How could i check that ?

I found your thread and it seems very similar to the issue i'm facing.

I'll check that on monday. I'm currently off :)

Thank you, i'll keep you updated.



From tom.teal at noaa.gov  Mon Nov 21 20:29:23 2011
From: tom.teal at noaa.gov (Tom Teal)
Date: Mon, 21 Nov 2011 14:29:23 -0500
Subject: [rsyslog] rsyslog on AIX
Message-ID: <4ECAA693.7090402@noaa.gov>

Hello,

Working on a rsyslog install on an AIX platform. 

The install of rsyslog worked with no errors and after editing the
rsyslog.conf file I get a Segmentation fault error:

Starting Rsyslogd... ./Srsyslog[66]: 9633992 Segmentation fault(coredump)
done.


Here are the statements added to the rsyslog.conf file (ip address was
masked with 9's for this email):

######## TCP Forwarding #########
$WorkDirectory /var/log/rsyslog # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

# Temporary log server IP address
*.* @@999.999.999.999:5140

Here is the output from the dbx debug tool of the core file:

Type 'help' for help.
[using memory image in /etc/rc.d/rc2.d/core]
reading symbolic information ...
Segmentation fault in glink.strlen at 0xd55da6d0 ($t1)
0xd55da6d0 (strlen+0x30) 800c0000         lwz   r0,0x0(r12)
(dbx)  thread  state-k     wchan    state-u    k-tid   mode held scope
function
>$t1     run                  running  30277685     k   no   sys  strlen
(dbx) glink.strlen() at 0xd55da6d0
netClassInit(pModInfo = 0x30012e68), line 1636 in "net.c"
should_use_so_bsdcompat(0x5, 0x30012e70, 0x30012e88, 0x30005ab8,
0x30012e68), line 1665 in "net.c"
doModInit(modInit = 0xf0ef4494, name = "lmnet", pModHdlr = 0x00000003),
line 425 in "modules.c"
Load(pModName = "lmnet"), line 851 in "modules.c"
UseObj(srcFile = "conf.c", pObjName = "net", pObjFile = "lmnet", pIf =
0x300013e0), line 1157 in "obj.c"
confClassInit(pModInfo = (nil)), line 1294 in "conf.c"
rsrtInit(ppErrObj = 0x2ff226f4, pObjIF = 0x300039d0), line 186 in
"rsyslog.c"
InitGlobalClasses(), line 2247 in "syslogd.c"
legacyOptsParseTCP(ch = '^P', arg = warning: Unable to access address
0x1 from core
(invalid char ptr (0x00000001))), line 2610 in "syslogd.c"
main(argc = 5, argv = 0x2ff2293c), line 2907 in "syslogd.c"

Here is the rsyslog script that executes:

#!/usr/bin/ksh
#
# rsyslog:     Startup script for rsyslogd.
#
# Description: Rsyslogd is an enhanced multi-threaded syslogd supporting,
#              among others, MySQL, syslog/tcp, RFC 3195, permitted
#              sender lists, filtering on any message part, and fine
#              grain output format control.
#
# Mar 09, 2011, Michael Perzl (michael at perzl.org)
#

NAME=rsyslog
PROG=${NAME}d

RSYSLOGD_BIN=/opt/freeware/sbin/${PROG}

PIDFILE=/var/run/${PROG}.pid


# define some generic commands
AWK=/usr/bin/awk
CAT=/usr/bin/cat
ECHO=/usr/bin/echo
GREP=/usr/bin/grep
KILL=/usr/bin/kill
MKDIR=/usr/bin/mkdir
PRINTF=/usr/bin/printf
PS=/usr/bin/ps
RM=/usr/bin/rm


# check for missing binaries (stale symlinks should not happen)
test -x ${RSYSLOGD_BIN} ||
    {
      $ECHO "${RSYSLOGD_BIN} not installed"
      if [ "$1" = "stop" ] ; then
          exit 0
      else
          exit 5
      fi
    }


# source config
if [ -f /etc/sysconfig/${NAME} ] ; then
    . /etc/sysconfig/${NAME}
fi


case "$1" in
    start)
        if [ -r ${PIDFILE} ]; then
            pid=`$CAT ${PIDFILE}`
            if [ "`$PS -ef | $GREP -v grep | $GREP ${PROG} | $GREP
${pid} | $AWK '{ print $2 }'`" = "${pid}" ] ; then
                $ECHO "Rsyslogd is already running with PID ${pid}."
                exit 1
            else
                $RM -f ${PIDFILE}
            fi
        fi
        $PRINTF "Starting Rsyslogd... "

        ## start daemon and write PID to file ${PIDFILE}
        $MKDIR -p /var/run
        ${RSYSLOGD_BIN} -i "${PIDFILE}" ${SYSLOGD_OPTIONS}
        $ECHO "done."
        ;;
    stop)
        $PRINTF "Stopping Rsyslogd... "
        ## stop daemon
        if [ -r ${PIDFILE} ]; then
            $KILL -TERM `$CAT ${PIDFILE}`
            $RM -f ${PIDFILE}
        fi
        $ECHO "done."
        ;;
    status)
        if [ -r ${PIDFILE} ]; then
            pid=`$CAT ${PIDFILE}`
            if [ `$PS -ef | $GREP -v grep | $GREP ${PROG} | $GREP ${pid}
| $AWK '{ print $2 }'` = ${pid} ] ; then
                $ECHO "Rsyslogd is running with PID ${pid}."
            fi
        else
            $ECHO "Rsyslogd is not running."
        fi
        ;;
    condrestart)
        if [ -r ${PIDFILE} ]; then
            pid=`$CAT ${PIDFILE}`
            if [ `$PS -ef | $GREP -v grep | $GREP ${PROG} | $GREP ${pid}
| $AWK '{ print $2 }'` = ${pid} ] ; then
                $0 stop
                $0 start
            fi
        fi
        ;;
    restart)
        ## Stop the service and regardless of whether it was
        ## running or not, start it again.
        $0 stop
        $0 start
        ;;
    *)
        $ECHO "Usage: $0 {start|stop|status|condrestart|?restart}"
        exit 1
        ;;
esac


The debug log seems to be complaining about a string length.  Any help
assisting in correcting this problem would be greatly appreciated.

Tom


From rpkelly22 at gmail.com  Mon Nov 21 20:38:54 2011
From: rpkelly22 at gmail.com (Ryan Kelly)
Date: Mon, 21 Nov 2011 14:38:54 -0500
Subject: [rsyslog] rsyslog on AIX
In-Reply-To: <4ECAA693.7090402@noaa.gov>
References: <4ECAA693.7090402@noaa.gov>
Message-ID: <20111121193854.GA19775@llserver.lakeliving.com>

> # Temporary log server IP address
> *.* @@999.999.999.999:5140
I would guess this address is part of the problem. Could you try with
a valid address?

-Ryan

From tom.teal at noaa.gov  Mon Nov 21 20:44:08 2011
From: tom.teal at noaa.gov (Tom Teal)
Date: Mon, 21 Nov 2011 14:44:08 -0500
Subject: [rsyslog] rsyslog on AIX
In-Reply-To: <20111121193854.GA19775@llserver.lakeliving.com>
References: <4ECAA693.7090402@noaa.gov>
	<20111121193854.GA19775@llserver.lakeliving.com>
Message-ID: <4ECAAA08.7020003@noaa.gov>

Ryan,

I only masked the ip address for the email but the rsyslog.conf does
have a valid address. 

Tom



On 11/21/2011 2:38 PM, Ryan Kelly wrote:
>> # Temporary log server IP address
>> *.* @@999.999.999.999:5140
> I would guess this address is part of the problem. Could you try with
> a valid address?
>
> -Ryan
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 
Tom Teal
Systems Administrator
NOAA's National Climatic Data Center
Veach-Baley Federal Building
151 Patton Avenue, Suite 320
Asheville, NC 28801
STG Inc. - Government Contractor
Tel: 828-271-4436
Fax: 828-271-4009
tom.teal at noaa.gov


From rgerhards at hq.adiscon.com  Tue Nov 22 16:00:45 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 22 Nov 2011 16:00:45 +0100
Subject: [rsyslog] rsyslog and journald
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814DF@GRFEXC.intern.adiscon.com>

Hi all,

I had a couple of questions on rsyslog and journald. Rather than answering
all individually, I a) found out what journald is and b) blogged about my
first impression on the relationship:

http://blog.gerhards.net/2011/11/journald-and-rsyslog.html

I hope this is useful.
Rainer


From brian_dorling at t-online.de  Wed Nov 23 09:16:42 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 08:16:42 +0000 (UTC)
Subject: [rsyslog] Ominous "Called LogError,
	msg: Could not create tcp listener, ignoring port 10514" message
Message-ID: 

Hi,
saw all the infos about problems with ports below 1024, so chnaged my remote
logging to use 10514 and am getting the same error. According to all the
discussions I read, port 10514 should work with dropped permissions.
USing rsyslog 4.6.4, supplied with Ubuntu 11.04.
Didn't find anything detailing exactly which version solved the dropped
permissions problem.

Has it been solved now? 

ANy ideas why I should get this error on a port > 1024?

Thanks in advance Brian 


From brian_dorling at t-online.de  Wed Nov 23 12:42:07 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 11:42:07 +0000 (UTC)
Subject: [rsyslog]
	=?utf-8?q?Ominous_=22Called_LogError=2C=09msg=3A_Could_?=
	=?utf-8?q?not_create_tcp_listener=2C_ignoring_port_10514=22_messag?=
	=?utf-8?q?e?=
References: 
Message-ID: 

Brian Dorling  t-online.de> writes:

> 
> Hi,
> saw all the infos about problems with ports below 1024, so chnaged my remote
> logging to use 10514 and am getting the same error. According to all the
> discussions I read, port 10514 should work with dropped permissions.
> USing rsyslog 4.6.4, supplied with Ubuntu 11.04.
> Didn't find anything detailing exactly which version solved the dropped
> permissions problem.
> 
> Has it been solved now? 
> 
> ANy ideas why I should get this error on a port > 1024?
> 
> Thanks in advance Brian 
> 
> 
Sorry, running this on Debian 6.0 with Kernel 2.6.32-5-686. Not Ubuntu.
Strange is that this has been working for months, and now it stopped.
Also, went to port 10514 after getting the same error on port 514.

Cheers Brian




From rgerhards at hq.adiscon.com  Wed Nov 23 12:43:47 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 23 Nov 2011 12:43:47 +0100
Subject: [rsyslog] Ominous "Called LogError,
	msg: Could not create tcp listener, ignoring port 10514" message
In-Reply-To: 
References: 
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>

Probably a debug log of the startup phase would be useful. Does it work
without dropping privileges?

rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Brian Dorling
> Sent: Wednesday, November 23, 2011 12:42 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog]Ominous "Called LogError, msg: Could not create tcp
> listener, ignoring port 10514" message
> 
> Brian Dorling  t-online.de> writes:
> 
> >
> > Hi,
> > saw all the infos about problems with ports below 1024, so chnaged my
> > remote logging to use 10514 and am getting the same error. According
> > to all the discussions I read, port 10514 should work with dropped
> permissions.
> > USing rsyslog 4.6.4, supplied with Ubuntu 11.04.
> > Didn't find anything detailing exactly which version solved the
> > dropped permissions problem.
> >
> > Has it been solved now?
> >
> > ANy ideas why I should get this error on a port > 1024?
> >
> > Thanks in advance Brian
> >
> >
> Sorry, running this on Debian 6.0 with Kernel 2.6.32-5-686. Not Ubuntu.
> Strange is that this has been working for months, and now it stopped.
> Also, went to port 10514 after getting the same error on port 514.
> 
> Cheers Brian
> 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From brian_dorling at t-online.de  Wed Nov 23 12:59:41 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 11:59:41 +0000 (UTC)
Subject: [rsyslog] Ominous
References: 
	
	<9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>
Message-ID: 

Rainer Gerhards  hq.adiscon.com> writes:

> 
> Probably a debug log of the startup phase would be useful. Does it work
> without dropping privileges?
> 
> rainer

Rainer, I just sent the log and will try without dropping privileges and report
back on that.
Cheers




From brian_dorling at t-online.de  Wed Nov 23 13:17:07 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 12:17:07 +0000 (UTC)
Subject: [rsyslog]
	=?utf-8?q?Ominous_=22Called_LogError=2C=09msg=3A_Could_?=
	=?utf-8?q?not_create_tcp_listener=2C_ignoring_port_10514=22_messag?=
	=?utf-8?q?e?=
References: 
	
	<9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>
Message-ID: 

Rainer Gerhards  hq.adiscon.com> writes:

> 
> Probably a debug log of the startup phase would be useful. Does it work
> without dropping privileges?
> 
> rainer
On port 10514 it does not work, but on port 514 it seesm to work Ok as root.

Not 100% sure if port 10514 is really free:

zotac-sda1:root at zotac: netstat -nap | grep 514
tcp        0      0 0.0.0.0:10514           0.0.0.0:*               LISTEN     
16615/sh        
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN     
18750/rsyslogd  
tcp6       0      0 :::10514                :::*                    LISTEN     
16615/sh        
tcp6       0      0 :::514                  :::*                    LISTEN     
18750/rsyslogd  
udp        0      0 0.0.0.0:514             0.0.0.0:*                          
18750/rsyslogd  
udp    70224      0 0.0.0.0:10514           0.0.0.0:*                          
16615/sh        
udp6       0      0 :::514                  :::*                               
18750/rsyslogd  
udp6       0      0 :::10514                :::*                               
16615/sh    

Still, 514 as non-root does not work  when permissions are dropped.
I guess port 10514 is a red-herring maybe?

Do you want a startup log for port 514 as non-root? 

Cheers




From rgerhards at hq.adiscon.com  Wed Nov 23 14:26:31 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 23 Nov 2011 14:26:31 +0100
Subject: [rsyslog] Ominous "Called LogError,
	msg: Could not create tcp listener, ignoring port 10514" message
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814EF@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Brian Dorling
> Sent: Wednesday, November 23, 2011 1:17 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog]Ominous "Called LogError, msg: Could not create tcp
> listener, ignoring port 10514" message
> 
> Rainer Gerhards  hq.adiscon.com> writes:
> 
> >
> > Probably a debug log of the startup phase would be useful. Does it
> > work without dropping privileges?
> >
> > rainer
> On port 10514 it does not work, but on port 514 it seesm to work Ok as
root.
> 
> Not 100% sure if port 10514 is really free:

In the debug log, I see error 98, which I think usually is "port is already
in use".

> 
> zotac-sda1:root at zotac: netstat -nap | grep 514
> tcp        0      0 0.0.0.0:10514           0.0.0.0:*               LISTEN
> 16615/sh
> tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN
> 18750/rsyslogd
> tcp6       0      0 :::10514                :::*                    LISTEN
> 16615/sh
> tcp6       0      0 :::514                  :::*                    LISTEN
> 18750/rsyslogd
> udp        0      0 0.0.0.0:514             0.0.0.0:*
> 18750/rsyslogd
> udp    70224      0 0.0.0.0:10514           0.0.0.0:*
> 16615/sh
> udp6       0      0 :::514                  :::*
> 18750/rsyslogd
> udp6       0      0 :::10514                :::*
> 16615/sh
> 
> Still, 514 as non-root does not work  when permissions are dropped.
> I guess port 10514 is a red-herring maybe?
> 
> Do you want a startup log for port 514 as non-root?

Yes, please. Note that versions prior to 6 did not have really proper priv
drop code, but what you try should work....

Rainer

> Cheers
> 
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From brian_dorling at t-online.de  Wed Nov 23 15:05:45 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 14:05:45 +0000 (UTC)
Subject: [rsyslog]
	=?utf-8?q?Ominous_=22Called_LogError=2C=09msg=3A_Could_?=
	=?utf-8?q?not_create_tcp_listener=2C_ignoring_port_10514=22_messag?=
	=?utf-8?q?e?=
References: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA72814EF@GRFEXC.intern.adiscon.com>
Message-ID: 

Rainer Gerhards  hq.adiscon.com> writes:
> 
> In the debug log, I see error 98, which I think usually is "port is already
> in use".
...
> > 
> > Do you want a startup log for port 514 as non-root?
> 
> Yes, please. Note that versions prior to 6 did not have really proper priv
> drop code, but what you try should work....

Rainer,
now ist working again as non root. I think I know what the problem was:

Looking in top i saw a process called "sh /usr/sbin/mailtext-nosyslog.sh"
hanging around.
So I killed it and now rsyslogd starts OK with port 514 as non-root. 

I guess this is the culprit:

if $msg contains_i 'WAN is up' and $source contains_i 'buffalo' then 
^/usr/sbin/mailtext-nosyslog.sh;WAN

So, is this synchronous? Is there a better way to call my own scripts?

Strange thing was that I did reboot yesterday evening, so the problem seems to
have come back
sometime yesterday evening.

Cheers Brian


From brian_dorling at t-online.de  Wed Nov 23 16:13:13 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 15:13:13 +0000 (UTC)
Subject: [rsyslog] =?utf-8?q?Ominous?=
References: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA72814EF@GRFEXC.intern.adiscon.com>
	
Message-ID: 

Brian Dorling  t-online.de> writes:


> I guess this is the culprit:
> 
> if $msg contains_i 'WAN is up' and $source contains_i 'buffalo' then 
> ^/usr/sbin/mailtext-nosyslog.sh;WAN
> 
Not, quite. I changed the script to use another email sender last week. It
expects two parms.
1 = Subject, 2 = body. If 2 is empty it should EOT itself after 60 seconds.
Seems like
it does, sometimes. As rsyslog calls the script with just 1 parm (from the 
template) then that prgram would just hang. And that seems to have hung 
rsyslog. 
Phew, at least that is clear now! 

Question. Any way to pass 2 string parms out of a template?

Thanks for the help.

Cheers Brian




From rgerhards at hq.adiscon.com  Wed Nov 23 16:30:21 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 23 Nov 2011 16:30:21 +0100
Subject: [rsyslog] Ominous
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72814EF@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814F3@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Brian Dorling
> Sent: Wednesday, November 23, 2011 4:13 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] Ominous
> 
> Brian Dorling  t-online.de> writes:
> 
> 
> > I guess this is the culprit:
> >
> > if $msg contains_i 'WAN is up' and $source contains_i 'buffalo' then
> > ^/usr/sbin/mailtext-nosyslog.sh;WAN
> >
> Not, quite. I changed the script to use another email sender last week. It
> expects two parms.
> 1 = Subject, 2 = body. If 2 is empty it should EOT itself after 60 seconds.
> Seems like
> it does, sometimes. As rsyslog calls the script with just 1 parm (from the
> template) then that prgram would just hang. And that seems to have hung
> rsyslog.
> Phew, at least that is clear now!
> 
> Question. Any way to pass 2 string parms out of a template?
> 
> Thanks for the help.

Actually, the exec program action is a real quick and dirty hack. I suggest
to have a look at omprog :)

Rainer

From brian_dorling at t-online.de  Wed Nov 23 18:06:35 2011
From: brian_dorling at t-online.de (Brian Dorling)
Date: Wed, 23 Nov 2011 17:06:35 +0000 (UTC)
Subject: [rsyslog] Ominous
References: <9B6E2A8877C38245BFB15CC491A11DA72814ED@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA72814EF@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA72814F3@GRFEXC.intern.adiscon.com>
Message-ID: 

Rainer Gerhards  hq.adiscon.com> writes:


> > 
> > Question. Any way to pass 2 string parms out of a template?

> Actually, the exec program action is a real quick and dirty hack. I suggest
> to have a look at omprog :)

OK, but I didn't see a way to pass two strings. I do appreciate that omprog
may be safer for the resiliency of rsyslog though and I'll try it out.

Cheers Brian




From rgerhards at hq.adiscon.com  Wed Nov 23 18:41:44 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 23 Nov 2011 18:41:44 +0100
Subject: [rsyslog] FYI: funding rsyslog development
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>

Hi all,

I am probably forced by circumstances to make some changes in the current
licensing of rsyslog. I have elaborated in this blog post:

http://blog.gerhards.net/2011/11/funding-rsyslog-development.html

Feedback is of course appreciated (but please be gentle with me ;)).

Rainer

From a.chapellon at horoa.net  Thu Nov 24 15:36:35 2011
From: a.chapellon at horoa.net (Alexandre Chapellon)
Date: Thu, 24 Nov 2011 15:36:35 +0100
Subject: [rsyslog] advices about SQL insertions
Message-ID: <4ECE5673.8060205@horoa.net>

Hello,

I have syslog messages I want to put in a database. I don't want to 
store raw content of the message but only valuable data found in the 
message.
To extract the data I wish to insert, I can eihter use field and regex 
property replacer features of rsyslog or use SQL funstions like (eg) 
trim, substring etc... Both works.
What would you recommend to ensure the lowest impact on the ressources 
of the server?

Regards.
-- 


Alexandre Chapellon

Ing?nierie des syst?mes open sources et r?seaux.
Follow me on twitter: @alxgomz 


From rgerhards at hq.adiscon.com  Thu Nov 24 15:48:01 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 24 Nov 2011 15:48:01 +0100
Subject: [rsyslog] advices about SQL insertions
In-Reply-To: <4ECE5673.8060205@horoa.net>
References: <4ECE5673.8060205@horoa.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281503@GRFEXC.intern.adiscon.com>

Tough question... it depends. Field-based extraction is *Very* fast and hard
to beat. So if you can, go with that. If you can re-use Rexpes inside the SQL
engine, it is probably better to do it there, as rsyslog templates can not
carry submatches across different regexes. Actually, I'd benchmark the whole
thing if I had the task...

Rainer


> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
> Sent: Thursday, November 24, 2011 3:37 PM
> To: rsyslog-users
> Subject: [rsyslog] advices about SQL insertions
> 
> Hello,
> 
> I have syslog messages I want to put in a database. I don't want to
> store raw content of the message but only valuable data found in the
> message.
> To extract the data I wish to insert, I can eihter use field and regex
> property replacer features of rsyslog or use SQL funstions like (eg)
> trim, substring etc... Both works.
> What would you recommend to ensure the lowest impact on the ressources
> of the server?
> 
> Regards.
> --
> 
> 
> Alexandre Chapellon
> 
> Ing?nierie des syst?mes open sources et r?seaux.
> Follow me on twitter: @alxgomz 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From a.chapellon at horoa.net  Thu Nov 24 15:56:48 2011
From: a.chapellon at horoa.net (Alexandre Chapellon)
Date: Thu, 24 Nov 2011 15:56:48 +0100
Subject: [rsyslog] advices about SQL insertions
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281503@GRFEXC.intern.adiscon.com>
References: <4ECE5673.8060205@horoa.net>
	<9B6E2A8877C38245BFB15CC491A11DA7281503@GRFEXC.intern.adiscon.com>
Message-ID: <4ECE5B30.6040103@horoa.net>



Le 24/11/2011 15:48, Rainer Gerhards a ?crit :
> Tough question... it depends. Field-based extraction is *Very* fast and hard
> to beat. So if you can, go with that. If you can re-use Rexpes inside the SQL
> engine, it is probably better to do it there, as rsyslog templates can not
> carry submatches across different regexes.
More specificly, if I use the same regex several times in a template (to 
extract several submatch), is the regex evaluated each time or only once?

As an example:

/$template PF_SPF_PG, "INSERT INTO mailsecurity \
         (datelog, ipmx, msgsrv, hostip, exemptonspf, reason, mailfrom, 
rcptto, hostnameclient) \
         VALUES \
         ('%timegenerated:::date-rfc3339%', \
         '%hostname%', \
         'SPF', \
         btrim('%msg:F,32:13%', '[]')::inet, \
         trim(trailing ';' from '%msg:F,59:3%'), \
         '%msg:F,59:3%', \
         '%msg:R,ERE,1,BLANK:.*; from=<(.*)> to=<(.*)>.* 
helo=<(.*)>--end%', \
         '%msg:R,ERE,2,BLANK:.*; from=<(.*)> to=<(.*)>.* 
helo=<(.*)>--end%', \
         '%msg:R,ERE,3,BLANK:.*; from=<(.*)> to=<(.*)>.* 
helo=<(.*)>--end%')",stdsql/

>   Actually, I'd benchmark the whole
> thing if I had the task...
Rainer
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
>> Sent: Thursday, November 24, 2011 3:37 PM
>> To: rsyslog-users
>> Subject: [rsyslog] advices about SQL insertions
>>
>> Hello,
>>
>> I have syslog messages I want to put in a database. I don't want to
>> store raw content of the message but only valuable data found in the
>> message.
>> To extract the data I wish to insert, I can eihter use field and regex
>> property replacer features of rsyslog or use SQL funstions like (eg)
>> trim, substring etc... Both works.
>> What would you recommend to ensure the lowest impact on the ressources
>> of the server?
>>
>> Regards.
>> --
>> 
>>
>> Alexandre Chapellon
>>
>> Ing?nierie des syst?mes open sources et r?seaux.
>> Follow me on twitter: @alxgomz
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 


Alexandre Chapellon

Ing?nierie des syst?mes open sources et r?seaux.
Follow me on twitter: @alxgomz 


From rgerhards at hq.adiscon.com  Thu Nov 24 15:58:20 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Thu, 24 Nov 2011 15:58:20 +0100
Subject: [rsyslog] advices about SQL insertions
In-Reply-To: <4ECE5B30.6040103@horoa.net>
References: <4ECE5673.8060205@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281503@GRFEXC.intern.adiscon.com>
	<4ECE5B30.6040103@horoa.net>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281504@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
> Sent: Thursday, November 24, 2011 3:57 PM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] advices about SQL insertions
> 
> 
> 
> Le 24/11/2011 15:48, Rainer Gerhards a ?crit :
> > Tough question... it depends. Field-based extraction is *Very* fast
> and hard
> > to beat. So if you can, go with that. If you can re-use Rexpes inside
> the SQL
> > engine, it is probably better to do it there, as rsyslog templates
> can not
> > carry submatches across different regexes.
> More specificly, if I use the same regex several times in a template
> (to
> extract several submatch), is the regex evaluated each time or only
> once?

That's far easier to answer: multiple times.

rainer
> 
> As an example:
> 
> /$template PF_SPF_PG, "INSERT INTO mailsecurity \
>          (datelog, ipmx, msgsrv, hostip, exemptonspf, reason, mailfrom,
> rcptto, hostnameclient) \
>          VALUES \
>          ('%timegenerated:::date-rfc3339%', \
>          '%hostname%', \
>          'SPF', \
>          btrim('%msg:F,32:13%', '[]')::inet, \
>          trim(trailing ';' from '%msg:F,59:3%'), \
>          '%msg:F,59:3%', \
>          '%msg:R,ERE,1,BLANK:.*; from=<(.*)> to=<(.*)>.*
> helo=<(.*)>--end%', \
>          '%msg:R,ERE,2,BLANK:.*; from=<(.*)> to=<(.*)>.*
> helo=<(.*)>--end%', \
>          '%msg:R,ERE,3,BLANK:.*; from=<(.*)> to=<(.*)>.*
> helo=<(.*)>--end%')",stdsql/
> 
> >   Actually, I'd benchmark the whole
> > thing if I had the task...
> Rainer
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
> >> Sent: Thursday, November 24, 2011 3:37 PM
> >> To: rsyslog-users
> >> Subject: [rsyslog] advices about SQL insertions
> >>
> >> Hello,
> >>
> >> I have syslog messages I want to put in a database. I don't want to
> >> store raw content of the message but only valuable data found in the
> >> message.
> >> To extract the data I wish to insert, I can eihter use field and
> regex
> >> property replacer features of rsyslog or use SQL funstions like (eg)
> >> trim, substring etc... Both works.
> >> What would you recommend to ensure the lowest impact on the
> ressources
> >> of the server?
> >>
> >> Regards.
> >> --
> >> 
> >>
> >> Alexandre Chapellon
> >>
> >> Ing?nierie des syst?mes open sources et r?seaux.
> >> Follow me on twitter: @alxgomz
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> 
> --
> 
> 
> Alexandre Chapellon
> 
> Ing?nierie des syst?mes open sources et r?seaux.
> Follow me on twitter: @alxgomz 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From a.chapellon at horoa.net  Thu Nov 24 16:03:10 2011
From: a.chapellon at horoa.net (Alexandre Chapellon)
Date: Thu, 24 Nov 2011 16:03:10 +0100
Subject: [rsyslog] advices about SQL insertions
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281504@GRFEXC.intern.adiscon.com>
References: <4ECE5673.8060205@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281503@GRFEXC.intern.adiscon.com>
	<4ECE5B30.6040103@horoa.net>
	<9B6E2A8877C38245BFB15CC491A11DA7281504@GRFEXC.intern.adiscon.com>
Message-ID: <4ECE5CAE.9010107@horoa.net>

OK thank you... maybe recognition of similar regex, to avoid multiple 
evaluation of the same thing, would be nice.
Does anybody else would like that feature?

Le 24/11/2011 15:58, Rainer Gerhards a ?crit :
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
>> Sent: Thursday, November 24, 2011 3:57 PM
>> To: rsyslog at lists.adiscon.com
>> Subject: Re: [rsyslog] advices about SQL insertions
>>
>>
>>
>> Le 24/11/2011 15:48, Rainer Gerhards a ?crit :
>>> Tough question... it depends. Field-based extraction is *Very* fast
>> and hard
>>> to beat. So if you can, go with that. If you can re-use Rexpes inside
>> the SQL
>>> engine, it is probably better to do it there, as rsyslog templates
>> can not
>>> carry submatches across different regexes.
>> More specificly, if I use the same regex several times in a template
>> (to
>> extract several submatch), is the regex evaluated each time or only
>> once?
> That's far easier to answer: multiple times.
>
> rainer
>> As an example:
>>
>> /$template PF_SPF_PG, "INSERT INTO mailsecurity \
>>           (datelog, ipmx, msgsrv, hostip, exemptonspf, reason, mailfrom,
>> rcptto, hostnameclient) \
>>           VALUES \
>>           ('%timegenerated:::date-rfc3339%', \
>>           '%hostname%', \
>>           'SPF', \
>>           btrim('%msg:F,32:13%', '[]')::inet, \
>>           trim(trailing ';' from '%msg:F,59:3%'), \
>>           '%msg:F,59:3%', \
>>           '%msg:R,ERE,1,BLANK:.*; from=<(.*)>  to=<(.*)>.*
>> helo=<(.*)>--end%', \
>>           '%msg:R,ERE,2,BLANK:.*; from=<(.*)>  to=<(.*)>.*
>> helo=<(.*)>--end%', \
>>           '%msg:R,ERE,3,BLANK:.*; from=<(.*)>  to=<(.*)>.*
>> helo=<(.*)>--end%')",stdsql/
>>
>>>    Actually, I'd benchmark the whole
>>> thing if I had the task...
>> Rainer
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Alexandre Chapellon
>>>> Sent: Thursday, November 24, 2011 3:37 PM
>>>> To: rsyslog-users
>>>> Subject: [rsyslog] advices about SQL insertions
>>>>
>>>> Hello,
>>>>
>>>> I have syslog messages I want to put in a database. I don't want to
>>>> store raw content of the message but only valuable data found in the
>>>> message.
>>>> To extract the data I wish to insert, I can eihter use field and
>> regex
>>>> property replacer features of rsyslog or use SQL funstions like (eg)
>>>> trim, substring etc... Both works.
>>>> What would you recommend to ensure the lowest impact on the
>> ressources
>>>> of the server?
>>>>
>>>> Regards.
>>>> --
>>>> 
>>>>
>>>> Alexandre Chapellon
>>>>
>>>> Ing?nierie des syst?mes open sources et r?seaux.
>>>> Follow me on twitter: @alxgomz
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>> --
>> 
>>
>> Alexandre Chapellon
>>
>> Ing?nierie des syst?mes open sources et r?seaux.
>> Follow me on twitter: @alxgomz
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 


Alexandre Chapellon

Ing?nierie des syst?mes open sources et r?seaux.
Follow me on twitter: @alxgomz 


From a.chapellon at horoa.net  Thu Nov 24 18:53:15 2011
From: a.chapellon at horoa.net (Alexandre Chapellon)
Date: Thu, 24 Nov 2011 18:53:15 +0100
Subject: [rsyslog] timereported timezone?
Message-ID: <4ECE848B.6070203@horoa.net>

I quickly read the rfc3164 and found this:

"

...

    For implementers that do choose to construct syslog messages with the
    RECOMMENDED format, the following guidance is offered.

          If the originally formed message has a TIMESTAMP in the HEADER
          part, then it SHOULD be the local time of the device within its
          timezone...

"

So I guess the timezone of the device is not store in the syslog message 
by itself. Right?

Is there a way to include the local timezonein the header? or at least 
in the message? (I would really prefer in the header)

regards.
-- 


Alexandre Chapellon

Ing?nierie des syst?mes open sources et r?seaux.
Follow me on twitter: @alxgomz 


From david at lang.hm  Fri Nov 25 04:57:08 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 24 Nov 2011 19:57:08 -0800 (PST)
Subject: [rsyslog] timereported timezone?
In-Reply-To: <4ECE848B.6070203@horoa.net>
References: <4ECE848B.6070203@horoa.net>
Message-ID: 

On Thu, 24 Nov 2011, Alexandre Chapellon wrote:

> I quickly read the rfc3164 and found this:
>
> "
>
> ...
>
>   For implementers that do choose to construct syslog messages with the
>   RECOMMENDED format, the following guidance is offered.
>
>         If the originally formed message has a TIMESTAMP in the HEADER
>         part, then it SHOULD be the local time of the device within its
>         timezone...
>
> "
>
> So I guess the timezone of the device is not store in the syslog message by 
> itself. Right?

correct.

> Is there a way to include the local timezonein the header? or at least in the 
> message? (I would really prefer in the header)

you can't do it with the RFC3164 format, I belive that the latest RFC does 
have a timestamp format that includes the timezone, but I don't think that 
it's a good idea to use local time in any case.

I think you are far better off running your servers (and the timestamps on 
the logs) on UTC. This not only avoids the problems of "where is this 
server located, so what time zone is it in", but it also doesn't have 
daylight savings time changes (with all the related problems of jobs 
running multiple times or not at all)

When my company first setup servers across the country, I raised a fuss 
against running them on the local timezone and instead we ran all servers 
on the timezone of our first datacenter. 13 years later the company still 
hasn't changed this and twice a year there is a special maintinance to 
make sure that all products that have been deployed work properly after 
the timezone changes. Several years ago the Security systems (which I 
moved on to manage) all got changed to run UTC, initially as an accident 
(a new OS upgrade didn't set the timezone), and I have seen the 
side-by-side comparison, it works _MUCH_ better to have the systems on 
UTC. I've found that the problem of 'midnight' being either 4pm or 5pm 
pacific time just really doesn't matter.

David Lang

From david at lang.hm  Fri Nov 25 05:13:00 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 24 Nov 2011 20:13:00 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>
Message-ID: 

I'm disappointed to see this, please keep us as informed as you can on 
these discussions.

David Lang

On Wed, 23 Nov 2011, Rainer Gerhards wrote:

> Hi all,
>
> I am probably forced by circumstances to make some changes in the current
> licensing of rsyslog. I have elaborated in this blog post:
>
> http://blog.gerhards.net/2011/11/funding-rsyslog-development.html
>
> Feedback is of course appreciated (but please be gentle with me ;)).
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>

From rgerhards at hq.adiscon.com  Fri Nov 25 07:21:37 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 07:21:37 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: 
References: <4ECE848B.6070203@horoa.net>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>



> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 4:57 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] timereported timezone?
> 
> On Thu, 24 Nov 2011, Alexandre Chapellon wrote:
> 
> > I quickly read the rfc3164 and found this:
> >
> > "
> >
> > ...
> >
> >   For implementers that do choose to construct syslog messages with
> the
> >   RECOMMENDED format, the following guidance is offered.
> >
> >         If the originally formed message has a TIMESTAMP in the
> HEADER
> >         part, then it SHOULD be the local time of the device within
> its
> >         timezone...
> >
> > "
> >
> > So I guess the timezone of the device is not store in the syslog
> message by
> > itself. Right?
> 
> correct.
> 
> > Is there a way to include the local timezonein the header? or at
> least in the
> > message? (I would really prefer in the header)
> 
> you can't do it with the RFC3164 format, I belive that the latest RFC
> does
> have a timestamp format that includes the timezone, but I don't think
> that
> it's a good idea to use local time in any case.

RFC5424 provides high-precision timestamps with timezone information. Rsyslog
does so for many years. It is a mayor frustration for me that all distros
turn off that feature (which is enabled by default).

But of course that doesn't help if the original sender does not provide TZ
information.
rainer

From rgerhards at hq.adiscon.com  Fri Nov 25 07:46:51 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 07:46:51 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 5:13 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> I'm disappointed to see this, please keep us as informed as you can on
> these discussions.

Will definitely do. I have fought hard to avoid this, but as it currently
looks I am running out of alternatives and arguments. The arrival of the
journald idea probably gave it the final push but the core problem is that
very, very few supported development (and even more thanks to those who
did!). So I have to say that I myself become convinced that a no-nonsense
dual licensing model (like the original MySQL, for example) can be beneficial
to the community at large. There *are* some bigger entities out there that
would potentially be willing to provide notable funding, but not under the
terms of GPL. Why not make use of that? If we do not provide them with the
ability, they simply go for syslog-ng. I have spent many years of my life
with this logging thing. I would like to see it continue to grow.

Rainer  

> 
> David Lang
> 
> On Wed, 23 Nov 2011, Rainer Gerhards wrote:
> 
> > Hi all,
> >
> > I am probably forced by circumstances to make some changes in the
> current
> > licensing of rsyslog. I have elaborated in this blog post:
> >
> > http://blog.gerhards.net/2011/11/funding-rsyslog-development.html
> >
> > Feedback is of course appreciated (but please be gentle with me ;)).
> >
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From david at lang.hm  Fri Nov 25 08:27:37 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 24 Nov 2011 23:27:37 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: david at lang.hm
>>
>> I'm disappointed to see this, please keep us as informed as you can on
>> these discussions.
>
> Will definitely do. I have fought hard to avoid this, but as it currently
> looks I am running out of alternatives and arguments. The arrival of the
> journald idea probably gave it the final push but the core problem is that
> very, very few supported development (and even more thanks to those who
> did!). So I have to say that I myself become convinced that a no-nonsense
> dual licensing model (like the original MySQL, for example) can be beneficial
> to the community at large. There *are* some bigger entities out there that
> would potentially be willing to provide notable funding, but not under the
> terms of GPL. Why not make use of that? If we do not provide them with the
> ability, they simply go for syslog-ng. I have spent many years of my life
> with this logging thing. I would like to see it continue to grow.

the key question revolves around how the dual licensing would work

MySQL sold GPL exemptions so that companies could use their code in 
non-GPL proprietary applicaions (although some of it was based on a 
questionable interpretation of the GPL)

syslog-ng uses the 'open sore' approach where some features are not made 
available in the free version, only in the proprietary version.

how are you thinking that this would work for rsyslog?



There's also the issue of sponsered work that was done with the belief 
that the result would be released under the GPL. Some of that work is 
fairly core. This would be a potential problem for the MySQL approach.

David Lang

> Rainer
>
>>
>> David Lang
>>
>> On Wed, 23 Nov 2011, Rainer Gerhards wrote:
>>
>>> Hi all,
>>>
>>> I am probably forced by circumstances to make some changes in the
>> current
>>> licensing of rsyslog. I have elaborated in this blog post:
>>>
>>> http://blog.gerhards.net/2011/11/funding-rsyslog-development.html
>>>
>>> Feedback is of course appreciated (but please be gentle with me ;)).
>>>
>>> Rainer
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>

From david at lang.hm  Fri Nov 25 08:57:28 2011
From: david at lang.hm (david at lang.hm)
Date: Thu, 24 Nov 2011 23:57:28 -0800 (PST)
Subject: [rsyslog] timereported timezone?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
References: <4ECE848B.6070203@horoa.net>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

>> you can't do it with the RFC3164 format, I belive that the latest RFC
>> does
>> have a timestamp format that includes the timezone, but I don't think
>> that
>> it's a good idea to use local time in any case.
>
> RFC5424 provides high-precision timestamps with timezone information. Rsyslog
> does so for many years. It is a mayor frustration for me that all distros
> turn off that feature (which is enabled by default).
>
> But of course that doesn't help if the original sender does not provide TZ
> information.

the thing is that in practice, the precision of the timestamp beyond the 
second really doesn't matter, except in rather unusual situations.

the TZ does matter, but that's easily solved by using UTC, and it avoids 
all the other TZ related problems.

people to who the server really needs to be set for the local TZ for the 
applicaiton to work are likely to also end up needing to run a farm of 
servers for clients in each TZ as they grow, and having seen this in 
operation, it causes problems so big that simple time errors in the log 
pale by comparison.

So all this makes high precision timestamps and TZ info in the log 
messages not be worth very much, especially compared to the ease of using 
legacy tools and scripts to deal with the log messages (not that it would 
be that hard to teach these tools and scripts about the new info, it's 
just not worh enough to bother)

David Lang

From rgerhards at hq.adiscon.com  Fri Nov 25 09:03:17 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 09:03:17 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 8:28 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> From: david at lang.hm
> >>
> >> I'm disappointed to see this, please keep us as informed as you can
> on
> >> these discussions.
> >
> > Will definitely do. I have fought hard to avoid this, but as it
> currently
> > looks I am running out of alternatives and arguments. The arrival of
> the
> > journald idea probably gave it the final push but the core problem is
> that
> > very, very few supported development (and even more thanks to those
> who
> > did!). So I have to say that I myself become convinced that a no-
> nonsense
> > dual licensing model (like the original MySQL, for example) can be
> beneficial
> > to the community at large. There *are* some bigger entities out there
> that
> > would potentially be willing to provide notable funding, but not
> under the
> > terms of GPL. Why not make use of that? If we do not provide them
> with the
> > ability, they simply go for syslog-ng. I have spent many years of my
> life
> > with this logging thing. I would like to see it continue to grow.
> 
> the key question revolves around how the dual licensing would work
> 
> MySQL sold GPL exemptions so that companies could use their code in
> non-GPL proprietary applicaions (although some of it was based on a
> questionable interpretation of the GPL)

>From my personal PoV it would probably go into that direction, though
probably with a different vehicle than the GPL exemption. The key point is
that we could probably aquire funding from vendors who would like to
integrate rsyslog into their products, but are not willing to do so under
GPL.

> syslog-ng uses the 'open sore' approach where some features are not
> made
> available in the free version, only in the proprietary version.

I like "open sore" more than "open core", it's up to the point :)

> how are you thinking that this would work for rsyslog?

This is definitely something possible (and also discussed). Maybe it was my
fault to prevent this in the past two or three years. We would probably be
better off if I had not rejected the idea. Quite honestly, this concerns me
much more than dual licensing, because it makes features unavailable to
regular users, whereas with dual licensing this does not (necessarily)
happen.

> There's also the issue of sponsered work that was done with the belief
> that the result would be released under the GPL. Some of that work is
> fairly core. This would be a potential problem for the MySQL approach.

Definitely, depending on the reaction this can be a big problem. I am right
now analyzing the code. I have found one large contribution that falls under
this category. There are a couple of smaller ones and there are other
contributions. One thing that my peers tell me is that it may even make sense
to rewrite the core (after all, there is room for improvement) than to ditch
the idea. Think it that way: development was almost exclusively paid for by
Adiscon. What problem would it be for Adiscon if we do a 4-month rewrite,
with real improvement, in the sake of settling these copyright issues? I have
to say this argument is hard to beat. Now think about "open sore": of course
Adiscon can do this new design and license *that part* under a commercial
license. That would be a loss for the community, while dual licensing would
do no harm (at least I think so).

But the issue is actually open again, and I appreciate all your feedback. The
problem is that Adiscon finally gets impatiently funding rsyslog, so I am
down to doing that in my spare time and on those rare occasions where someone
actually pays for that work (e.g. via support contracts). Adiscon doesn't
want to ditch rsyslog, that's for sure. But I am told that my development
time must be paid for, and that means I need to work on other projects
(that's exactly what folks like Red Hat said to their employees, so it is not
unreasonable). The situation has been worsened by the arrival of the journald
idea. It did very much harm my position. My peers now say "look, nobody is
interested in actually helping with the project. If they improve logging,
they prefer to do it via their own project". That seems to be a true
assumption (it would have been easy to do what journal does in rsyslog
context -- if I had have more time, I'd already written a similar log store).
Journald will also considerably shrink the ability to create other value out
of the rsyslog project by pushing it into a nice. The core question behind
this is if there is enough interest in the open source world to keep rsyslog
a totally free solution. The sore reality is that we have some great
contributors (including you!) but the vast majority of work is done by me.
Looking at other logging projects, my peers tell me, (almost?) all of them
have moved to some commercial funding. Logging seems to be far too less sexy
to attract the masses.

Again, feedback appreciated.

Rainer

> 
> David Lang
> 
> > Rainer
> >
> >>
> >> David Lang
> >>
> >> On Wed, 23 Nov 2011, Rainer Gerhards wrote:
> >>
> >>> Hi all,
> >>>
> >>> I am probably forced by circumstances to make some changes in the
> >> current
> >>> licensing of rsyslog. I have elaborated in this blog post:
> >>>
> >>> http://blog.gerhards.net/2011/11/funding-rsyslog-development.html
> >>>
> >>> Feedback is of course appreciated (but please be gentle with me
> ;)).
> >>>
> >>> Rainer
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From rgerhards at hq.adiscon.com  Fri Nov 25 09:04:56 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 09:04:56 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: 
References: <4ECE848B.6070203@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281515@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 8:57 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] timereported timezone?
> 
> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> 
> >> you can't do it with the RFC3164 format, I belive that the latest
> RFC
> >> does
> >> have a timestamp format that includes the timezone, but I don't
> think
> >> that
> >> it's a good idea to use local time in any case.
> >
> > RFC5424 provides high-precision timestamps with timezone information.
> Rsyslog
> > does so for many years. It is a mayor frustration for me that all
> distros
> > turn off that feature (which is enabled by default).
> >
> > But of course that doesn't help if the original sender does not
> provide TZ
> > information.
> 
> the thing is that in practice, the precision of the timestamp beyond
> the
> second really doesn't matter, except in rather unusual situations.
> 
> the TZ does matter, but that's easily solved by using UTC, and it
> avoids
> all the other TZ related problems.
> 
> people to who the server really needs to be set for the local TZ for
> the
> applicaiton to work are likely to also end up needing to run a farm of
> servers for clients in each TZ as they grow, and having seen this in
> operation, it causes problems so big that simple time errors in the log
> pale by comparison.
> 
> So all this makes high precision timestamps and TZ info in the log
> messages not be worth very much, especially compared to the ease of
> using
> legacy tools and scripts to deal with the log messages (not that it
> would
> be that hard to teach these tools and scripts about the new info, it's
> just not worh enough to bother)

A key point of the precise timestamp is that it provides a quite usable order
relation on a machine's logs.

Rainer

From david at lang.hm  Fri Nov 25 09:51:22 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 00:51:22 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

>> -----Original Message-----
>> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
>>
>>
>> MySQL sold GPL exemptions so that companies could use their code in
>> non-GPL proprietary applicaions (although some of it was based on a
>> questionable interpretation of the GPL)
>
> From my personal PoV it would probably go into that direction, though
> probably with a different vehicle than the GPL exemption. The key point is
> that we could probably aquire funding from vendors who would like to
> integrate rsyslog into their products, but are not willing to do so under
> GPL.

interesting, I didn' know that there were oppurtunities here.

>> syslog-ng uses the 'open sore' approach where some features are not
>> made
>> available in the free version, only in the proprietary version.
>
> I like "open sore" more than "open core", it's up to the point :)

I was trying to be polite ;-)

>> how are you thinking that this would work for rsyslog?
>
> This is definitely something possible (and also discussed). Maybe it was my
> fault to prevent this in the past two or three years. We would probably be
> better off if I had not rejected the idea. Quite honestly, this concerns me
> much more than dual licensing, because it makes features unavailable to
> regular users, whereas with dual licensing this does not (necessarily)
> happen.
>
>> There's also the issue of sponsered work that was done with the belief
>> that the result would be released under the GPL. Some of that work is
>> fairly core. This would be a potential problem for the MySQL approach.
>
> Definitely, depending on the reaction this can be a big problem. I am right
> now analyzing the code. I have found one large contribution that falls under
> this category. There are a couple of smaller ones and there are other
> contributions. One thing that my peers tell me is that it may even make sense
> to rewrite the core (after all, there is room for improvement) than to ditch
> the idea. Think it that way: development was almost exclusively paid for by
> Adiscon. What problem would it be for Adiscon if we do a 4-month rewrite,
> with real improvement, in the sake of settling these copyright issues? I have
> to say this argument is hard to beat. Now think about "open sore": of course
> Adiscon can do this new design and license *that part* under a commercial
> license. That would be a loss for the community, while dual licensing would
> do no harm (at least I think so).
>
> But the issue is actually open again, and I appreciate all your feedback. The
> problem is that Adiscon finally gets impatiently funding rsyslog, so I am
> down to doing that in my spare time and on those rare occasions where someone
> actually pays for that work (e.g. via support contracts). Adiscon doesn't
> want to ditch rsyslog, that's for sure. But I am told that my development
> time must be paid for, and that means I need to work on other projects
> (that's exactly what folks like Red Hat said to their employees, so it is not
> unreasonable). The situation has been worsened by the arrival of the journald
> idea. It did very much harm my position. My peers now say "look, nobody is
> interested in actually helping with the project. If they improve logging,
> they prefer to do it via their own project". That seems to be a true
> assumption (it would have been easy to do what journal does in rsyslog
> context -- if I had have more time, I'd already written a similar log store).
> Journald will also considerably shrink the ability to create other value out
> of the rsyslog project by pushing it into a nice. The core question behind
> this is if there is enough interest in the open source world to keep rsyslog
> a totally free solution. The sore reality is that we have some great
> contributors (including you!) but the vast majority of work is done by me.
> Looking at other logging projects, my peers tell me, (almost?) all of them
> have moved to some commercial funding. Logging seems to be far too less sexy
> to attract the masses.

you are far from the onlyone unhappy about the journal proposal (take a 
look at lwn.net for examples), but one thing to point out is that the 
author of that proposal doesn't like anything in the linux ecosystem, his 
standard mode of operation is to decide that everything that has existed 
before is junk and propose a solution that does a portion of what was done 
before with one or two new features and then proceed to label anyone who 
disagrees with him as an old fogy stick-in-the-mud who just isn't 
interestedin progress.

I think it would significantly help to undermine this proposal if rsyslog 
were to gain the ability to add a checksum to the output template. I 
believe that you already keep the prior message around (to support the 
'last message repeated' functionality). how hard would it be to add a 
property that gets calculated on the fly when it's used?

part of the problem with contributing to rsyslog is finding good examples 
of how to implement things. there is a lot of functionality, but there are 
so many helper routines and pieces that figuring out how to use it is 
hard.

David Lang

From marcin at mejor.pl  Fri Nov 25 10:05:33 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Fri, 25 Nov 2011 10:05:33 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
References: <4ECE848B.6070203@horoa.net>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
Message-ID: <4ECF5A5D.7040206@mejor.pl>

W dniu 25.11.2011 07:21, Rainer Gerhards pisze:
Hello,
> RFC5424 provides high-precision timestamps with timezone information. Rsyslog
> does so for many years. It is a mayor frustration for me that all distros
> turn off that feature (which is enabled by default).

i don't agree:
2011-11-25T10:04:13.047096+01:00 localhost kernel: [   41.560258] 
ata4.01: configured for UDMA/133
I'm using gentoo :)
Regards

From rgerhards at hq.adiscon.com  Fri Nov 25 10:33:05 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 10:33:05 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: <4ECF5A5D.7040206@mejor.pl>
References: <4ECE848B.6070203@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
	<4ECF5A5D.7040206@mejor.pl>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281518@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Marcin Miroslaw
> Sent: Friday, November 25, 2011 10:06 AM
> To: rsyslog at lists.adiscon.com
> Subject: Re: [rsyslog] timereported timezone?
> 
> W dniu 25.11.2011 07:21, Rainer Gerhards pisze:
> Hello,
> > RFC5424 provides high-precision timestamps with timezone information.
> Rsyslog
> > does so for many years. It is a mayor frustration for me that all
> distros
> > turn off that feature (which is enabled by default).
> 
> i don't agree:
> 2011-11-25T10:04:13.047096+01:00 localhost kernel: [   41.560258]
> ata4.01: configured for UDMA/133
> I'm using gentoo :)

Oh, really? On by *default*? That would make my day...

rainer

> Regards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From marcin at mejor.pl  Fri Nov 25 10:38:34 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Fri, 25 Nov 2011 10:38:34 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281518@GRFEXC.intern.adiscon.com>
References: <4ECE848B.6070203@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
	<4ECF5A5D.7040206@mejor.pl>
	<9B6E2A8877C38245BFB15CC491A11DA7281518@GRFEXC.intern.adiscon.com>
Message-ID: <4ECF621A.1090709@mejor.pl>

W dniu 25.11.2011 10:33, Rainer Gerhards pisze:
> Oh, really? On by *default*? That would make my day...

Yes by default and unconditionally:
$ grep 3164 rsyslog-5.8.5.ebuild
                 --enable-pmrfc3164sd
:)

From alorbach at ro1.adiscon.com  Fri Nov 25 10:39:07 2011
From: alorbach at ro1.adiscon.com (Andre Lorbach)
Date: Fri, 25 Nov 2011 10:39:07 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	
Message-ID: 

> I think it would significantly help to undermine this proposal if rsyslog
were to
> gain the ability to add a checksum to the output template. I believe that
you
> already keep the prior message around (to support the 'last message
> repeated' functionality). how hard would it be to add a property that gets
> calculated on the fly when it's used?
> 
> part of the problem with contributing to rsyslog is finding good examples
of
> how to implement things. there is a lot of functionality, but there are so
> many helper routines and pieces that figuring out how to use it is hard.
> 
> David Lang

It is funny but I had a discussion with Rainer earlier and we came to the
same conclusion. I also liked the Idea of journald to have a checksum-chain
which makes manipulation of logdata very difficult, and technically it should
be no problem to implement it into the current rsyslog code. It would make
this argument for journald obselete. 

Best regards,
Andre Lorbach 

From rgerhards at hq.adiscon.com  Fri Nov 25 11:47:24 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 11:47:24 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281519@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 9:51 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> 
> >> -----Original Message-----
> >> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> >>
> >>
> >> MySQL sold GPL exemptions so that companies could use their code in
> >> non-GPL proprietary applicaions (although some of it was based on a
> >> questionable interpretation of the GPL)
> >
> > From my personal PoV it would probably go into that direction, though
> > probably with a different vehicle than the GPL exemption. The key
> point is
> > that we could probably aquire funding from vendors who would like to
> > integrate rsyslog into their products, but are not willing to do so
> under
> > GPL.
> 
> interesting, I didn' know that there were oppurtunities here.

Definitely, probably more than in commercial licensing except if using "open
sore".

> 
> >> syslog-ng uses the 'open sore' approach where some features are not
> >> made
> >> available in the free version, only in the proprietary version.
> >
> > I like "open sore" more than "open core", it's up to the point :)
> 
> I was trying to be polite ;-)
> 
> >> how are you thinking that this would work for rsyslog?
> >
> > This is definitely something possible (and also discussed). Maybe it
> was my
> > fault to prevent this in the past two or three years. We would
> probably be
> > better off if I had not rejected the idea. Quite honestly, this
> concerns me
> > much more than dual licensing, because it makes features unavailable
> to
> > regular users, whereas with dual licensing this does not
> (necessarily)
> > happen.
> >
> >> There's also the issue of sponsered work that was done with the
> belief
> >> that the result would be released under the GPL. Some of that work
> is
> >> fairly core. This would be a potential problem for the MySQL
> approach.
> >
> > Definitely, depending on the reaction this can be a big problem. I am
> right
> > now analyzing the code. I have found one large contribution that
> falls under
> > this category. There are a couple of smaller ones and there are other
> > contributions. One thing that my peers tell me is that it may even
> make sense
> > to rewrite the core (after all, there is room for improvement) than
> to ditch
> > the idea. Think it that way: development was almost exclusively paid
> for by
> > Adiscon. What problem would it be for Adiscon if we do a 4-month
> rewrite,
> > with real improvement, in the sake of settling these copyright
> issues? I have
> > to say this argument is hard to beat. Now think about "open sore": of
> course
> > Adiscon can do this new design and license *that part* under a
> commercial
> > license. That would be a loss for the community, while dual licensing
> would
> > do no harm (at least I think so).
> >
> > But the issue is actually open again, and I appreciate all your
> feedback. The
> > problem is that Adiscon finally gets impatiently funding rsyslog, so
> I am
> > down to doing that in my spare time and on those rare occasions where
> someone
> > actually pays for that work (e.g. via support contracts). Adiscon
> doesn't
> > want to ditch rsyslog, that's for sure. But I am told that my
> development
> > time must be paid for, and that means I need to work on other
> projects
> > (that's exactly what folks like Red Hat said to their employees, so
> it is not
> > unreasonable). The situation has been worsened by the arrival of the
> journald
> > idea. It did very much harm my position. My peers now say "look,
> nobody is
> > interested in actually helping with the project. If they improve
> logging,
> > they prefer to do it via their own project". That seems to be a true
> > assumption (it would have been easy to do what journal does in
> rsyslog
> > context -- if I had have more time, I'd already written a similar log
> store).
> > Journald will also considerably shrink the ability to create other
> value out
> > of the rsyslog project by pushing it into a nice. The core question
> behind
> > this is if there is enough interest in the open source world to keep
> rsyslog
> > a totally free solution. The sore reality is that we have some great
> > contributors (including you!) but the vast majority of work is done
> by me.
> > Looking at other logging projects, my peers tell me, (almost?) all of
> them
> > have moved to some commercial funding. Logging seems to be far too
> less sexy
> > to attract the masses.
> 
> you are far from the onlyone unhappy about the journal proposal (take a
> look at lwn.net for examples), but one thing to point out is that the
> author of that proposal doesn't like anything in the linux ecosystem,
> his
> standard mode of operation is to decide that everything that has
> existed
> before is junk and propose a solution that does a portion of what was
> done
> before with one or two new features and then proceed to label anyone
> who
> disagrees with him as an old fogy stick-in-the-mud who just isn't
> interestedin progress.
> 
> I think it would significantly help to undermine this proposal if
> rsyslog
> were to gain the ability to add a checksum to the output template. I
> believe that you already keep the prior message around (to support the
> 'last message repeated' functionality). how hard would it be to add a
> property that gets calculated on the fly when it's used?

The journald proposal has two parts: vaporware and technically correct stuff.
As I wrote in my blog, I think there is value in that proposal. Especially
the new logging API and the log store are useful ideas (syslog-ng commercial
fork has them, rsyslog would have them if I had found time to do so, it's not
really hard but takes some time to implement). The checksums are definitely
on the vaporware side of the proposal. It sounds nice, but lessens security,
because some folks may actually believe that they are better off with them
(and so the attacker is relieved that he can bless his attack by simply
regenerating the hashes ;)).

There are a couple of more serious proposals available, e.g.

http://tools.ietf.org/html/rfc5848

Albert Mietus has (around 2003!) implemented an early version of these ideas
in FreeBSD syslogd:

http://albert.mietus.nl/show.IT/securing_syslog_onFreeBSD.html

It's well known for years, so why are there so few implementations? There are
some *serious issues* involved when using signature chains and log
processing, even when done correctly (think filtering and relays for
example). I know, I've written a paper on that, but don't find it on the web
right now (it was for limited audience, will check on that and see if I can
release it).

The bottom line is that it works somewhat on a per-logfile basis, and
journald seems to use that option (but the way currently proposed sounds like
being the thought of a crypto script kiddie without much time to do the basic
homework ;)). Adding such a useless feature to the file output in rsyslog is
probably not much effort. But beware of the false sense of security. Anyhow,
I'll consider this. BUT... I don't think it will stop journald. Far more than
technical merit, Red Hat's power is to count on here. I know, rsyslog also
went great with Red Hat's help, so it is somewhat OK to make them drop its
importance ;) It is just that it has some serious political implications to
the project and it couldn't have come at a worse time.

> part of the problem with contributing to rsyslog is finding good
> examples
> of how to implement things. there is a lot of functionality, but there
> are
> so many helper routines and pieces that figuring out how to use it is
> hard.

It's definitely not easy to work through rsyslog's code, but
a) I've often started to help out people just to learn that they dropped
interest as soon as what they needed was implemented (big win for them, total
loss for me)
b) similar projects in the logging space face very similar challenges

In general, it would be beneficial if I had time to do some big refactoring
of rsyslog. The project has grown very much in very short time. There are
many things that I now know to do better. *If* we actually do such a big
rewrite, the big question is if I can find agreement inside Adiscon to do it
under GPL only, without the ability to license or any other funding visible
(actually, the question even more is if I shouldn't better use that effort
for the commercial Windows product line, which would also benefit from some
of the ideas).

As a personal rant, I am a bit disappointed by those users that, for example
on the forum, post things like "hey, can't you answer my question quickly,
this is urgent, I have a big project depending on this. Why does it take so
long?". Or those that ask me to do some review of their commercial plugin to
make sure their boss is happy. Or those others employed by big software
companies that actually make their living out of commercial software (and no
OSS at all) but run their shop on open source and insist on their weird
problem with rsyslog deserve most of my attention. All of them, of course, do
not spend a single buck for helping out the project, beware of purchasing a
single support incident when it is urgent. They prefer lamenting. I have to
admit that I am becoming totally tired of these folks. What is worth noting
is that they provide *excellent* argument for my peers that most corporations
don't fund projects until they absolutely need to. Some other users would
like to spend some money, but are unable because their companies do not
understand open source, so they need to have something they can license (yes,
they must *license* it, a support contract will not do). Of course, there are
many other users, as the mailing list regulars know, and I am grateful for
all their help and motivation. There is also a (very small) set of companies
that have helped the project and I am very grateful for that (but can't
mention their names due to NDA).  But at the end of the day, I think my
peer's question "Can't we make changes to at least provide funding for the
work we do?" is a very valid one. As is "will this purely open source thing
ever work?". Again, journald is devastating in this regard, as it assigns a
definite end-of-life on *widespread* use of rsyslog, so my argument "let's
wait, time will work for us" all of the sudden was totally blown away... It
is also devastating, as it pushes rsyslog into a strictly commercial user
base in the long term, and so fuels the question why there is any sense in
providing help to commercial users without getting funding from them? Just
think about the *real* case where we provide rsyslog for free to some company
which in turn charges me for their software -- no joke! I am doing volunteer
astronomy speeches in our community, and the software I need to do some of
the animations is the one in question. Isn't that strange? I have to pay them
for my non-profit use of their software, but they get my software for free
for their heavy for-profit use... ;) [and I know about a this because their
folks tend to complain that our for-free support is so slow (but much faster
than their paid-for support)]... When I think about such samples, I, too, ask
myself if I am just plainly cracy...

With sad regards ;)
Rainer

From rgerhards at hq.adiscon.com  Fri Nov 25 12:14:53 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 12:14:53 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: <4ECF621A.1090709@mejor.pl>
References: <4ECE848B.6070203@horoa.net><9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com><4ECF5A5D.7040206@mejor.pl><9B6E2A8877C38245BFB15CC491A11DA7281518@GRFEXC.intern.adiscon.com>
	<4ECF621A.1090709@mejor.pl>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728151B@GRFEXC.intern.adiscon.com>

> W dniu 25.11.2011 10:33, Rainer Gerhards pisze:
> > Oh, really? On by *default*? That would make my day...
> 
> Yes by default and unconditionally:
> $ grep 3164 rsyslog-5.8.5.ebuild
>                  --enable-pmrfc3164sd
> :)

Excellent news! :-D

Rainer

From epiphani at gmail.com  Fri Nov 25 15:12:37 2011
From: epiphani at gmail.com (Aaron Wiebe)
Date: Fri, 25 Nov 2011 09:12:37 -0500
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com>
	
Message-ID: 

I'm with David... come on people, pony up some cash.

On Thu, Nov 24, 2011 at 11:13 PM,   wrote:
> I'm disappointed to see this, please keep us as informed as you can on these
> discussions.
>
> David Lang
>
> On Wed, 23 Nov 2011, Rainer Gerhards wrote:
>
>> Hi all,
>>
>> I am probably forced by circumstances to make some changes in the current
>> licensing of rsyslog. I have elaborated in this blog post:
>>
>> http://blog.gerhards.net/2011/11/funding-rsyslog-development.html
>>
>> Feedback is of course appreciated (but please be gentle with me ;)).
>>
>> Rainer
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>

From rgerhards at hq.adiscon.com  Fri Nov 25 15:18:16 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 15:18:16 +0100
Subject: [rsyslog] journald crypto hype
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281522@GRFEXC.intern.adiscon.com>

Hi all,

discussing with David I said that journald's hash chaining method does not
work. I now dug out the paper I mentioned and also have blogged on what the
(pretty obvious) problem is:

http://blog.gerhards.net/2011/11/journald-log-hash-chaining-is-broken.html

enjoy ;)

Rainer

From cclark at quadrantsec.com  Fri Nov 25 16:39:17 2011
From: cclark at quadrantsec.com (Champ Clark III [Quadrant])
Date: Fri, 25 Nov 2011 10:39:17 -0500
Subject: [rsyslog] journald crypto hype
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281522@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA7281522@GRFEXC.intern.adiscon.com>
Message-ID: <3AD19FC8-E71E-4C32-A8DB-AB7437F01A00@quadrantsec.com>

Nice read, thanks....

On Nov 25, 2011, at 9:18 AM, Rainer Gerhards wrote:

> Hi all,
> 
> discussing with David I said that journald's hash chaining method does not
> work. I now dug out the paper I mentioned and also have blogged on what the
> (pretty obvious) problem is:
> 
> http://blog.gerhards.net/2011/11/journald-log-hash-chaining-is-broken.html
> 
> enjoy ;)
> 
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/



Champ Clark III
(office) 904.253.7856
(mobile) 850.443.2440 
(SOC) 800.538.9357 ext 101
cclark at quadrantsec.com
www.quadrantsec.com


From cclark at quadrantsec.com  Fri Nov 25 16:28:21 2011
From: cclark at quadrantsec.com (Champ Clark III [Quadrant])
Date: Fri, 25 Nov 2011 10:28:21 -0500
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
Message-ID: <88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>

IMHO the Journald is already vapor.   Half the "issues" journeld is "solving" have been "solved" for some time. 

You could split up the modules verses open source/commercial.   We're doing this with Sagan and a very _specific_ output module.  I can't go into details,   but the module ties in with an already existing commercial product.  Since that is the case,  it was determined that this specific module would be a "commercial" module and not available as a open source offering.   If this specific module was open source, then the commercial entity would have no reason to re-emburse our development time. They'd just take it and use it as there own.    The organization in mind already uses _many_ open source products without returning anything to the community.   

rsyslog could do something similar.  

You could also start offering service around rsyslog.  That is,  not just development work,  but deployment.   I'm sure there are many organizations that'd like to do very specific things where logs,  but don't know how to deploy rsyslog to do such things. 





Champ Clark III
(office) 904.253.7856
(mobile) 850.443.2440 
(SOC) 800.538.9357 ext 101
cclark at quadrantsec.com
www.quadrantsec.com


From rgerhards at hq.adiscon.com  Fri Nov 25 17:47:17 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 17:47:17 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	<88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Champ Clark III [Quadrant]
> Sent: Friday, November 25, 2011 4:28 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> IMHO the Journald is already vapor.   Half the "issues" journeld is
"solving"
> have been "solved" for some time.

We'll see how it progresses... As I wrote on my blog initially, it has some
interesting things. If Lennart could have resisted to make a 14 point bullet
list and stayed with the three or four issues that are actually a problem,
the paper would look much more credible. The conclusion is also not necessary
right: these things could be improved within the existing frameworks.

> You could split up the modules verses open source/commercial.   We're
> doing this with Sagan and a very _specific_ output module.  I can't go into
> details,   but the module ties in with an already existing commercial
product.
> Since that is the case,  it was determined that this specific module would
be a
> "commercial" module and not available as a open source offering.   If this
> specific module was open source, then the commercial entity would have no
> reason to re-emburse our development time. They'd just take it and use it
as
> there own.    The organization in mind already uses _many_ open source
> products without returning anything to the community.
>
> rsyslog could do something similar.

Yeah, that could work. Maybe I should ask for some proposals ;) (if I start
with something and that will be a failure, I have definitely lost with my
peers). I have to admit that I objected this idea for a long time, because I
thought all features should be available to the general public. I see this is
not point in your case, but your case is very specific. All things I
currently see would benefit the general public as well.

> You could also start offering service around rsyslog.  That is,  not just
> development work,  but deployment.   I'm sure there are many organizations
> that'd like to do very specific things where logs,  but don't know how to
> deploy rsyslog to do such things.

We are offering support contracts and some folks have bought them. Those who
have are very happy (at least I think so and hear so) But they are very few
in numbers. Deployment services seems to be a bit problematic. Firstly,
because we are a small shop and have no worldwide presence. At least locally
I become quite a bit frustrated as I know about a couple of German projects
where we even weren't asked to provide services (and no bid issued). Not that
we were too expensive, they had their existing partners (which is fine). I
have to admit I was a bit upset with one project that happened right in my
Neighborhood (around 60 miles away) where some specific plugin was developed.
The consultant that was tasked with this development even asked me to audit
it (for free, of course) so that he could make sure he could pass it without
problems to his customers (and, of course, this was never meant to be
contributed back, just an inhouse solution that gains the company in question
competitive advantage). There is a reason that my peers get impatient (and me
upset at times ;)).

One non-intrusive thing we will try is to create a special version of our
Windows tools and make this work very well with rsyslog. That at least is a
way for those guys in big corporations that want to support us can do so even
if the company does not understand open source. There is some hope that this
works, but it is weak as we already offer these tools and tell folks that
they can use them to help fund the project. Not sure how much more a renamed
and somewhat streamlined agent for Windows will bring. Bet let's hope for the
best (bottom line: we try to explore unintrusive ways to find new funding
streams).

We will also definitely dual-license the normalizing products. I have great
hope for them and they are something the market needs (even with journald).
Dual-licensing ist vital here, as these things most probably go into other
vendor's products. As I said, others have the same issues: syslog-ng did not
receive proper funding before going commercial and php-syslog-ng (now
logzilla) also went commercial because of the missing support. There are
numerous other samples. I still try to remain as open and free as possible,
because I really believe open source has already created a much better world.
In my personal opinion,  businesses (especially big ones) should help fund
projects, because they have commercial benefit. The smaller guys, edu, home
folks and all the rest of the non-profits should reap the benefit of that
work. To me, this is fair sharing and the way our society should evolve. It's
a shame that from time to time we receive some contributions from home users
while the big guys try to avoid spending as much as possible. As much as I
appreciate the home users contributions as they value my work, I *really*
think this is the wrong route for a project that brings strong commercial
benefit to many large for-profit organizations. But enough of that rant ;)

Thanks again for all your comments!
Rainer

From rgerhards at hq.adiscon.com  Fri Nov 25 17:58:17 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 17:58:17 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com><88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
	<9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281529@GRFEXC.intern.adiscon.com>

> In my personal opinion,  businesses (especially big ones) should help fund
> projects, because they have commercial benefit. The smaller guys, edu,
> home folks and all the rest of the non-profits should reap the benefit of
that
> work. To me, this is fair sharing and the way our society should evolve.
It's a
> shame that from time to time we receive some contributions from home
> users while the big guys try to avoid spending as much as possible. As much
> as I appreciate the home users contributions as they value my work, I
> *really* think this is the wrong route for a project that brings strong
> commercial benefit to many large for-profit organizations. But enough of
that
> rant ;)

Looks like this is my favorite rant ;) The full version, with a much more
subtle view, is available here:

http://blog.gerhards.net/2009/11/paying-for-open-source-projects.html

don't take the word "paying" inside the URL too strict (you'll get it once
you read the first paragraph).

I thought I share this link for those interested.

Rainer

From cclark at quadrantsec.com  Fri Nov 25 19:32:20 2011
From: cclark at quadrantsec.com (Champ Clark III [Quadrant])
Date: Fri, 25 Nov 2011 13:32:20 -0500
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	<88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
	<9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
Message-ID: <51F97304-10D6-46A7-897F-269F2C46369F@quadrantsec.com>


On Nov 25, 2011, at 11:47 AM, Rainer Gerhards wrote:
> 
> We'll see how it progresses... As I wrote on my blog initially, it has some
> interesting things. If Lennart could have resisted to make a 14 point bullet
> list and stayed with the three or four issues that are actually a problem,
> the paper would look much more credible. The conclusion is also not necessary
> right: these things could be improved within the existing frameworks.

I've not really seen anything a "like" about the over all proposal (of course IMHO!).  Also,  it doesn't change the fact that other devices (routers/firewall/etc) will continue using syslog.  So,  the entire thing is moot (again, IMHO).
> 
> Yeah, that could work. Maybe I should ask for some proposals ;) (if I start
> with something and that will be a failure, I have definitely lost with my
> peers). I have to admit that I objected this idea for a long time, because I
> thought all features should be available to the general public. I see this is
> not point in your case, but your case is very specific. All things I
> currently see would benefit the general public as well.

In our case (Sagan),  we simply _knew_ based on this organizations usage of OSS in the past,  they'd simply re-package and sell it.   This output plugin is a very nitch plugin and requires you to already have a very expensive product.   We don't want that to happen,  which lead us to a "commerical" plugin.   I should point out,  this plug in is not completed.  We're still developing it and our thoughts are that it won't impact the normal OSS Sagan.  I'd be _very_ surprised if a standard user "asked" for this plugin for free. 
> 
>> You could also start offering service around rsyslog.  That is,  not just
>> development work,  but deployment.   I'm sure there are many organizations
>> that'd like to do very specific things where logs,  but don't know how to
>> deploy rsyslog to do such things.
> 
> We are offering support contracts and some folks have bought them. Those who
> have are very happy (at least I think so and hear so) But they are very few
> in numbers. Deployment services seems to be a bit problematic. Firstly,
> because we are a small shop and have no worldwide presence. At least locally
> I become quite a bit frustrated as I know about a couple of German projects
> where we even weren't asked to provide services (and no bid issued). Not that
> we were too expensive, they had their existing partners (which is fine). I
> have to admit I was a bit upset with one project that happened right in my
> Neighborhood (around 60 miles away) where some specific plugin was developed.
> The consultant that was tasked with this development even asked me to audit
> it (for free, of course) so that he could make sure he could pass it without
> problems to his customers (and, of course, this was never meant to be
> contributed back, just an inhouse solution that gains the company in question
> competitive advantage). There is a reason that my peers get impatient (and me
> upset at times ;)).

Wow..  that is annoying.   I think for smaller deployments,   not being world wide might be a issue.   However,  I have to believe
that rsyslog is being used in much larger deployments around the world.   Getting "in the door" with such organizations is another matter.  Also,  it's likely they already have the deployment working as they want it.

> One non-intrusive thing we will try is to create a special version of our
> Windows tools and make this work very well with rsyslog. That at least is a
> way for those guys in big corporations that want to support us can do so even
> if the company does not understand open source. There is some hope that this
> works, but it is weak as we already offer these tools and tell folks that
> they can use them to help fund the project. Not sure how much more a renamed
> and somewhat streamlined agent for Windows will bring. Bet let's hope for the
> best (bottom line: we try to explore unintrusive ways to find new funding
> streams).

One problem here is that you're working on the "engine" that collects logs.  While it's the most important,  it's not that sexy (from a buyers stand point).  Look at software like Splunk.    That's been an amazing success,  partly because it can create pretty pictures .... and it's useful.  Rsyslog,  by itself,  can't do this and is so embedded on the "back end",  it's not that "sexy" (to most!) 

Perhaps packaging a rsyslog based back end with a nice front end as a "appliance" might work.   Don't get to caught up with the term "appliance".    I'm thinking a pre-configured Linux distro setup for logging and visualization of logs.  Of course,  using the fame you've received for being the author of rsyslog and it being already used by thousands of organizations.  

I'm just pondering on ideas... 

> We will also definitely dual-license the normalizing products. I have great
> hope for them and they are something the market needs (even with journald).
> Dual-licensing ist vital here, as these things most probably go into other
> vendor's products. As I said, others have the same issues: syslog-ng did not
> receive proper funding before going commercial and php-syslog-ng (now
> logzilla) also went commercial because of the missing support. There are

Yes,  but last time I looked,  Logzilla user base had dropped a good bit.  Also,  the last time I looked,  the pricing model was way out of wack (IMHO).   I'd research how successful going commercial turned out for Logzilla.  I don't think it's worked out as well,  however,  I'm not 100% sure.

> numerous other samples. I still try to remain as open and free as possible,
> because I really believe open source has already created a much better world.
> In my personal opinion,  businesses (especially big ones) should help fund
> projects, because they have commercial benefit. The smaller guys, edu, home
> folks and all the rest of the non-profits should reap the benefit of that
> work. To me, this is fair sharing and the way our society should evolve. It's


> a shame that from time to time we receive some contributions from home users
> while the big guys try to avoid spending as much as possible. As much as I
> appreciate the home users contributions as they value my work, I *really*
> think this is the wrong route for a project that brings strong commercial
> benefit to many large for-profit organizations. But enough of that rant ;)

Dual license of the normalization library isn't a bad idea.  I to believe in open source,  but I also like to have a roof over my head and eat food from time to time :)

Were you thinking about normalization licenses for commercial usage?  Sagan already uses the liblognorm for some stuff and it's been a great value for Sagan to sanely be able to normalize syslog input.    Sagan itself is OSS,  however,   we are using Sagan in some commercial (monitoring) settings.   While Sagan can operate without liiblognorm,  it certainly works better with it.    I'm pretty sure the organization I work at would be willing to licenses the normalization library when we use it in commercial setting.    I haven't talked with anyone about it at the office yet so I don't want to make any promises.    I believe I could make an argument for at least funding assisting for liblognorm.   Dual licensing would be important.  IE - one free/OSS and another for commercial usage.  




Champ Clark III
(office) 904.253.7856
(mobile) 850.443.2440 
(SOC) 800.538.9357 ext 101
cclark at quadrantsec.com
www.quadrantsec.com


From david at lang.hm  Fri Nov 25 20:47:46 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 11:47:46 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <51F97304-10D6-46A7-897F-269F2C46369F@quadrantsec.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	<88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
	<9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
	<51F97304-10D6-46A7-897F-269F2C46369F@quadrantsec.com>
Message-ID: 

On Fri, 25 Nov 2011, Champ Clark III [Quadrant] wrote:

> On Nov 25, 2011, at 11:47 AM, Rainer Gerhards wrote:
>
>> One non-intrusive thing we will try is to create a special version of our
>> Windows tools and make this work very well with rsyslog. That at least is a
>> way for those guys in big corporations that want to support us can do so even
>> if the company does not understand open source. There is some hope that this
>> works, but it is weak as we already offer these tools and tell folks that
>> they can use them to help fund the project. Not sure how much more a renamed
>> and somewhat streamlined agent for Windows will bring. Bet let's hope for the
>> best (bottom line: we try to explore unintrusive ways to find new funding
>> streams).

A good tool to send logs from windows boxes with support would be very 
useful. there are a number of 'free' tools out there, and some other 
fairly cheap ones (including from adiscon), but the ones that are pure 
community projects tend to not keep up with windows releases (and have no 
support), while the ones from commercial companies are all really intended 
for other things and just happen to do syslog as well.

David Lang

>> We will also definitely dual-license the normalizing products. I have great
>> hope for them and they are something the market needs (even with journald).
>> Dual-licensing ist vital here, as these things most probably go into other
>> vendor's products. As I said, others have the same issues: syslog-ng did not
>> receive proper funding before going commercial and php-syslog-ng (now
>> logzilla) also went commercial because of the missing support. There are

The normalizing tools do look like a really good dual-licensing 
oppurtunity. There are a lot of regex engines out there, but they get 
really inefficient when dealing with large rulesets. A good parse tree 
engine would help a LOT of commercial software, and for performance 
reasons, they really want to have it linked directly to their software.

David Lang

From david at lang.hm  Fri Nov 25 20:52:59 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 11:52:59 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	<88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com>
	<9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com>
	<51F97304-10D6-46A7-897F-269F2C46369F@quadrantsec.com>
	
Message-ID: 

On Fri, 25 Nov 2011, david at lang.hm wrote:

> On Fri, 25 Nov 2011, Champ Clark III [Quadrant] wrote:
>
>> On Nov 25, 2011, at 11:47 AM, Rainer Gerhards wrote:
>> 
>>> One non-intrusive thing we will try is to create a special version of our
>>> Windows tools and make this work very well with rsyslog. That at least is 
>>> a
>>> way for those guys in big corporations that want to support us can do so 
>>> even
>>> if the company does not understand open source. There is some hope that 
>>> this
>>> works, but it is weak as we already offer these tools and tell folks that
>>> they can use them to help fund the project. Not sure how much more a 
>>> renamed
>>> and somewhat streamlined agent for Windows will bring. Bet let's hope for 
>>> the
>>> best (bottom line: we try to explore unintrusive ways to find new funding
>>> streams).
>
> A good tool to send logs from windows boxes with support would be very 
> useful. there are a number of 'free' tools out there, and some other fairly 
> cheap ones (including from adiscon), but the ones that are pure community 
> projects tend to not keep up with windows releases (and have no support), 
> while the ones from commercial companies are all really intended for other 
> things and just happen to do syslog as well.

one other reason this would be a good idea is that it would clearly 
deliniate the support as being for rsyslog. If the companies were to spend 
the similar amounts of money on the existing adiscon windows agent, the 
company would have no way of knowing that the reason for this was the 
rsyslog support.

As a side note on this, I'm hoping that this rsyslog windows agent could 
be done by grafting a windows eventlog input onto most of the existing 
rsyslog capabilities. It would be great it this was to mean that a lot of 
the existing capabilities were available to logging from windows (relp, 
disk assisted queues, failover if a receiver is down, etc)

David Lang



From david at lang.hm  Fri Nov 25 21:05:40 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 12:05:40 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281519@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281519@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

> Again, journald is devastating in this regard, as it assigns a definite 
> end-of-life on *widespread* use of rsyslog, so my argument "let's wait, 
> time will work for us" all of the sudden was totally blown away... It is 
> also devastating, as it pushes rsyslog into a strictly commercial user 
> base in the long term, and so fuels the question why there is any sense 
> in providing help to commercial users without getting funding from them?

It is an interesting question as to how much RedHat is going to be pushing 
this, especially in the face of such widespread complaining from 
sysadmins. I don't think they want to have to be the sole source for linux 
logging any more than Adiscon does, especially since the journal proposal 
doesn't come anywhere close to satisfying the needs of the RedHat 
Enterprise customers.

I really wonder how much this proposal has the support of RedHat, and how 
much is just LP.

David Lang

From robert at schetterer.org  Fri Nov 25 21:01:15 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Fri, 25 Nov 2011 21:01:15 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
Message-ID: <4ECFF40B.3000903@schetterer.org>

Hi @ll,
i am new to this list
and not very familar with rsyslog so sorry if my question was
asked/answered before

i wanna goal, filtering spamhaus rbl ips from postfix mail log
i allready done
( on suse )

mail.info
-/var/log/mail.info;RSYSLOG_TraditionalFileFormat
if $msg contains 'blocked using zen.spamhaus.org' then
/var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat

this works fine, but as i have a lot of spambots
fail2ban ist to slow with this allready filtered log

so i wann have a second or combinated rsyslog rule which results
only in the ip

a relevant log  entry example looks like this

postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
[190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
[190.24.212.146] blocked using zen.spamhaus.org;
from=, to=, proto=ESMTP,
helo=

the plan would be , set the ip output to i.e fifo pipe
and read the output via some daemon script/program to create an iptables
ip reject rule with timestamp via ipset map, which expires auto after 24
h, this should fast as possible

so anybody with an example for filtering this ?

by the way
someone speculated rsyslog may able to do the whole job
filtering ip and do the ipset action in one step , is this true ?
i am not that far in the docs yet

-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

From rgerhards at hq.adiscon.com  Fri Nov 25 22:03:36 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 22:03:36 +0100
Subject: [rsyslog] windows agent - was: FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com><88C985D2-0404-4D6E-8827-0CA6300B6D13@quadrantsec.com><9B6E2A8877C38245BFB15CC491A11DA7281528@GRFEXC.intern.adiscon.com><51F97304-10D6-46A7-897F-269F2C46369F@quadrantsec.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728152B@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 8:53 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> On Fri, 25 Nov 2011, david at lang.hm wrote:
> 
> > On Fri, 25 Nov 2011, Champ Clark III [Quadrant] wrote:
> >
> >> On Nov 25, 2011, at 11:47 AM, Rainer Gerhards wrote:
> >>
> >>> One non-intrusive thing we will try is to create a special version
> of our
> >>> Windows tools and make this work very well with rsyslog. That at
> least is
> >>> a
> >>> way for those guys in big corporations that want to support us can
> do so
> >>> even
> >>> if the company does not understand open source. There is some hope
> that
> >>> this
> >>> works, but it is weak as we already offer these tools and tell
> folks that
> >>> they can use them to help fund the project. Not sure how much more
> a
> >>> renamed
> >>> and somewhat streamlined agent for Windows will bring. Bet let's
> hope for
> >>> the
> >>> best (bottom line: we try to explore unintrusive ways to find new
> funding
> >>> streams).
> >
> > A good tool to send logs from windows boxes with support would be
> very
> > useful. there are a number of 'free' tools out there, and some other
> fairly
> > cheap ones (including from adiscon), but the ones that are pure
> community
> > projects tend to not keep up with windows releases (and have no
> support),
> > while the ones from commercial companies are all really intended for
> other
> > things and just happen to do syslog as well.
> 
> one other reason this would be a good idea is that it would clearly
> deliniate the support as being for rsyslog. If the companies were to
> spend
> the similar amounts of money on the existing adiscon windows agent, the
> company would have no way of knowing that the reason for this was the
> rsyslog support.

That's part of the idea :)

> As a side note on this, I'm hoping that this rsyslog windows agent
> could
> be done by grafting a windows eventlog input onto most of the existing
> rsyslog capabilities. It would be great it this was to mean that a lot
> of
> the existing capabilities were available to logging from windows (relp,
> disk assisted queues, failover if a receiver is down, etc)

It is important to keep in mind that I originally designed the Windows
engine. It has much evolved since then, but its concepts are still quite
similar to rsyslog. What we will do initially is grab the existing engine,
work a bit on the defaults to set if for good integration with rsyslog. Plus
some solid doc on how to handle various cases. The existing engine already
supports what you say, even though disk queues are implemented via a totally
different concept, failover works a bit differently and RELP is there. Over
time (if things work out), we plan to tie things even closer together. One
important step would be support of native windows event log fields, so that
we will have the corresponding name-value pairs on the rsyslog side directly
accessible and addressable via properties. This can go either via Adiscon
proprietary SETP protocol or via some new CEE/JSON way of doing things.

I hope that an initial version will be available soon. We have worked on
spec'ing this for a while now. For sure there will be room for improvement
when it is out, but I think it will have lots of good options.

Among others, it will have both support for the legacy and current Windows
Event Log system and handling of local files (think of that as a
much-enhanced version of imfile, the Windows engine is *much* better in this
regard). I have also fought hard to make it available as syslog relay for
small remote offices. I am not sure if that is really useful, but can
envision it is.

Rainer
 
> David Lang
> 
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From rgerhards at hq.adiscon.com  Fri Nov 25 22:07:04 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 22:07:04 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281519@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728152C@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
> Sent: Friday, November 25, 2011 9:06 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] FYI: funding rsyslog development
> 
> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> 
> > Again, journald is devastating in this regard, as it assigns a
> definite
> > end-of-life on *widespread* use of rsyslog, so my argument "let's
> wait,
> > time will work for us" all of the sudden was totally blown away... It
> is
> > also devastating, as it pushes rsyslog into a strictly commercial
> user
> > base in the long term, and so fuels the question why there is any
> sense
> > in providing help to commercial users without getting funding from
> them?
> 
> It is an interesting question as to how much RedHat is going to be
> pushing
> this, especially in the face of such widespread complaining from
> sysadmins. I don't think they want to have to be the sole source for
> linux
> logging any more than Adiscon does, especially since the journal
> proposal
> doesn't come anywhere close to satisfying the needs of the RedHat
> Enterprise customers.
> 
> I really wonder how much this proposal has the support of RedHat, and
> how
> much is just LP.

That's of course something to think of. I really don't know how Red Hat works
internally, but I would *speculate* that some rough strategic line is
conveyed to the team. On the other hand, there are lots of indications that
it is just LP (if looking closely, what I have done the past two days ;)).

HOWEVER, given the fact that this is so tied in into systemd, this alone is
probably a selling point (and this tight integration is probably one thing
that I really don't like about the whole idea ... it feels a bit like
becoming locked in on the whole set of technology).

Rainer

From rgerhards at hq.adiscon.com  Fri Nov 25 22:10:34 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Fri, 25 Nov 2011 22:10:34 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <4ECFF40B.3000903@schetterer.org>
References: <4ECFF40B.3000903@schetterer.org>
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
> Sent: Friday, November 25, 2011 9:01 PM
> To: rsyslog-users
> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
> 
> Hi @ll,
> i am new to this list
> and not very familar with rsyslog so sorry if my question was
> asked/answered before
> 
> i wanna goal, filtering spamhaus rbl ips from postfix mail log
> i allready done
> ( on suse )
> 
> mail.info
> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
> if $msg contains 'blocked using zen.spamhaus.org' then
> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
> 
> this works fine, but as i have a lot of spambots
> fail2ban ist to slow with this allready filtered log
> 
> so i wann have a second or combinated rsyslog rule which results
> only in the ip
> 
> a relevant log  entry example looks like this
> 
> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
> [190.24.212.146] blocked using zen.spamhaus.org;
> from=, to=, proto=ESMTP,
> helo=
> 
> the plan would be , set the ip output to i.e fifo pipe
> and read the output via some daemon script/program to create an
> iptables
> ip reject rule with timestamp via ipset map, which expires auto after
> 24
> h, this should fast as possible
> 
> so anybody with an example for filtering this ?
> 
> by the way
> someone speculated rsyslog may able to do the whole job
> filtering ip and do the ipset action in one step , is this true ?
> i am not that far in the docs yet

I have to admit that I do not fully understand your question. But it has been
a long and busy day today ;) Could you try to describe in two or three
sentences what you actually want to achieve?

rainer
> 
> --
> Best Regards
> 
> MfG Robert Schetterer
> 
> Germany/Munich/Bavaria
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

From david at lang.hm  Fri Nov 25 22:42:18 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 13:42:18 -0800 (PST)
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728152C@GRFEXC.intern.adiscon.com>
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281514@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281519@GRFEXC.intern.adiscon.com>
	
	<9B6E2A8877C38245BFB15CC491A11DA728152C@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>
>> It is an interesting question as to how much RedHat is going to be 
>> pushing this, especially in the face of such widespread complaining 
>> from sysadmins. I don't think they want to have to be the sole source 
>> for linux logging any more than Adiscon does, especially since the 
>> journal proposal doesn't come anywhere close to satisfying the needs of 
>> the RedHat Enterprise customers.
>>
>> I really wonder how much this proposal has the support of RedHat, and 
>> how much is just LP.
>
> That's of course something to think of. I really don't know how Red Hat works
> internally, but I would *speculate* that some rough strategic line is
> conveyed to the team. On the other hand, there are lots of indications that
> it is just LP (if looking closely, what I have done the past two days ;)).
>
> HOWEVER, given the fact that this is so tied in into systemd, this alone is
> probably a selling point (and this tight integration is probably one thing
> that I really don't like about the whole idea ... it feels a bit like
> becoming locked in on the whole set of technology).

I suspect that this concern (and the uncertin nature of the journal) is 
going to delay the acceptance of systemd as the default for some distros. 
In spite of how LP talks sometimes, systemd is not yet accepted as the 
future of linux. The fact that it is linux-only is probably going to keep 
it out of debian for the forseable future (they have the hurd and BSD 
kernel options) and ubuntu is still sponsering upstart.

David Lang


From david at lang.hm  Fri Nov 25 22:47:35 2011
From: david at lang.hm (david at lang.hm)
Date: Fri, 25 Nov 2011 13:47:35 -0800 (PST)
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>
References: <4ECFF40B.3000903@schetterer.org>
	<9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>
Message-ID: 

On Fri, 25 Nov 2011, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>>
>> Hi @ll,
>> i am new to this list
>> and not very familar with rsyslog so sorry if my question was
>> asked/answered before
>>
>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>> i allready done
>> ( on suse )
>>
>> mail.info
>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>> if $msg contains 'blocked using zen.spamhaus.org' then
>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>
>> this works fine, but as i have a lot of spambots
>> fail2ban ist to slow with this allready filtered log
>>
>> so i wann have a second or combinated rsyslog rule which results
>> only in the ip
>>
>> a relevant log  entry example looks like this
>>
>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>> [190.24.212.146] blocked using zen.spamhaus.org;
>> from=, to=, proto=ESMTP,
>> helo=
>>
>> the plan would be , set the ip output to i.e fifo pipe
>> and read the output via some daemon script/program to create an
>> iptables
>> ip reject rule with timestamp via ipset map, which expires auto after
>> 24
>> h, this should fast as possible
>>
>> so anybody with an example for filtering this ?
>>
>> by the way
>> someone speculated rsyslog may able to do the whole job
>> filtering ip and do the ipset action in one step , is this true ?
>> i am not that far in the docs yet
>
> I have to admit that I do not fully understand your question. But it has been
> a long and busy day today ;) Could you try to describe in two or three
> sentences what you actually want to achieve?

I think that what is is wanting to dois to create a custom output format 
that just contains the IP address (which would be something along the 
lines of a regex to match "\[([0-9.]+)\]" and log the output to a fifo for 
another script to handle.

that script would have to modify the iptables rule and schedule for the 
rule to be removed 24 hours later (I don't believe that iptables supports 
an expiration time directly). SEC (simple event correltator) is exactly 
the right type of tool for this type of work, just think through what will 
happen if the system gets rebooted before the 24 hours is up.

He then asks if rsyslog can do everything, rsyslog could execute a command 
and pass it the IP address , but it doesn't have the ability to schedule 
things for the future, so it's not the right tool for this job.

David Lang

From robert at schetterer.org  Fri Nov 25 23:16:40 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Fri, 25 Nov 2011 23:16:40 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>
References: <4ECFF40B.3000903@schetterer.org>
	<9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>
Message-ID: <4ED013C8.9020509@schetterer.org>

Am 25.11.2011 22:10, schrieb Rainer Gerhards:
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>> Sent: Friday, November 25, 2011 9:01 PM
>> To: rsyslog-users
>> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
>>
>> Hi @ll,
>> i am new to this list
>> and not very familar with rsyslog so sorry if my question was
>> asked/answered before
>>
>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>> i allready done
>> ( on suse )
>>
>> mail.info
>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>> if $msg contains 'blocked using zen.spamhaus.org' then
>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>
>> this works fine, but as i have a lot of spambots
>> fail2ban ist to slow with this allready filtered log
>>
>> so i wann have a second or combinated rsyslog rule which results
>> only in the ip
>>
>> a relevant log  entry example looks like this
>>
>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>> [190.24.212.146] blocked using zen.spamhaus.org;
>> from=, to=, proto=ESMTP,
>> helo=
>>
>> the plan would be , set the ip output to i.e fifo pipe
>> and read the output via some daemon script/program to create an
>> iptables
>> ip reject rule with timestamp via ipset map, which expires auto after
>> 24
>> h, this should fast as possible
>>
>> so anybody with an example for filtering this ?
>>
>> by the way
>> someone speculated rsyslog may able to do the whole job
>> filtering ip and do the ipset action in one step , is this true ?
>> i am not that far in the docs yet
> 
> I have to admit that I do not fully understand your question. But it has been
> a long and busy day today ;) Could you try to describe in two or three
> sentences what you actually want to achieve?

ok

this is a exmaple from the mail log

postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
[190.24.212.146]:57855: 550 5.7.1 Service unavailable; client

( this is the relevant match part )
[190.24.212.146] blocked using zen.spamhaus.org;


from=, to=, proto=ESMTP,
helo=

i wann have filtered 190.24.212.146 only by rsyslog
and do this
echo +190.24.212.146 > /proc/net/xt_recent/SMTP

if possible directly all done from rsyslog

if not ,its enough to pipe fifo 190.24.212.146 only
i will script some daemon reading the fifo then

but as someother answered i think rsyslog could do it in one step

clear now?




> 
> rainer
>>
>> --
>> Best Regards
>>
>> MfG Robert Schetterer
>>
>> Germany/Munich/Bavaria
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/


From robert at schetterer.org  Fri Nov 25 23:24:47 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Fri, 25 Nov 2011 23:24:47 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: 
References: <4ECFF40B.3000903@schetterer.org>
	<9B6E2A8877C38245BFB15CC491A11DA728152D@GRFEXC.intern.adiscon.com>
	
Message-ID: <4ED015AF.5070809@schetterer.org>

Am 25.11.2011 22:47, schrieb david at lang.hm:
> On Fri, 25 Nov 2011, Rainer Gerhards wrote:
> 
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>>>
>>> Hi @ll,
>>> i am new to this list
>>> and not very familar with rsyslog so sorry if my question was
>>> asked/answered before
>>>
>>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>>> i allready done
>>> ( on suse )
>>>
>>> mail.info
>>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>>> if $msg contains 'blocked using zen.spamhaus.org' then
>>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>>
>>> this works fine, but as i have a lot of spambots
>>> fail2ban ist to slow with this allready filtered log
>>>
>>> so i wann have a second or combinated rsyslog rule which results
>>> only in the ip
>>>
>>> a relevant log  entry example looks like this
>>>
>>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>> [190.24.212.146] blocked using zen.spamhaus.org;
>>> from=, to=, proto=ESMTP,
>>> helo=
>>>
>>> the plan would be , set the ip output to i.e fifo pipe
>>> and read the output via some daemon script/program to create an
>>> iptables
>>> ip reject rule with timestamp via ipset map, which expires auto after
>>> 24
>>> h, this should fast as possible
>>>
>>> so anybody with an example for filtering this ?
>>>
>>> by the way
>>> someone speculated rsyslog may able to do the whole job
>>> filtering ip and do the ipset action in one step , is this true ?
>>> i am not that far in the docs yet
>>
>> I have to admit that I do not fully understand your question. But it
>> has been
>> a long and busy day today ;) Could you try to describe in two or three
>> sentences what you actually want to achieve?
> 
> I think that what is is wanting to dois to create a custom output format
> that just contains the IP address (which would be something along the
> lines of a regex to match "\[([0-9.]+)\]" and log the output to a fifo
> for another script to handle.
> 
> that script would have to modify the iptables rule and schedule for the
> rule to be removed 24 hours later (I don't believe that iptables
> supports an expiration time directly).

this example works for small tests i.e autp expire for 120 seconds
rule

iptables -I INPUT -p tcp --dport 25 -m state --state NEW -m recent
--name SMTP --rcheck --seconds 120 -j REJECT --reject-with tcp-reset

add ip i.e 127.0.0.1

echo +127.0.0.1 > /proc/net/xt_recent/SMTP

delete

echo -127.0.0.1 > /proc/net/xt_recent/SMTP

 SEC (simple event correltator) is
> exactly the right type of tool for this type of work, just think through
> what will happen if the system gets rebooted before the 24 hours is up.

mailservers get rebooted very rare, anyway its not a problem here

> 
> He then asks if rsyslog can do everything, rsyslog could execute a
> command and pass it the IP address , but it doesn't have the ability to
> schedule things for the future, so it's not the right tool for this job.

ok understand rsyslog can filter this an do some action
thats ok for me, any idea which it should look like in the conf ?

> 
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/


From marc.schiffbauer at mightycare.de  Sat Nov 26 00:50:05 2011
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Sat, 26 Nov 2011 00:50:05 +0100 (CET)
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <4ED013C8.9020509@schetterer.org>
Message-ID: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>

----- Urspr?ngliche Mail -----
> Am 25.11.2011 22:10, schrieb Rainer Gerhards:
> >> -----Original Message-----
> >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> >> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
> >> Sent: Friday, November 25, 2011 9:01 PM
> >> To: rsyslog-users
> >> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
> >>
> >> Hi @ll,
> >> i am new to this list
> >> and not very familar with rsyslog so sorry if my question was
> >> asked/answered before
> >>
> >> i wanna goal, filtering spamhaus rbl ips from postfix mail log
> >> i allready done
> >> ( on suse )
> >>
> >> mail.info
> >> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
> >> if $msg contains 'blocked using zen.spamhaus.org' then
> >> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
> >>
> >> this works fine, but as i have a lot of spambots
> >> fail2ban ist to slow with this allready filtered log
> >>
> >> so i wann have a second or combinated rsyslog rule which results
> >> only in the ip
> >>
> >> a relevant log  entry example looks like this
> >>
> >> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
> >> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
> >> [190.24.212.146] blocked using zen.spamhaus.org;
> >> from=, to=,
> >> proto=ESMTP,
> >> helo=
> >>
> >> the plan would be , set the ip output to i.e fifo pipe
> >> and read the output via some daemon script/program to create an
> >> iptables
> >> ip reject rule with timestamp via ipset map, which expires auto
> >> after
> >> 24
> >> h, this should fast as possible
> >>
> >> so anybody with an example for filtering this ?
> >>
> >> by the way
> >> someone speculated rsyslog may able to do the whole job
> >> filtering ip and do the ipset action in one step , is this true ?
> >> i am not that far in the docs yet
> > 
> > I have to admit that I do not fully understand your question. But
> > it has been
> > a long and busy day today ;) Could you try to describe in two or
> > three
> > sentences what you actually want to achieve?
> 
> ok
> 
> this is a exmaple from the mail log
> 
> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
> 
> ( this is the relevant match part )
> [190.24.212.146] blocked using zen.spamhaus.org;
> 
> 
> from=, to=, proto=ESMTP,
> helo=
> 
> i wann have filtered 190.24.212.146 only by rsyslog
> and do this
> echo +190.24.212.146 > /proc/net/xt_recent/SMTP
> 
> if possible directly all done from rsyslog
> 
> if not ,its enough to pipe fifo 190.24.212.146 only
> i will script some daemon reading the fifo then
> 
> but as someother answered i think rsyslog could do it in one step
> 
> clear now?

Hi Robert,

maybe something like that would work (untested):

$template MyTemplate,"+%msg:R,ERE,1,BLANK:.*\[([0-9.]+)\].*--end%"
if $msg contains 'blocked using zen.spamhaus.org' \
then /proc/net/xt_recent/SMTP;MyTemplate

-Marc


From cclark at quadrantsec.com  Sat Nov 26 01:29:41 2011
From: cclark at quadrantsec.com (Champ Clark III [Quadrant])
Date: Fri, 25 Nov 2011 19:29:41 -0500
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>
References: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>
Message-ID: <660C427E-0BE4-4AA4-BEF6-6BD2E09B2AA1@quadrantsec.com>

You could also use Sagan to trigger on the event,  extract the information (via liblognorm) and pass it to a external plug in (which fires iptables).  Pretty straight forward.   I plan on having snortsam integration into Sagan soon as well.

On Nov 25, 2011, at 6:50 PM, Marc Schiffbauer wrote:

> ----- Urspr?ngliche Mail -----
>> Am 25.11.2011 22:10, schrieb Rainer Gerhards:
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>>>> Sent: Friday, November 25, 2011 9:01 PM
>>>> To: rsyslog-users
>>>> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
>>>> 
>>>> Hi @ll,
>>>> i am new to this list
>>>> and not very familar with rsyslog so sorry if my question was
>>>> asked/answered before
>>>> 
>>>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>>>> i allready done
>>>> ( on suse )
>>>> 
>>>> mail.info
>>>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>>>> if $msg contains 'blocked using zen.spamhaus.org' then
>>>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>>> 
>>>> this works fine, but as i have a lot of spambots
>>>> fail2ban ist to slow with this allready filtered log
>>>> 
>>>> so i wann have a second or combinated rsyslog rule which results
>>>> only in the ip
>>>> 
>>>> a relevant log  entry example looks like this
>>>> 
>>>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>>>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>>> [190.24.212.146] blocked using zen.spamhaus.org;
>>>> from=, to=,
>>>> proto=ESMTP,
>>>> helo=
>>>> 
>>>> the plan would be , set the ip output to i.e fifo pipe
>>>> and read the output via some daemon script/program to create an
>>>> iptables
>>>> ip reject rule with timestamp via ipset map, which expires auto
>>>> after
>>>> 24
>>>> h, this should fast as possible
>>>> 
>>>> so anybody with an example for filtering this ?
>>>> 
>>>> by the way
>>>> someone speculated rsyslog may able to do the whole job
>>>> filtering ip and do the ipset action in one step , is this true ?
>>>> i am not that far in the docs yet
>>> 
>>> I have to admit that I do not fully understand your question. But
>>> it has been
>>> a long and busy day today ;) Could you try to describe in two or
>>> three
>>> sentences what you actually want to achieve?
>> 
>> ok
>> 
>> this is a exmaple from the mail log
>> 
>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>> 
>> ( this is the relevant match part )
>> [190.24.212.146] blocked using zen.spamhaus.org;
>> 
>> 
>> from=, to=, proto=ESMTP,
>> helo=
>> 
>> i wann have filtered 190.24.212.146 only by rsyslog
>> and do this
>> echo +190.24.212.146 > /proc/net/xt_recent/SMTP
>> 
>> if possible directly all done from rsyslog
>> 
>> if not ,its enough to pipe fifo 190.24.212.146 only
>> i will script some daemon reading the fifo then
>> 
>> but as someother answered i think rsyslog could do it in one step
>> 
>> clear now?
> 
> Hi Robert,
> 
> maybe something like that would work (untested):
> 
> $template MyTemplate,"+%msg:R,ERE,1,BLANK:.*\[([0-9.]+)\].*--end%"
> if $msg contains 'blocked using zen.spamhaus.org' \
> then /proc/net/xt_recent/SMTP;MyTemplate
> 
> -Marc
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> 



Champ Clark III
(office) 904.253.7856
(mobile) 850.443.2440 
(SOC) 800.538.9357 ext 101
cclark at quadrantsec.com
www.quadrantsec.com


From robert at schetterer.org  Sat Nov 26 08:22:59 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Sat, 26 Nov 2011 08:22:59 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>
References: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>
Message-ID: <4ED093D3.7000201@schetterer.org>

Am 26.11.2011 00:50, schrieb Marc Schiffbauer:
> ----- Urspr?ngliche Mail -----
>> Am 25.11.2011 22:10, schrieb Rainer Gerhards:
>>>> -----Original Message-----
>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>>>> Sent: Friday, November 25, 2011 9:01 PM
>>>> To: rsyslog-users
>>>> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
>>>>
>>>> Hi @ll,
>>>> i am new to this list
>>>> and not very familar with rsyslog so sorry if my question was
>>>> asked/answered before
>>>>
>>>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>>>> i allready done
>>>> ( on suse )
>>>>
>>>> mail.info
>>>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>>>> if $msg contains 'blocked using zen.spamhaus.org' then
>>>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>>>
>>>> this works fine, but as i have a lot of spambots
>>>> fail2ban ist to slow with this allready filtered log
>>>>
>>>> so i wann have a second or combinated rsyslog rule which results
>>>> only in the ip
>>>>
>>>> a relevant log  entry example looks like this
>>>>
>>>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>>>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>>> [190.24.212.146] blocked using zen.spamhaus.org;
>>>> from=, to=,
>>>> proto=ESMTP,
>>>> helo=
>>>>
>>>> the plan would be , set the ip output to i.e fifo pipe
>>>> and read the output via some daemon script/program to create an
>>>> iptables
>>>> ip reject rule with timestamp via ipset map, which expires auto
>>>> after
>>>> 24
>>>> h, this should fast as possible
>>>>
>>>> so anybody with an example for filtering this ?
>>>>
>>>> by the way
>>>> someone speculated rsyslog may able to do the whole job
>>>> filtering ip and do the ipset action in one step , is this true ?
>>>> i am not that far in the docs yet
>>>
>>> I have to admit that I do not fully understand your question. But
>>> it has been
>>> a long and busy day today ;) Could you try to describe in two or
>>> three
>>> sentences what you actually want to achieve?
>>
>> ok
>>
>> this is a exmaple from the mail log
>>
>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>
>> ( this is the relevant match part )
>> [190.24.212.146] blocked using zen.spamhaus.org;
>>
>>
>> from=, to=, proto=ESMTP,
>> helo=
>>
>> i wann have filtered 190.24.212.146 only by rsyslog
>> and do this
>> echo +190.24.212.146 > /proc/net/xt_recent/SMTP
>>
>> if possible directly all done from rsyslog
>>
>> if not ,its enough to pipe fifo 190.24.212.146 only
>> i will script some daemon reading the fifo then
>>
>> but as someother answered i think rsyslog could do it in one step
>>
>> clear now?
> 
> Hi Robert,
> 
> maybe something like that would work (untested):
> 
> $template MyTemplate,"+%msg:R,ERE,1,BLANK:.*\[([0-9.]+)\].*--end%"
> if $msg contains 'blocked using zen.spamhaus.org' \
> then /proc/net/xt_recent/SMTP;MyTemplate
> 
> -Marc
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

Hi Marc,
its early in the morning here *g
my eyes miss some echo + in your example
but the rest looks very promising to me, thx

-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

From robert at schetterer.org  Sat Nov 26 08:32:05 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Sat, 26 Nov 2011 08:32:05 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <660C427E-0BE4-4AA4-BEF6-6BD2E09B2AA1@quadrantsec.com>
References: <7aa8de54-f724-4154-88c9-2ff646b21f22@zimbra>
	<660C427E-0BE4-4AA4-BEF6-6BD2E09B2AA1@quadrantsec.com>
Message-ID: <4ED095F5.7080800@schetterer.org>

Am 26.11.2011 01:29, schrieb Champ Clark III [Quadrant]:
> You could also use Sagan to trigger on the event,  extract the information (via liblognorm) and pass it to a external plug in (which fires iptables).  Pretty straight forward.   I plan on having snortsam integration into Sagan soon as well.

dont know sagan
only read some stuff on the web
but after all, i tried to solve the problem with fail2ban and ipset
it was to slow, guess
other like parsing stuff isnt much faster

i have currently 5 spambot cons per second left , after massive ip and
netblocking, and con limiting !

thats a lot of logging, so any solution must be as fast as possible
simple and lightweight, using pure rsyslog seems to me the fastest way
by log
analyse and create dynamic rejects

perhaps, there are more better ways on tcp level
but i am not ready to investigate time here/there yet

> 
> On Nov 25, 2011, at 6:50 PM, Marc Schiffbauer wrote:
> 
>> ----- Urspr?ngliche Mail -----
>>> Am 25.11.2011 22:10, schrieb Rainer Gerhards:
>>>>> -----Original Message-----
>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>>>> bounces at lists.adiscon.com] On Behalf Of Robert Schetterer
>>>>> Sent: Friday, November 25, 2011 9:01 PM
>>>>> To: rsyslog-users
>>>>> Subject: [rsyslog] mail log ip filter spamhaus rsyslog
>>>>>
>>>>> Hi @ll,
>>>>> i am new to this list
>>>>> and not very familar with rsyslog so sorry if my question was
>>>>> asked/answered before
>>>>>
>>>>> i wanna goal, filtering spamhaus rbl ips from postfix mail log
>>>>> i allready done
>>>>> ( on suse )
>>>>>
>>>>> mail.info
>>>>> -/var/log/mail.info;RSYSLOG_TraditionalFileFormat
>>>>> if $msg contains 'blocked using zen.spamhaus.org' then
>>>>> /var/log/spamhaus.log;RSYSLOG_TraditionalFileFormat
>>>>>
>>>>> this works fine, but as i have a lot of spambots
>>>>> fail2ban ist to slow with this allready filtered log
>>>>>
>>>>> so i wann have a second or combinated rsyslog rule which results
>>>>> only in the ip
>>>>>
>>>>> a relevant log  entry example looks like this
>>>>>
>>>>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>>>>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>>>> [190.24.212.146] blocked using zen.spamhaus.org;
>>>>> from=, to=,
>>>>> proto=ESMTP,
>>>>> helo=
>>>>>
>>>>> the plan would be , set the ip output to i.e fifo pipe
>>>>> and read the output via some daemon script/program to create an
>>>>> iptables
>>>>> ip reject rule with timestamp via ipset map, which expires auto
>>>>> after
>>>>> 24
>>>>> h, this should fast as possible
>>>>>
>>>>> so anybody with an example for filtering this ?
>>>>>
>>>>> by the way
>>>>> someone speculated rsyslog may able to do the whole job
>>>>> filtering ip and do the ipset action in one step , is this true ?
>>>>> i am not that far in the docs yet
>>>>
>>>> I have to admit that I do not fully understand your question. But
>>>> it has been
>>>> a long and busy day today ;) Could you try to describe in two or
>>>> three
>>>> sentences what you actually want to achieve?
>>>
>>> ok
>>>
>>> this is a exmaple from the mail log
>>>
>>> postfix/postscreen[32120]: NOQUEUE: reject: RCPT from
>>> [190.24.212.146]:57855: 550 5.7.1 Service unavailable; client
>>>
>>> ( this is the relevant match part )
>>> [190.24.212.146] blocked using zen.spamhaus.org;
>>>
>>>
>>> from=, to=, proto=ESMTP,
>>> helo=
>>>
>>> i wann have filtered 190.24.212.146 only by rsyslog
>>> and do this
>>> echo +190.24.212.146 > /proc/net/xt_recent/SMTP
>>>
>>> if possible directly all done from rsyslog
>>>
>>> if not ,its enough to pipe fifo 190.24.212.146 only
>>> i will script some daemon reading the fifo then
>>>
>>> but as someother answered i think rsyslog could do it in one step
>>>
>>> clear now?
>>
>> Hi Robert,
>>
>> maybe something like that would work (untested):
>>
>> $template MyTemplate,"+%msg:R,ERE,1,BLANK:.*\[([0-9.]+)\].*--end%"
>> if $msg contains 'blocked using zen.spamhaus.org' \
>> then /proc/net/xt_recent/SMTP;MyTemplate
>>
>> -Marc
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>>
> 
> 
> 
> Champ Clark III
> (office) 904.253.7856
> (mobile) 850.443.2440 
> (SOC) 800.538.9357 ext 101
> cclark at quadrantsec.com
> www.quadrantsec.com
> 
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/


-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

From a.piesk at gmx.net  Sat Nov 26 13:54:34 2011
From: a.piesk at gmx.net (Andreas Piesk)
Date: Sat, 26 Nov 2011 13:54:34 +0100
Subject: [rsyslog] rsyslog hangs at startup
In-Reply-To: <4EC6C04E.60301@gmx.net>
References: <4EADBB77.5040801@gmx.net>	<4EAE9DF0.8080703@gmx.net>	<4EAFBB9E.60805@gmx.net>	<9B6E2A8877C38245BFB15CC491A11DA72813F7@GRFEXC.intern.adiscon.com>	<4EB1B6BE.8050201@gmx.net>
	<4EC6C04E.60301@gmx.net>
Message-ID: <4ED0E18A.3040604@gmx.net>

On 18.11.2011 21:30, Andreas Piesk wrote:
> 
> Rainer, have you found some time to look at the issue?
> 
> what about the idea to use one mutex for both operations? i mean, wouldn't replacing the mutexes
> mutLoadUnload and mutObjGlobalOp with a single mutex, type PTHREAD_MUTEX_RECURSIVE, solve the problem?
> 

i fixed the problem with the attached patch against 5.8.6.
it may not be an optimal solution but at least rsyslog doesn't hang anymore at bootup.

regards,
-ap
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rsyslog-deadlock.patch
URL: 

From marc.schiffbauer at mightycare.de  Sat Nov 26 17:05:57 2011
From: marc.schiffbauer at mightycare.de (Marc Schiffbauer)
Date: Sat, 26 Nov 2011 17:05:57 +0100 (CET)
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog
In-Reply-To: <4ED093D3.7000201@schetterer.org>
Message-ID: <8eb8e235-4b42-4b2b-80ad-26f6a3466692@zimbra>

----- Urspr?ngliche Mail -----
> 
> Hi Marc,
> its early in the morning here *g
> my eyes miss some echo + in your example
> but the rest looks very promising to me, thx
> 

The "+" is at the bginning of the template and an extra echo is not needed because
rsyslog should directly write to the proc file.

-Marc

From robert at schetterer.org  Sat Nov 26 23:40:27 2011
From: robert at schetterer.org (Robert Schetterer)
Date: Sat, 26 Nov 2011 23:40:27 +0100
Subject: [rsyslog] mail log ip filter spamhaus  rsyslog / solved
In-Reply-To: <8eb8e235-4b42-4b2b-80ad-26f6a3466692@zimbra>
References: <8eb8e235-4b42-4b2b-80ad-26f6a3466692@zimbra>
Message-ID: <4ED16ADB.6080407@schetterer.org>

Am 26.11.2011 17:05, schrieb Marc Schiffbauer:
> ----- Urspr?ngliche Mail -----
>>
>> Hi Marc,
>> its early in the morning here *g
>> my eyes miss some echo + in your example
>> but the rest looks very promising to me, thx
>>
> 
> The "+" is at the bginning of the template and an extra echo is not needed because
> rsyslog should directly write to the proc file.
> 
> -Marc
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

Hi Marc, as offlist allready posted
this i.e works on suse, so for others as idea to list

mail.info
-/var/log/mail.info;RSYSLOG_TraditionalFileFormat
$template MyTemplate,"%msg:R,ERE,1,BLANK:.*\[([0-9.]+)\].*--end%\n"
if $msg contains 'blocked using zen.spamhaus.org' then
|/tmp/testpipe;MyTemplate



!/bin/bash

while true
do
if read line  /proc/net/xt_recent/SPAM
    fi
done


it isnt perfect like this, but my basics problem is solved , thx for helping
-- 
Best Regards

MfG Robert Schetterer

Germany/Munich/Bavaria

From rgerhards at hq.adiscon.com  Sun Nov 27 18:27:31 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Sun, 27 Nov 2011 18:27:31 +0100
Subject: [rsyslog] FYI: funding rsyslog development
In-Reply-To: 
References: <9B6E2A8877C38245BFB15CC491A11DA72814F9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA7281513@GRFEXC.intern.adiscon.com>
	
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281532@GRFEXC.intern.adiscon.com>

> syslog-ng uses the 'open sore' approach where some features are not
> made
> available in the free version, only in the proprietary version.
> 
> how are you thinking that this would work for rsyslog?

I have been preparing today for some Adiscon-internal discussions next week.
Let's assume for a moment we would follow a similar model. I obviously made
one mistake with rsyslog license files. I mentioned several times that I
consider plugins as separate projects [1] and don't have any concerns if
commercial plugins exist (not only from Adiscon but others as well). HOWEVER,
I did not add the necessary exemption into rsyslog's GPLv3 license. So I
created a problem: even if I can persuade folks to move to that open core
model, we could still not do so without rewriting parts of rsyslog so that
Adiscon holds the complete copyright. This obviously is a non-brainer because
under this provisions it sounds much more attractive to move to the
dual-license ability or spent the work on some other platform. So I think it
will be very hard to sell this idea.

One solution is to add the exemption *now*. However, this requires agreement
of at least those folks that had major contributions. It may probably even
possible to do this without explicit permission from everyone, as long as
there is no objection. The reason is that we always stated that commercial
plugins are OK with our interpretation. At least under German law, this is
nearly as good as if the exemption had been written explicitly.

Feeback would be deeply appreciate. This will be an important point in the
internal discussions.

Thanks again for all feedback up to now, I will of course keep you posted.

Rainer

[1] http://www.rsyslog.com/doc/dev_oplugins.html

From a.chapellon at horoa.net  Tue Nov 29 00:46:11 2011
From: a.chapellon at horoa.net (Alexandre Chapellon)
Date: Tue, 29 Nov 2011 00:46:11 +0100
Subject: [rsyslog] timereported timezone?
In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
References: <4ECE848B.6070203@horoa.net>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
Message-ID: <4ED41D43.70307@horoa.net>



Le 25/11/2011 07:21, Rainer Gerhards a ?crit :
>
>> -----Original Message-----
>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>> Sent: Friday, November 25, 2011 4:57 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] timereported timezone?
>>
>> On Thu, 24 Nov 2011, Alexandre Chapellon wrote:
>>
>>> I quickly read the rfc3164 and found this:
>>>
>>> "
>>>
>>> ...
>>>
>>>    For implementers that do choose to construct syslog messages with
>> the
>>>    RECOMMENDED format, the following guidance is offered.
>>>
>>>          If the originally formed message has a TIMESTAMP in the
>> HEADER
>>>          part, then it SHOULD be the local time of the device within
>> its
>>>          timezone...
>>>
>>> "
>>>
>>> So I guess the timezone of the device is not store in the syslog
>> message by
>>> itself. Right?
>> correct.
>>
>>> Is there a way to include the local timezonein the header? or at
>> least in the
>>> message? (I would really prefer in the header)
>> you can't do it with the RFC3164 format, I belive that the latest RFC
>> does
>> have a timestamp format that includes the timezone, but I don't think
>> that
>> it's a good idea to use local time in any case.
> RFC5424 provides high-precision timestamps with timezone information. Rsyslog
> does so for many years. It is a mayor frustration for me that all distros
> turn off that feature (which is enabled by default).
>
> But of course that doesn't help if the original sender does not provide TZ
> information.
> rainer
OK great, I only use rsyslog all along... So, I guess I have to 
recompile the whole syslog forwarding chain?
What will happen for message not carying TZ when they arrive on a 
recompiled rsyslog server? The locale TZ of the final rsyslog 
destination server will apply?

> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/

-- 


Alexandre Chapellon

Ing?nierie des syst?mes open sources et r?seaux.
Follow me on twitter: @alxgomz 


From david at lang.hm  Tue Nov 29 05:19:40 2011
From: david at lang.hm (david at lang.hm)
Date: Mon, 28 Nov 2011 20:19:40 -0800 (PST)
Subject: [rsyslog] timereported timezone?
In-Reply-To: <4ED41D43.70307@horoa.net>
References: <4ECE848B.6070203@horoa.net>
	
	<9B6E2A8877C38245BFB15CC491A11DA7281512@GRFEXC.intern.adiscon.com>
	<4ED41D43.70307@horoa.net>
Message-ID: 

On Tue, 29 Nov 2011, Alexandre Chapellon wrote:

>
> Le 25/11/2011 07:21, Rainer Gerhards a ?crit :
>> 
>>> -----Original Message-----
>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
>>> bounces at lists.adiscon.com] On Behalf Of david at lang.hm
>>> Sent: Friday, November 25, 2011 4:57 AM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] timereported timezone?
>>> 
>>> On Thu, 24 Nov 2011, Alexandre Chapellon wrote:
>>> 
>>>> I quickly read the rfc3164 and found this:
>>>> 
>>>> "
>>>> 
>>>> ...
>>>>
>>>>    For implementers that do choose to construct syslog messages with
>>> the
>>>>    RECOMMENDED format, the following guidance is offered.
>>>>
>>>>          If the originally formed message has a TIMESTAMP in the
>>> HEADER
>>>>          part, then it SHOULD be the local time of the device within
>>> its
>>>>          timezone...
>>>> 
>>>> "
>>>> 
>>>> So I guess the timezone of the device is not store in the syslog
>>> message by
>>>> itself. Right?
>>> correct.
>>> 
>>>> Is there a way to include the local timezonein the header? or at
>>> least in the
>>>> message? (I would really prefer in the header)
>>> you can't do it with the RFC3164 format, I belive that the latest RFC
>>> does
>>> have a timestamp format that includes the timezone, but I don't think
>>> that
>>> it's a good idea to use local time in any case.
>> RFC5424 provides high-precision timestamps with timezone information. 
>> Rsyslog
>> does so for many years. It is a mayor frustration for me that all distros
>> turn off that feature (which is enabled by default).
>> 
>> But of course that doesn't help if the original sender does not provide TZ
>> information.
>> rainer
> OK great, I only use rsyslog all along... So, I guess I have to recompile the 
> whole syslog forwarding chain?
> What will happen for message not carying TZ when they arrive on a recompiled 
> rsyslog server? The locale TZ of the final rsyslog destination server will 
> apply?

If you are using rsyslog from the sending server on, then all you need to 
do is to change the format that you use to send the message to one that 
includes the high precision timestamp

David Lang


From nathans at aconex.com  Tue Nov 29 05:22:41 2011
From: nathans at aconex.com (Nathan Scott)
Date: Tue, 29 Nov 2011 15:22:41 +1100 (EST)
Subject: [rsyslog] system not recovering after network issue
In-Reply-To: <0a284d16-60f4-401d-aeea-7d59e8cb9d51@wsrmckee>
Message-ID: <188844487.3061.1322540561655.JavaMail.root@acxmail-au2.aconex.com>

Just a quick follow-up to Rodney's mail, we've been
running with the attached patch to counter this issue
for a little while now, and so far without a repeat
incident.

cheers.

----- Original Message -----
> Hello,
> 
> With the following config:
> ...
> # Send all logs onto the local relay
> #
> *.*;syslog.!=info @@log1;RSYSLOG_ForwardFormat
> $ActionExecOnlyWhenPreviousIsSuspended on
> & @@log2
> & /var/spool/rsyslog-buffer
> $ActionExecOnlyWhenPreviousIsSuspended off
> 
> 
> We have systems stuck in a connect state that do not appear to be
> recovering:
> 
> # strace -p 32318
> Process 32318 attached - interrupt to quit
> connect(1, {sa_family=AF_INET, sin_port=htons(514),
> sin_addr=inet_addr("log2")}, 16 
> Process 32318 detached
> 
> 
> Loaded symbols for /lib64/libnss_dns.so.2
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/rsyslog/lmnsd_ptcp.so...done.
> Loaded symbols for /lib64/rsyslog/lmnsd_ptcp.so
> 0x000000319ca0cf2b in connect () from /lib64/libpthread.so.0
> #0 0x000000319ca0cf2b in connect () from /lib64/libpthread.so.0
> #1 0x00002aaaab0d1d65 in Connect (pNsd=0x2aaaac8abe10, family= optimized out>, port=, host= out>) at nsd_ptcp.c:684
> #2 0x000000000040ff29 in TCPSendInit ()
> #3 0x0000000000410038 in doTryResume ()
> #4 0x0000000000436d30 in actionTryResume ()
> #5 0x0000000000437393 in submitBatch ()
> #6 0x0000000000437978 in processBatchMain ()
> #7 0x0000000000435896 in doSubmitToActionQBatch ()
> #8 0x00000000004361f9 in doSubmitToActionQNotAllMarkBatch ()
> #9 0x00000000004325b8 in processBatchDoActions ()
> #10 0x000000000041d5d8 in llExecFunc ()
> #11 0x0000000000432933 in processBatch ()
> #12 0x00000000004319de in processBatchDoRules ()
> #13 0x000000000041d5d8 in llExecFunc ()
> #14 0x0000000000431f04 in processBatch ()
> #15 0x000000000040b5cf in msgConsumer ()
> #16 0x0000000000430dcd in ConsumerReg ()
> #17 0x000000000042a51c in wtiWorker ()
> #18 0x000000000042a136 in wtpWorker ()
> #19 0x000000319ca062f7 in start_thread () from /lib64/libpthread.so.0
> #20 0x000000319c2d1b6d in clone () from /lib64/libc.so.6
> 
> $ sudo /usr/sbin/lsof -p 20064
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> rsyslogd 20064 root cwd DIR 8,7 4096 2 /
> rsyslogd 20064 root rtd DIR 8,7 4096 2 /
> rsyslogd 20064 root txt REG 8,7 441537 1310792 /sbin/rsyslogd
> rsyslogd 20064 root mem REG 8,7 134400 885017 /lib64/ld-2.5.so
> rsyslogd 20064 root mem REG 8,7 1699912 885018 /lib64/libc-2.5.so
> rsyslogd 20064 root mem REG 8,7 23360 885019 /lib64/libdl-2.5.so
> rsyslogd 20064 root mem REG 8,7 141440 885023 /lib64/libpthread-2.5.so
> rsyslogd 20064 root mem REG 8,7 53448 885024 /lib64/librt-2.5.so
> rsyslogd 20064 root mem REG 8,6 85928 164425 /usr/lib64/libz.so.1.2.3
> rsyslogd 20064 root mem REG 8,7 92736 884796 /lib64/libresolv-2.5.so
> rsyslogd 20064 root mem REG 8,7 53880 884764
> /lib64/libnss_files-2.5.so
> rsyslogd 20064 root mem REG 8,7 23632 884762 /lib64/libnss_dns-2.5.so
> rsyslogd 20064 root mem REG 8,7 93320 885005
> /lib64/rsyslog/lmnsd_ptcp.so
> rsyslogd 20064 root mem REG 8,7 75802 884921 /lib64/rsyslog/lmnet.so
> rsyslogd 20064 root mem REG 8,7 1295631 884806
> /lib64/rsyslog/imuxsock.so
> rsyslogd 20064 root mem REG 8,7 81914 884794 /lib64/rsyslog/imklog.so
> rsyslogd 20064 root mem REG 8,7 57594 884804 /lib64/rsyslog/imudp.so
> rsyslogd 20064 root mem REG 8,7 37373 884801
> /lib64/rsyslog/impstats.so
> rsyslogd 20064 root mem REG 8,7 90803 884994
> /lib64/rsyslog/lmnetstrms.so
> rsyslogd 20064 root mem REG 8,7 35770 884873
> /lib64/rsyslog/lmtcpclt.so
> rsyslogd 20064 root 0u unix 0xffff81031e95e0c0 876257540 /dev/log
> rsyslogd 20064 root 1u IPv4 1000073229 TCP
> app1.lhr.acx:43293->192.168.132.143:shell (SYN_SENT)
> rsyslogd 20064 root 2r 0000 0,10 0 876257542 eventpoll
> rsyslogd 20064 root 3u IPv6 876257538 UDP *:syslog
> rsyslogd 20064 root 4u IPv4 876257539 UDP *:syslog
> rsyslogd 20064 root 8r REG 0,3 0 4026531849 /proc/kmsg
> 
> 
> The suspected code is in nsd_ptcp.c as it does not appear to allow for
> a timeout with a NODELAY or other mechanism on the connect.
> 
> if((pThis->sock = socket(res->ai_family, res->ai_socktype,
> res->ai_protocol)) == -1) {
> ABORT_FINALIZE(RS_RET_IO_ERROR);
> }
> 
> if(connect(pThis->sock, res->ai_addr, res->ai_addrlen) != 0) {
> ABORT_FINALIZE(RS_RET_IO_ERROR);
> }
> 
> 
> 
> Rgds
> Rodney
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com

-- 
Nathan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rsyslog-connect-flags.patch
Type: text/x-patch
Size: 1026 bytes
Desc: not available
URL: 

From rgerhards at hq.adiscon.com  Tue Nov 29 13:05:40 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Tue, 29 Nov 2011 13:05:40 +0100
Subject: [rsyslog] trusted (fields/properties) now also in rsyslog
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281549@GRFEXC.intern.adiscon.com>

Hi all,

I thought I'd buy in on one of journald's better ideas, here are the details:

http://blog.gerhards.net/2011/11/trusted-properties-in-rsyslog.html

I will see that I can release today, but would like to merge into v6, if
possible... Expect an announcement soon. I'd also like to know what you think
about this feature.

Thanks,
Rainer

From friedl at hq.adiscon.com  Tue Nov 29 17:45:00 2011
From: friedl at hq.adiscon.com (Florian Riedl)
Date: Tue, 29 Nov 2011 17:45:00 +0100
Subject: [rsyslog] rsyslog 5.9.4 (devel) released
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7281565@GRFEXC.intern.adiscon.com>

Hi all,

This release provides support for "trusted properties", which may enhance
overall system security. This is a new concept and feedback on it is
appreciated. For more details on trusted properties, please visit

http://www.rsyslog.com/what-are-trusted-properties/

or Rainer's blog post with some more background about trusted properties:

http://blog.gerhards.net/2011/11/trusted-properties-in-rsyslog.html

In addition to this feature, we have reduced dependency on libgcrypt and
fixed some bugs.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-4-v5-devel/

Download:

http://www.rsyslog.com/rsyslog-5-9-4-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

From marcin at mejor.pl  Tue Nov 29 23:04:45 2011
From: marcin at mejor.pl (=?ISO-8859-2?Q?Marcin_Miros=B3aw?=)
Date: Tue, 29 Nov 2011 23:04:45 +0100
Subject: [rsyslog] system not recovering after network issue
In-Reply-To: <188844487.3061.1322540561655.JavaMail.root@acxmail-au2.aconex.com>
References: <188844487.3061.1322540561655.JavaMail.root@acxmail-au2.aconex.com>
Message-ID: <4ED556FD.1050106@mejor.pl>

W dniu 2011-11-29 05:22, Nathan Scott pisze:
> Just a quick follow-up to Rodney's mail, we've been
> running with the attached patch to counter this issue
> for a little while now, and so far without a repeat
> incident.

Hello!
Thank your for this patch! I'm still testing it but as for know it looks
works correctly for me. It solves problem avoiding me to upgrade rsyslog
to version higher than 5.6.5.
I hope this patch will be included in tree.
Thanks!


From marcin at mejor.pl  Wed Nov 30 09:33:48 2011
From: marcin at mejor.pl (=?UTF-8?B?TWFyY2luIE1pcm9zxYJhdw==?=)
Date: Wed, 30 Nov 2011 09:33:48 +0100
Subject: [rsyslog] system not recovering after network issue
In-Reply-To: <4ED556FD.1050106@mejor.pl>
References: <188844487.3061.1322540561655.JavaMail.root@acxmail-au2.aconex.com>
	<4ED556FD.1050106@mejor.pl>
Message-ID: <4ED5EA6C.8050903@mejor.pl>

W dniu 29.11.2011 23:04, Marcin Miros?aw pisze:
> Hello!
> Thank your for this patch! I'm still testing it but as for know it looks
> works correctly for me. It solves problem avoiding me to upgrade rsyslog
> to version higher than 5.6.5.
> I hope this patch will be included in tree.

Hi,
my joy was premature. My problem wasn't fixed with those patch:(
Regards!

From rgerhards at hq.adiscon.com  Wed Nov 30 17:42:12 2011
From: rgerhards at hq.adiscon.com (Rainer Gerhards)
Date: Wed, 30 Nov 2011 17:42:12 +0100
Subject: [rsyslog] Serious Problems with Syslog...
Message-ID: <9B6E2A8877C38245BFB15CC491A11DA728156F@GRFEXC.intern.adiscon.com>

...were named in the journald paper. I have now taken time to look in-depth
at them. In conclusion, I don't see any problem so serious that the whole
logging system must be replaced. Please have a look at the analysis at

http://blog.gerhards.net/2011/11/serious-syslog-problems.html

I'd also appreciate if you could forward this link to interested parties.

Thanks,
Rainer