From rgerhards at hq.adiscon.com Tue Sep 1 10:51:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 10:51:35 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> > As a side-note: I think that my UDP message loss may partly be related > to DNS > resolution. I will try this in a lab tomorrow. But I still think a lot > of > packets never leave the source system. This may be related to the > virtual > environment I am currently using for the lab. I hope to be able to > generate > the traffic by a program, because that offers me the flexibility (now > and in > the future) to test complex messages scenarios (what, granted, does not > help > if it does not expose the problem...). Very interesting - I just did a couple of tests with UDP and various DNS resolution settings. The message loss I see is definitely related to DNS resolution. This is especially interesting as in my lab setup there should be no need to do more than the initial query. This points into some area that either is buggy or needs to be optimized. When I turn off DNS resolution, I have far fewer lost message. Still, there is between 1% and 10% loss for reasonable high traffic, but that is OK from my expectations given the lab environment I use. With DNS resolution, I have > 90% loss, and this difference is clearly not acceptable. I will look into this issue, but will try to find the segfault first (better not change the environment so that the bug moves to some other region). In the light of this, I'll probably rerun some of my tests today without reverse DNS resolution - the higher rate will hopefully trigger the bug in my lab. Rainer From rgerhards at hq.adiscon.com Tue Sep 1 12:26:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 12:26:31 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 01, 2009 10:52 AM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 / UDP message loss > > > As a side-note: I think that my UDP message loss may partly be > related > > to DNS > > resolution. I will try this in a lab tomorrow. But I still think a > lot > > of > > packets never leave the source system. This may be related to the > > virtual > > environment I am currently using for the lab. I hope to be able to > > generate > > the traffic by a program, because that offers me the flexibility (now > > and in > > the future) to test complex messages scenarios (what, granted, does > not > > help > > if it does not expose the problem...). > > Very interesting - I just did a couple of tests with UDP and various > DNS > resolution settings. The message loss I see is definitely related to > DNS > resolution. This is especially interesting as in my lab setup there > should be > no need to do more than the initial query. This points into some area > that > either is buggy or needs to be optimized. ... my simplistic requery-avoidance logic does not take the source port into account. So a requery is also done if the host is the same as before, but the port changes. Thus the difference. Needs to be optimized ;) While this does not point to an obvious bug, I'll still try to get a segfault without DNS resolution. Rainer From mikel at irontec.com Tue Sep 1 14:20:35 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 01 Sep 2009 14:20:35 +0200 Subject: [rsyslog] milliseconds timestamp Message-ID: <4A9D1193.4090806@irontec.com> hi Some news about this? http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html Maybe with a bounty? thanks From rgerhards at hq.adiscon.com Tue Sep 1 14:25:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 14:25:12 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD4@GRFEXC.intern.adiscon.com> Hi, Andre has just gone on vacation, expect a real answer in two weeks ;) But I don't think he had time to look at this (too many paid projects in the way...). So a bounty may be useful ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 01, 2009 2:21 PM > To: rsyslog-users > Subject: [rsyslog] milliseconds timestamp > > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Tue Sep 1 18:59:25 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Tue, 01 Sep 2009 18:59:25 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> Message-ID: <4A9D52ED.4090103@redhat.com> On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas From rgerhards at hq.adiscon.com Tue Sep 1 19:55:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:55:57 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001501ca2b2d$7e16309d$100013ac@intern.adiscon.com> Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 1 19:58:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:58:56 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001601ca2b2d$e8e76725$100013ac@intern.adiscon.com> I was too quick. I should have said "backported with ease **to v4**". v2 obviously is so outdated, that this will require a totally different effort. ----- Urspr?ngliche Nachricht ----- Von: "Rainer Gerhards" An: "rsyslog-users" Gesendet: 01.09.09 19:56 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Sep 2 01:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 16:06:17 -0700 (PDT) Subject: [rsyslog] -cX command line option Message-ID: with version 3+ do we really need to change the X in this option? if you run v5 with -c4 is it really going to do something different with the config file than if you use -c5? yes, there are new config options in the newer versions, and once in a while some depriciated config options stop working, but does changing from -c3 to -c4 to -c5 actually fix any of these? in my testing I keep switching between the v4 series and the v5 series and having to change the startup to give the correct -c flag has tripped me up more than once. it would also be helpful if rsyslog would spit out errors about unknown config files (either to the console or as syslog messages) without needing to be in debug mode. it may that it tries to do this, but I don't see them (either with the debian startup scripts or when starting it directly on the command line) David Lang From david at lang.hm Wed Sep 2 03:11:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 18:11:56 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251715849.4897.13.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: I got a core file with 4.2.0 I did git checkout -f v4.2.0 configure --enable-imfile and installed the result. I will go through the core file either later tonight or in the morning. in this case it did take a while for it to die. (over an hour) David Lang On Mon, 31 Aug 2009, Rainer Gerhards wrote: > Date: Mon, 31 Aug 2009 12:50:49 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > On Fri, 2009-08-28 at 14:55 -0700, david at lang.hm wrote: >> On Fri, 28 Aug 2009, Rainer Gerhards wrote: >>> Also, it would be good if you could --enable-rtinst --enable-debug and try >>> out that version on your machine. I am a bit concerned about the speed of the >>> resulting executable, it may be too slow. You do not need to run it in debug >>> mode itself. These option (especially--enable-debug) will activate in-depth >>> runtime checks (assert, will abort when something wrong happens) and my hope >>> is that they will catch the bug closer to the root cause. If so, I would need >>> the gdb abort info (actually enabling debug output would be an option some >>> time later). >>> >>> Please let me know what would be OK with you. >> >> I will give this a try. >> >> I was going to suggest that since we have the message getting corrupted it >> may make sense to make a temporary branch that has multiple message >> buffers and at various times through the message processing it makes a >> copy of the emssage to the buffer. when the system crashes I will be able >> to look at the core and see where the message is getting corrupted. > > David, I fear it is even more complicated than that. It looks like not > only the message got corrupted but the message object itself. There are > already two copies of some of the message elements, and they also look > inconsistent - except, if we really had a null message, that is one with > no content at all (and generating a message object from a null message, > I think, would be a bug in itself - but I am sure there are no such > messages in your actual traffic). If you think there could be a real > null message, I'd follow that path (will probably do so in any case...). > > I think that what really happens is that some part of the code runs > wild, thus invalidating some random part of the main memory. At some > times, it hits queue structures (or the message object that is held by > them) and if so, we will see the abort you experience. With that > scenario, duplicating the message buffer does not really help, because > looking at the corrupted message object would not provide any additional > information. > > However, if that's easy enough to reproduce, it would probably be good > if you could send me the core analysis (the backtrace and the print > statements) from a few (five maybe?) independent aborts. Maybe they show > a pattern. It would probably best to send them via private mail, as I am > not sure if they disclose more than they should. > >> >> I will see about doing a tcpdump at the time that I do this and send it to >> you (I'll need to check with management, but since we have a contract in >> place for other reasons I think we can do this) >> > > That would probably be a good thing. I've made some progress with my > testing tool, and I have created a basic version right now. Probably not > good enough to mimic your traffic pattern, but closer. I am doing a test > run for quite some time now, unfortunately so far without abort. > > Note that I run into the trouble with UDP - even though I've put some > one-ms sleeps into the code, I lose a lot of messages, as it looks even > before they hit the wire. It's always real trobulesome to test with > UDP... > > Rainer >> I can't do this late on a friday, but I should be able to do this monday >> afternoon. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 12:14:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:14:21 +0200 Subject: [rsyslog] abort in 4.2.1 In-Reply-To: References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: <1251886461.5821.8.camel@rgf11> David, thanks for this test. The outcome is obviously other than I hoped/expected, but that makes it very useful. Obviously I have been looking for the wrong root cause. Any abort information you can provide would be useful. Even more useful would be if you could try out some earlier releases. Not sure if that is possible from a feature point of view. If it is, I would appreciate if you could give v3-stable a try and, if and only if that fails, too, checkout v3.18.6 and try that one. The 3.18.6 is the version that Debian ships and so I know it has a lot of testers and received a lot of bug-finding attention (I thankfully receive lots of very qualified bug reports from the Debian community :)). Please let me know what is possible. In any case, the 4.2.0 failure even more points to environment-specific problems. Rainer On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: > I got a core file with 4.2.0 > > I did git checkout -f v4.2.0 configure --enable-imfile and installed the > result. > > I will go through the core file either later tonight or in the morning. > > in this case it did take a while for it to die. (over an hour) > > David Lang From rgerhards at hq.adiscon.com Wed Sep 2 12:23:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:23:02 +0200 Subject: [rsyslog] -cX command line option In-Reply-To: References: Message-ID: <1251886982.5821.17.camel@rgf11> On Tue, 2009-09-01 at 16:06 -0700, david at lang.hm wrote: > with version 3+ do we really need to change the X in this option? > if you run v5 with -c4 is it really going to do something different with > the config file than if you use -c5? > > yes, there are new config options in the newer versions, and once in a > while some depriciated config options stop working, but does changing from > -c3 to -c4 to -c5 actually fix any of these? The -cX is more a vehicle to change things like *defaults*, that is something that breaks existing configurations. So far, there is no difference between v4 and v5 in this regard. However, I would not like to give up this vehicle. That would actually force me to never change any defaults. > > in my testing I keep switching between the v4 series and the v5 series and > having to change the startup to give the correct -c flag has tripped me up > more than once. > > it would also be helpful if rsyslog would spit out errors about unknown > config files (either to the console or as syslog messages) without needing > to be in debug mode. The current versions already does this. I think they go to stderr (maybe stdout). > > it may that it tries to do this, but I don't see them (either with the > debian startup scripts or when starting it directly on the command line) > I could offer the follwing solution for what you describe: I could permit (in newer v3/v4 builds) to specify a higher version (-c5) and only sending an alert. Doing so, of course, means "I know what I do and I can live with any consequences from it" what should be fine for your use case. Please let me know if that would be helpful for you. Rainer From tbergfeld at hq.adiscon.com Wed Sep 2 14:41:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 2 Sep 2009 14:41:49 +0200 Subject: [rsyslog] rsyslog 4.4.1 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDDF@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.1., a member of the v4-development branch. This is a bug-fixing release, providing some important fixes for issues that have only been detected after the beta phase. Some of them are serious (like a segfault when UDP messageforwarding is activated), so users of 4.4.0 are urged to upgrade to this release. Have a look at the change log to see all new features included in this release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-173.phtml Changelog: http://www.rsyslog.com/Article398.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From mikel at irontec.com Wed Sep 2 14:52:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 14:52:02 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9D1193.4090806@irontec.com> References: <4A9D1193.4090806@irontec.com> Message-ID: <4A9E6A72.8080202@irontec.com> Ok, I will comunicate you if we decide. Is the development of phplogcon frezzed? the last version is of January 27 ... Thanks Mikel Jimenez wrote: > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > From rgerhards at hq.adiscon.com Wed Sep 2 14:56:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 14:56:34 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 2:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of January > 27 ... Definitely not, it is active. But it looks like the web site did not receive proper attention. I'll check what's going on... See the git log for what's going on: http://git.adiscon.com/?p=phplogcon.git;a=summary The pace of changes is somewhat lower than in the initial phase, because there have been more pressing projects. But I have talked with Andre over big reporting features, which he will (hopefully) be able to tackle once he is back from his vacation. Rainer From mikel at irontec.com Wed Sep 2 15:00:57 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 15:00:57 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> Message-ID: <4A9E6C89.4060309@irontec.com> Ahhh!! Ok Ok I see that it is active... so in near future the web page would be syncronized with the real state of the development? I usually use the web for news about phplogcon. (www.phplogcon.org) Thanks Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Wednesday, September 02, 2009 2:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Ok, I will comunicate you if we decide. >> >> Is the development of phplogcon frezzed? the last version is of January >> 27 ... >> > > Definitely not, it is active. But it looks like the web site did not receive > proper attention. I'll check what's going on... > > See the git log for what's going on: > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > The pace of changes is somewhat lower than in the initial phase, because > there have been more pressing projects. But I have talked with Andre over big > reporting features, which he will (hopefully) be able to tackle once he is > back from his vacation. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 15:03:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 15:03:46 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> <4A9E6C89.4060309@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ahhh!! Ok Ok > > I see that it is active... so in near future the web page would be > syncronized with the real state of the development? > > I usually use the web for news about phplogcon. (www.phplogcon.org) > Thanks I've already pinged the web folks. I agree, I also go to the sites. I think there also have been no release annoucements (actually my primary source of new release info). Interestingly, I just saw that freshmeat has announcements: http://freshmeat.net/projects/phplogcon/ ... strange ;) Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Wednesday, September 02, 2009 2:52 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] milliseconds timestamp > >> > >> Ok, I will comunicate you if we decide. > >> > >> Is the development of phplogcon frezzed? the last version is of > January > >> 27 ... > >> > > > > Definitely not, it is active. But it looks like the web site did not > receive > > proper attention. I'll check what's going on... > > > > See the git log for what's going on: > > > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > > > The pace of changes is somewhat lower than in the initial phase, > because > > there have been more pressing projects. But I have talked with Andre > over big > > reporting features, which he will (hopefully) be able to tackle once > he is > > back from his vacation. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Wed Sep 2 16:45:39 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 11:45:39 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" Message-ID: <4A9E8513.8020107@swiss-wireless.com.ar> Hello I'm pretty new to rsyslog. I know that you can specifiy 2 server for remote logging which will be handled "independent" i.e. rsyslog will log to the 2 server in parallel. What I want is a primary rsyslog server and a secondary rsyslog server and only if the primary is not avaiable the secondary should be used. Is this possible with rsyslog? Thanks for any hints Beat From rgerhards at hq.adiscon.com Wed Sep 2 19:06:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 19:06:14 +0200 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" References: <4A9E8513.8020107@swiss-wireless.com.ar> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Beat Meier > Sent: Wednesday, September 02, 2009 4:46 PM > To: rsyslog-users > Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with > "priority" > > Hello > > I'm pretty new to rsyslog. > I know that you can specifiy 2 server for remote logging which will be > handled "independent" > i.e. rsyslog will log to the 2 server in parallel. > What I want is a primary rsyslog server and a secondary rsyslog server > and only if the primary > is not avaiable the secondary should be used. Is this possible with > rsyslog? > > Thanks for any hints > > Beat > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 2 19:55:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 2 Sep 2009 10:55:32 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251886461.5821.8.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> <1251886461.5821.8.camel@rgf11> Message-ID: On Wed, 2 Sep 2009, Rainer Gerhards wrote: > David, > > thanks for this test. The outcome is obviously other than I > hoped/expected, but that makes it very useful. Obviously I have been > looking for the wrong root cause. > > Any abort information you can provide would be useful. Even more useful > would be if you could try out some earlier releases. Not sure if that is > possible from a feature point of view. > > If it is, I would appreciate if you could give v3-stable a try and, if > and only if that fails, too, checkout v3.18.6 and try that one. The > 3.18.6 is the version that Debian ships and so I know it has a lot of > testers and received a lot of bug-finding attention (I thankfully > receive lots of very qualified bug reports from the Debian > community :)). > > Please let me know what is possible. In any case, the 4.2.0 failure even > more points to environment-specific problems. I haven't gone back to the 3.x series, but I did several more runs with 4.2.0 doing the folloiwng killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r -h ; mv /core /core-4.2.0-10 I have several complete steps, as well as several partial sets of data. I will gzip them and attempt to send them to you directly. David Lang > Rainer > > On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: >> I got a core file with 4.2.0 >> >> I did git checkout -f v4.2.0 configure --enable-imfile and installed the >> result. >> >> I will go through the core file either later tonight or in the morning. >> >> in this case it did take a while for it to die. (over an hour) >> >> David Lang > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbe_ml at swiss-wireless.com.ar Wed Sep 2 22:18:09 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 17:18:09 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> References: <4A9E8513.8020107@swiss-wireless.com.ar> <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> Message-ID: <4A9ED301.2020500@swiss-wireless.com.ar> Thanks Rainer That's exactly what I looked for. One more question I use templates of the form: $template DynFileAuth,"/var/log/%HOSTNAME%/auth.log" $template DynFileSyslog,"/var/log/%HOSTNAME%/syslog" $template DynFileCron,"/var/log/%HOSTNAME%/cron.log" How can I use variables to replace the path /var/log So I can use something like: path="/var/log" $template DynFileAuth,"$path/%HOSTNAME%/auth.log" $template DynFileSyslog,"$path/%HOSTNAME%/syslog" $template DynFileCron,"$path/%HOSTNAME%/cron.log" Is the template way the only one? I think that template is expanded at runtime, isn't it? Is there a variable method that is expanded when daemon starts, for efficiency? I have nothing found in the wiki with search, no in the docu index, nor the man page of rsyslog.conf. Greetings and thanks Beat Rainer Gerhards wrote: >please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > > > From mbe_ml at swiss-wireless.com.ar Thu Sep 3 00:19:19 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 19:19:19 -0300 Subject: [rsyslog] Which librelp does work with rsyslog-3.18.6 ? Message-ID: <4A9EEF67.20502@swiss-wireless.com.ar> Hello I'm using rsyslog 3.18.6 on debian 4.0 (backport of rsyslog) There is no backport of a newer version with librelp support :-( so I downloaded librelp-0.1.3 and compiled it on debian 4.0. This will install librelp.so I have moved this to /usr/lib/rsyslog but rsyslog is complaining that he cannot find imrelp.so. I have configured the module imrelp as noted some ware else. I have now seen that debian version of relp (for debian 5.0) has imrelp.so and omrelp.so defined. Has the name of the shared object changed from an old release to 0.1.3? Nor renaming librelp.so nor changing module name did work. (renaming library results in an undedfined symbol: modInit) Which release of lbrelp can I use with rsyslog V3.18.6 or can I use librelp only with rsyslog-4.4? Greetings and thanks Beat From joe at joetify.com Thu Sep 3 05:11:28 2009 From: joe at joetify.com (Joe Williams) Date: Wed, 2 Sep 2009 20:11:28 -0700 Subject: [rsyslog] case sensitivity in templates Message-ID: <20090902201128.507f2449@der-dieb> Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 07:47:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 07:47:14 +0200 Subject: [rsyslog] case sensitivity in templates Message-ID: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> You need to check the property replacer documentation. There are options for case conversion. I don't know the exact syntax out of my head, but it is along the lines of %field:::ucase%. Hth rainer ----- Urspr?ngliche Nachricht ----- Von: "Joe Williams" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 05:19 Betreff: [rsyslog] case sensitivity in templates Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 3 12:23:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 12:23:45 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Hi David, > I haven't gone back to the 3.x series, but I did several more runs with > 4.2.0 doing the folloiwng > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r > -h ; mv /core /core-4.2.0-10 > > I have several complete steps, as well as several partial sets of data. > I > will gzip them and attempt to send them to you directly. Thanks for the data set, I am right now working on it. Unfortunately, as I feared, the core files do not really help. There is a big mismatch between your system environment and mine, and so gdb is not able to extract any useful information. All I see is that there are six threads in the system, and the rest is almost only question marks. So it would be great if you could issue the gdb commands in your environment and let me know the outcome. Thanks, Rainer From joe at joetify.com Thu Sep 3 17:51:23 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 08:51:23 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> Message-ID: <20090903085123.0bcea4b0@der-dieb> Thanks, that worked perfectly. -Joe On Thu, 3 Sep 2009 07:47:14 +0200 "Rainer Gerhards" wrote: > You need to check the property replacer documentation. There are > options for case conversion. I don't know the exact syntax out of my > head, but it is along the lines of %field:::ucase%. > > Hth > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Joe Williams" > An: "rsyslog at lists.adiscon.com" > Gesendet: 03.09.09 05:19 > Betreff: [rsyslog] case sensitivity in templates > > > Hello, I am new to the list sorry if this has been covered already. I > am logging using a per-host template like: > > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > The services that log directly to the rsyslog server (haproxy, etc) > are using all lower case hostname directories where as the logs that > use the rsyslog client daemon to log to the server are using the case > specified in the hostname which in my case have capital letters in > them. > > Is there any way to specify which to use? I would like to have a > single directory for each host regardless of the case used in the > hostname. It doesn't matter to me which case is used as long as it's > the same for all logs. > > Thanks. > > -Joe > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From joe at joetify.com Thu Sep 3 18:02:58 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 09:02:58 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <20090903085123.0bcea4b0@der-dieb> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> <20090903085123.0bcea4b0@der-dieb> Message-ID: <20090903090258.384a9214@der-dieb> BTW, if anyone else has this problem or something similar the doc is at: http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.phtml The fix was changing the template to be: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase%/debug" Thanks again. -Joe On Thu, 3 Sep 2009 08:51:23 -0700 Joe Williams wrote: > Thanks, that worked perfectly. > > -Joe > > > On Thu, 3 Sep 2009 07:47:14 +0200 > "Rainer Gerhards" wrote: > > > You need to check the property replacer documentation. There are > > options for case conversion. I don't know the exact syntax out of my > > head, but it is along the lines of %field:::ucase%. > > > > Hth > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Joe Williams" > > An: "rsyslog at lists.adiscon.com" > > Gesendet: 03.09.09 05:19 > > Betreff: [rsyslog] case sensitivity in templates > > > > > > Hello, I am new to the list sorry if this has been covered already. > > I am logging using a per-host template like: > > > > $template > > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > > > The services that log directly to the rsyslog server (haproxy, etc) > > are using all lower case hostname directories where as the logs that > > use the rsyslog client daemon to log to the server are using the > > case specified in the hostname which in my case have capital > > letters in them. > > > > Is there any way to specify which to use? I would like to have a > > single directory for each host regardless of the case used in the > > hostname. It doesn't matter to me which case is used as long as it's > > the same for all logs. > > > > Thanks. > > > > -Joe > > > > > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 18:05:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 18:05:42 +0200 Subject: [rsyslog] case sensitivity in templates References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com><20090903085123.0bcea4b0@der-dieb> <20090903090258.384a9214@der-dieb> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDF5@GRFEXC.intern.adiscon.com> > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercas > e%/debug" Now that you say it: it would probably make sense to use one of the "secpath" (or so) options to make this file writer more secure - see the doc you quoted for details. It will then look something along these lines: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase,secp ath-...%/debug" > > Thanks again. my pleasure :) Rainer From corsmith at gmail.com Thu Sep 3 19:39:15 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 13:39:15 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <8061fbee0909031039y637bdf52j72b1322cf2538f55@mail.gmail.com> I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:01:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:01:12 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Thats an very interesting effort. Do you needed to patch the source? Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 19:45 Betreff: [rsyslog] rsyslog 4.4.1 and solaris I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From corsmith at gmail.com Thu Sep 3 20:22:09 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 14:22:09 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> References: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909031122x4e6469fo426368e6b7363a84@mail.gmail.com> On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:58:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:58:27 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog-users" Gesendet: 03.09.09 20:22 Betreff: Re: [rsyslog] rsyslog 4.4.1 and solaris On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From srinivasan.sreenivasan at gmail.com Fri Sep 4 01:19:05 2009 From: srinivasan.sreenivasan at gmail.com (Srinivasan Sreenivasan) Date: Thu, 3 Sep 2009 18:19:05 -0500 Subject: [rsyslog] rsyslog on Solaris Message-ID: Hi, We are trying to run rsyslog version 4.4.1 on Solaris 2.8. We cannot get it do any logging. Rainer has a blog entry (its a bit dated) that says that rsyslogd does not do local logging on Solaris. Is that still valid? -Srini From joe at joetify.com Fri Sep 4 01:48:18 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 16:48:18 -0700 Subject: [rsyslog] logging wildcards Message-ID: <20090903164818.00c7bc9d@der-dieb> Hello again, I am trying to log everything (*.*) to /var/log/syslog but local*.*. I tried a couple different ways to do this but didn't find a solution. Is this possible? I have a couple services I want to log to their own file rather than syslog, messages, etc. Any help is appreciated. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:05:04 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903164818.00c7bc9d@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > Hello again, > > I am trying to log everything (*.*) to /var/log/syslog but local*.*. I > tried a couple different ways to do this but didn't find a solution. Is > this possible? I have a couple services I want to log to their own file > rather than syslog, messages, etc. if you just do *.* /var/log/syslog that will write everything to that file I'm not sure what you are trying to say when you say 'but local*.*' above. David Lang From joe at joetify.com Fri Sep 4 02:10:22 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 17:10:22 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> Message-ID: <20090903171022.5f37eade@der-dieb> I do not want everything to log to a single file, the local facilities I would like to log to there own file and not be caught by a wildcard. Thanks. -Joe On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > Hello again, > > > > I am trying to log everything (*.*) to /var/log/syslog but > > local*.*. I tried a couple different ways to do this but didn't > > find a solution. Is this possible? I have a couple services I want > > to log to their own file rather than syslog, messages, etc. > > if you just do > > *.* /var/log/syslog > > that will write everything to that file > > I'm not sure what you are trying to say when you say 'but local*.*' > above. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:24:06 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:24:06 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903171022.5f37eade@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > I do not want everything to log to a single file, the local facilities > I would like to log to there own file and not be caught by a wildcard. ahh, ok, you cannot say local*.* you would have to list local0.*,local1.*,.. to cover them all there are 16 facility numbers, and by filtering out local0-local7 you are wanting to eliminate exactly half of them as such it's probably just as easy to list all the ones you want to record as it is to say *.* and subtract half of them. David Lang > Thanks. > -Joe > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > david at lang.hm wrote: > >> On Thu, 3 Sep 2009, Joe Williams wrote: >> >>> Hello again, >>> >>> I am trying to log everything (*.*) to /var/log/syslog but >>> local*.*. I tried a couple different ways to do this but didn't >>> find a solution. Is this possible? I have a couple services I want >>> to log to their own file rather than syslog, messages, etc. >> >> if you just do >> >> *.* /var/log/syslog >> >> that will write everything to that file >> >> I'm not sure what you are trying to say when you say 'but local*.*' >> above. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > > From joe at joetify.com Fri Sep 4 05:44:31 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 20:44:31 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: <20090903204431.5a4a2008@der-dieb> Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > > david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From DGillies at fairfaxdigital.com.au Fri Sep 4 05:51:38 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Fri, 4 Sep 2009 13:51:38 +1000 Subject: [rsyslog] logging wildcards In-Reply-To: <20090903204431.5a4a2008@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: I think something like this should work: if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != 'local2' ) then file3 David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital Level 2, 1 Darling Island Road Pyrmont NSW 2009 -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com Subject: Re: [rsyslog] logging wildcards Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From oribani at gmail.com Fri Sep 4 07:25:47 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:25:47 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <378058110909032225i10d9eb60g3039134f21eb2d64@mail.gmail.com> On 9/3/09, Ori Bani wrote: >>> I'm sorry if this isn't quite the right place to ask, since maybe no >>> one here created the RPM that's in the CentOS base repository. But I >>> am guessing people here have installed RPMs like this before and can >>> help anyway.... >>> >>> When I ask yum on CentOS 5 about rsyslog, I get this (note older >>> version - too bad): >>> >>> Available Packages >>> Name : rsyslog >>> Arch : i386 >>> Version: 2.0.6 >>> Release: 1.el5 >>> Size : 198 k >>> Repo : base >>> Summary: Enhanced system logging and kernel message trapping daemons >>> Description: >>> Rsyslog is an enhanced multi-threaded syslogd supporting, among >>> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >>> filtering on any message part, and fine grain output format control. >>> It is quite compatible to stock sysklogd and can be used as a drop- >>> in replacement. Its advanced features make it suitable for >>> enterprise-class, encryption protected syslog relay chains while at >>> the same time being very easy to setup for the novice user. >> >> I use Scientific Linux 5.x and because they are RHEL derivatives I see >> the >> same thing in the SL repo's. >> >> I have used the rsyslog from the repo's yet, all my rsyslog servers are >> based >> on EL4, but I'll try to help below. > > Thank you for your help. > >>> My questions are a little bit newbie... before I try installing >>> this, I want to know what it's going to do to my system: >>> >>> 1) Will it disable syslogd and/or klogd? Or will it add itself using >>> the "alternatives" paradigm so I can switch between them that way? >>> If neither, does it include startup scripts at all? If they are there >>> but not used by default, is there a recommended way to make the >>> switch and not really screw things up? >> >> You should try this on a test box. I haven't tried it but I think it >> should >> remove syslog RPM's from your installation and then install rsyslog. It >> should >> also make a /etc/syslog.conf.rpmsave file which you can reference for use >> in >> /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog I don't think that last line is needed; rsyslog is already added for you during the install process by yum. > And use yum to remove ksyslogd/syslog > >>> 2) Will it add itself to my cron jobs? Specifically, I don't mind >>> (for now) leaving the log rotation alone (don't let rsyslog manage my >>> rotations). If it adds itself to my cron jobs, does that mean it >>> will remove the logrotate cron job? >> >> Not sure sorry. You should grab the src.rpm file from CentOS, install it >> and >> take a look at the rsyslog.spec and it'll show you what it does on the >> post >> install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > >>> 2.5) If I keep using the old logrotate with rsyslog, will that create >>> any conflicts? >> >> I don't see how any conflicts will occur with logroate, since rsyslog >> basically logs to the same files that syslog logs to. It's meant to be a >> drop >> in replacement. >> >> Maybe specific questions about rsyslog with CentOS (or other derivatives) >> would actually be better in the CentOS or Scientific Linux mailing lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 > >>> Generally my aim is not to commit 100% to rsyslog yet, so I don't >>> want to get to a situation where it's a lot of work to get back to >>> the default syslog setup. > From oribani at gmail.com Fri Sep 4 07:21:08 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:21:08 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <20090821015920.M76525@npgx.com.au> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> Message-ID: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> one here created the RPM that's in the CentOS base repository. But I >> am guessing people here have installed RPMs like this before and can >> help anyway.... >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> version - too bad): >> >> Available Packages >> Name : rsyslog >> Arch : i386 >> Version: 2.0.6 >> Release: 1.el5 >> Size : 198 k >> Repo : base >> Summary: Enhanced system logging and kernel message trapping daemons >> Description: >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> filtering on any message part, and fine grain output format control. >> It is quite compatible to stock sysklogd and can be used as a drop- >> in replacement. Its advanced features make it suitable for >> enterprise-class, encryption protected syslog relay chains while at >> the same time being very easy to setup for the novice user. > > I use Scientific Linux 5.x and because they are RHEL derivatives I see the > same thing in the SL repo's. > > I have used the rsyslog from the repo's yet, all my rsyslog servers are > based > on EL4, but I'll try to help below. Thank you for your help. >> My questions are a little bit newbie... before I try installing >> this, I want to know what it's going to do to my system: >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself using >> the "alternatives" paradigm so I can switch between them that way? >> If neither, does it include startup scripts at all? If they are there >> but not used by default, is there a recommended way to make the >> switch and not really screw things up? > > You should try this on a test box. I haven't tried it but I think it should > remove syslog RPM's from your installation and then install rsyslog. It > should > also make a /etc/syslog.conf.rpmsave file which you can reference for use in > /etc/rsyslog.conf I wouldn't actually expect it to remove any other packages - I've never seen a yum installation remove something else - that seems like trouble. In fact, it turns out that it didn't do a thing to syslog/ksyslogd. It just installed itself in parallel (and it's up to you to turn it on). Everything is in place (startup scripts, config file that is a mirror of syslog.conf, etc.) and you just have to chkconfig syslog off chkconfig rsyslog on service syslog stop service rsyslog start I guess if you're going to be more permanent: chkconfig --del syslog chkconfig --add rsyslog And use yum to remove ksyslogd/syslog >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> (for now) leaving the log rotation alone (don't let rsyslog manage my >> rotations). If it adds itself to my cron jobs, does that mean it >> will remove the logrotate cron job? > > Not sure sorry. You should grab the src.rpm file from CentOS, install it and > take a look at the rsyslog.spec and it'll show you what it does on the post > install section. That's above my skill level. Instead I tried it out. It also adds itself to /etc/logrotate.d/syslog so you don't have to touch any of this. Here is the modified file: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true endscript } As you see, it left syslog there and added rsyslog. Because I have turned off syslog, this won't suddenly start it up, will it? >> 2.5) If I keep using the old logrotate with rsyslog, will that create >> any conflicts? > > I don't see how any conflicts will occur with logroate, since rsyslog > basically logs to the same files that syslog logs to. It's meant to be a > drop > in replacement. > > Maybe specific questions about rsyslog with CentOS (or other derivatives) > would actually be better in the CentOS or Scientific Linux mailing lists? I did, but it didn't help. That's disappointing. https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> want to get to a situation where it's a lot of work to get back to >> the default syslog setup. From rgerhards at hq.adiscon.com Fri Sep 4 12:51:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Sep 2009 12:51:21 +0200 Subject: [rsyslog] Need help with RPM(yum) version on CentOS References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com><20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> I have nothing technically to add to this discussion, but I would like remind you on the rsyslog wiki at http://wiki.rsyslog.com There already is one entry, but for an older version, not sure if that helps: http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story In any case, I would appreciate if you could share any knowledge you gain via the wiki. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ori Bani > Sent: Friday, September 04, 2009 7:21 AM > To: rsyslog-users > Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS > > >> I'm sorry if this isn't quite the right place to ask, since maybe no > >> one here created the RPM that's in the CentOS base repository. But > I > >> am guessing people here have installed RPMs like this before and can > >> help anyway.... > >> > >> When I ask yum on CentOS 5 about rsyslog, I get this (note older > >> version - too bad): > >> > >> Available Packages > >> Name : rsyslog > >> Arch : i386 > >> Version: 2.0.6 > >> Release: 1.el5 > >> Size : 198 k > >> Repo : base > >> Summary: Enhanced system logging and kernel message trapping daemons > >> Description: > >> Rsyslog is an enhanced multi-threaded syslogd supporting, among > >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, > >> filtering on any message part, and fine grain output format control. > >> It is quite compatible to stock sysklogd and can be used as a drop- > >> in replacement. Its advanced features make it suitable for > >> enterprise-class, encryption protected syslog relay chains while at > >> the same time being very easy to setup for the novice user. > > > > I use Scientific Linux 5.x and because they are RHEL derivatives I > see the > > same thing in the SL repo's. > > > > I have used the rsyslog from the repo's yet, all my rsyslog servers > are > > based > > on EL4, but I'll try to help below. > > Thank you for your help. > > >> My questions are a little bit newbie... before I try installing > >> this, I want to know what it's going to do to my system: > >> > >> 1) Will it disable syslogd and/or klogd? Or will it add itself > using > >> the "alternatives" paradigm so I can switch between them that way? > >> If neither, does it include startup scripts at all? If they are > there > >> but not used by default, is there a recommended way to make the > >> switch and not really screw things up? > > > > You should try this on a test box. I haven't tried it but I think it > should > > remove syslog RPM's from your installation and then install rsyslog. > It > > should > > also make a /etc/syslog.conf.rpmsave file which you can reference for > use in > > /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog > > And use yum to remove ksyslogd/syslog > > >> 2) Will it add itself to my cron jobs? Specifically, I don't mind > >> (for now) leaving the log rotation alone (don't let rsyslog manage > my > >> rotations). If it adds itself to my cron jobs, does that mean it > >> will remove the logrotate cron job? > > > > Not sure sorry. You should grab the src.rpm file from CentOS, install > it and > > take a look at the rsyslog.spec and it'll show you what it does on > the post > > install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > > >> 2.5) If I keep using the old logrotate with rsyslog, will that > create > >> any conflicts? > > > > I don't see how any conflicts will occur with logroate, since rsyslog > > basically logs to the same files that syslog logs to. It's meant to > be a > > drop > > in replacement. > > > > Maybe specific questions about rsyslog with CentOS (or other > derivatives) > > would actually be better in the CentOS or Scientific Linux mailing > lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start > =0#forumpost83694 > > >> Generally my aim is not to commit 100% to rsyslog yet, so I don't > >> want to get to a situation where it's a lot of work to get back to > >> the default syslog setup. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Fri Sep 4 21:25:30 2009 From: henry78 at gmx.at (Henry) Date: Fri, 04 Sep 2009 21:25:30 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message Message-ID: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Hi! This puzzels me: This is my tcprecieve config file for rsyslog v4 on ubuntu: -----8<----- $ModLoad imtcp $InputTCPServerRun 514 # some dynamic templates $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" # log remote local1 to dynamic diretory if $fromhost-ip != '127.0.0.1' and \ $syslogfacility-text == 'local1' \ then -?DYNlocal1 ----->8----- I created /var/log/remote with sufficient privileges. Unfortunately this doesn't work. rsyslog crates a folder named after the remote host (myhostname) and creates the file local1.log (again: sufficient permissions: syslog:syslog 640). But it doesn't write to that file, but logs the error: -----8<----- Could not open dynamic file '/var/log/remote/myhostname/local1.log' - discarding message ----->8----- As you might guess my question is: Why isn't rsyslog able to open a file it is able to create? Any help or hint is really appreciated. -- kind regards, Henry From joe at joetify.com Fri Sep 4 21:33:17 2009 From: joe at joetify.com (Joe Williams) Date: Fri, 4 Sep 2009 12:33:17 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: <20090904123317.172e0ca4@der-dieb> Thanks David, that ended up working after changing the "or" to an "and". Also I ended up finding a good example of this sort of configuration at http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory -Joe On Fri, 4 Sep 2009 13:51:38 +1000 David Gillies wrote: > > I think something like this should work: > > if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != > 'local2' ) then file3 > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > > Fairfax Digital > Level 2, 1 Darling Island Road > Pyrmont NSW 2009 > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams > Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] logging wildcards > > > Sorry I think we are misunderstanding each other. What I am wanting > to do is this: > > ### > > local0.* FILE1 > local2.* FILE2 > > *.* (but not local0.* or local2.*) FILE3 > > ### > > Is that possible? > > Thanks again. > -Joe > > > > On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) > david at lang.hm wrote: > > > On Thu, 3 Sep 2009, Joe Williams wrote: > > > > > I do not want everything to log to a single file, the local > > > facilities I would like to log to there own file and not be > > > caught by a wildcard. > > > > ahh, ok, you cannot say local*.* you would have to list > > local0.*,local1.*,.. to cover them all > > > > there are 16 facility numbers, and by filtering out local0-local7 > > you are wanting to eliminate exactly half of them > > > > as such it's probably just as easy to list all the ones you want to > > record as it is to say *.* and subtract half of them. > > > > David Lang > > > > > Thanks. > > > -Joe > > > > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > > >> > > >>> Hello again, > > >>> > > >>> I am trying to log everything (*.*) to /var/log/syslog but > > >>> local*.*. I tried a couple different ways to do this but didn't > > >>> find a solution. Is this possible? I have a couple services I > > >>> want to log to their own file rather than syslog, messages, etc. > > >> > > >> if you just do > > >> > > >> *.* /var/log/syslog > > >> > > >> that will write everything to that file > > >> > > >> I'm not sure what you are trying to say when you say 'but > > >> local*.*' above. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -- > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended > recipient, any use, dissemination, reliance, forwarding, printing or > copying of this e-mail or any attached files is unauthorised. This > e-mail is subject to copyright. No part of it should be reproduced, > adapted or communicated without the written consent of the copyright > owner. If you have received this e-mail in error please advise the > sender immediately by return e-mail or telephone and delete all > copies. Fairfax does not guarantee the accuracy or completeness of > any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept > legal responsibility for the contents of this message or attached > files. _______________________________________________ rsyslog > mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Sat Sep 5 04:04:59 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 19:04:59 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file Message-ID: I ahve a config file that fixes up broken syslog messages that has the following $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8;TraditionalForwardFormat the upstream box is seeing things as I would expect, but the local /var/log/messages file is not is it incorrect to have two entries that both write to /var/log/messages? David Lang From david at lang.hm Sat Sep 5 08:03:23 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 23:03:23 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file In-Reply-To: References: Message-ID: On Fri, 4 Sep 2009, david at lang.hm wrote: > I ahve a config file that fixes up broken syslog messages that has the > following > > $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" > :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat > & @192.168.210.8;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.210.8;TraditionalForwardFormat > > > the upstream box is seeing things as I would expect, but the local > /var/log/messages file is not > > is it incorrect to have two entries that both write to /var/log/messages? never mind, I just spotted the extra *.* in there (nothing was reported when starting up) David Lang From oribani at gmail.com Sun Sep 6 03:52:04 2009 From: oribani at gmail.com (Ori Bani) Date: Sat, 5 Sep 2009 18:52:04 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> Message-ID: <378058110909051852t1ae1f4dgd7f00d830b3b6284@mail.gmail.com> On 9/4/09, Rainer Gerhards wrote: > I have nothing technically to add to this discussion, but I would like > remind > you on the rsyslog wiki at > > http://wiki.rsyslog.com > > There already is one entry, but for an older version, not sure if that > helps: > > http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story > > In any case, I would appreciate if you could share any knowledge you gain > via > the wiki. I added my info, but that page was designed by someone who assumes you aren't using yum (or any similar system), so I hope I added it in an acceptable way >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Ori Bani >> Sent: Friday, September 04, 2009 7:21 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS >> >> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> >> one here created the RPM that's in the CentOS base repository. But >> I >> >> am guessing people here have installed RPMs like this before and can >> >> help anyway.... >> >> >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> >> version - too bad): >> >> >> >> Available Packages >> >> Name : rsyslog >> >> Arch : i386 >> >> Version: 2.0.6 >> >> Release: 1.el5 >> >> Size : 198 k >> >> Repo : base >> >> Summary: Enhanced system logging and kernel message trapping daemons >> >> Description: >> >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> >> filtering on any message part, and fine grain output format control. >> >> It is quite compatible to stock sysklogd and can be used as a drop- >> >> in replacement. Its advanced features make it suitable for >> >> enterprise-class, encryption protected syslog relay chains while at >> >> the same time being very easy to setup for the novice user. >> > >> > I use Scientific Linux 5.x and because they are RHEL derivatives I >> see the >> > same thing in the SL repo's. >> > >> > I have used the rsyslog from the repo's yet, all my rsyslog servers >> are >> > based >> > on EL4, but I'll try to help below. >> >> Thank you for your help. >> >> >> My questions are a little bit newbie... before I try installing >> >> this, I want to know what it's going to do to my system: >> >> >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself >> using >> >> the "alternatives" paradigm so I can switch between them that way? >> >> If neither, does it include startup scripts at all? If they are >> there >> >> but not used by default, is there a recommended way to make the >> >> switch and not really screw things up? >> > >> > You should try this on a test box. I haven't tried it but I think it >> should >> > remove syslog RPM's from your installation and then install rsyslog. >> It >> > should >> > also make a /etc/syslog.conf.rpmsave file which you can reference for >> use in >> > /etc/rsyslog.conf >> >> I wouldn't actually expect it to remove any other packages - I've >> never seen a yum installation remove something else - that seems like >> trouble. In fact, it turns out that it didn't do a thing to >> syslog/ksyslogd. It just installed itself in parallel (and it's up to >> you to turn it on). Everything is in place (startup scripts, config >> file that is a mirror of syslog.conf, etc.) and you just have to >> >> chkconfig syslog off >> chkconfig rsyslog on >> service syslog stop >> service rsyslog start >> >> I guess if you're going to be more permanent: >> >> chkconfig --del syslog >> chkconfig --add rsyslog >> >> And use yum to remove ksyslogd/syslog >> >> >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> >> (for now) leaving the log rotation alone (don't let rsyslog manage >> my >> >> rotations). If it adds itself to my cron jobs, does that mean it >> >> will remove the logrotate cron job? >> > >> > Not sure sorry. You should grab the src.rpm file from CentOS, install >> it and >> > take a look at the rsyslog.spec and it'll show you what it does on >> the post >> > install section. >> >> That's above my skill level. Instead I tried it out. It also adds >> itself to /etc/logrotate.d/syslog so you don't have to touch any of >> this. Here is the modified file: >> >> /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler >> /var/log/boot.log /var/log/cron { >> sharedscripts >> postrotate >> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> >> /dev/null || true >> /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> >> /dev/null || true >> endscript >> } >> >> As you see, it left syslog there and added rsyslog. Because I have >> turned off syslog, this won't suddenly start it up, will it? >> >> >> 2.5) If I keep using the old logrotate with rsyslog, will that >> create >> >> any conflicts? >> > >> > I don't see how any conflicts will occur with logroate, since rsyslog >> > basically logs to the same files that syslog logs to. It's meant to >> be a >> > drop >> > in replacement. >> > >> > Maybe specific questions about rsyslog with CentOS (or other >> derivatives) >> > would actually be better in the CentOS or Scientific Linux mailing >> lists? >> >> I did, but it didn't help. That's disappointing. >> >> https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start >> =0#forumpost83694 >> >> >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> >> want to get to a situation where it's a lot of work to get back to >> >> the default syslog setup. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 20:17:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 13:17:50 -0500 Subject: [rsyslog] syslog server and reports Message-ID: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> I have some debian lenny servers sending their logs (via TCP) to a central rsyslog server. Every remote servers has at /etc/rsyslog.conf: *.* @@IP_CENTRAL_SERVER So, I can see in the central syslog server all logs without problems. I'm looking for a single and simple report, like logwatch for example who process all logs and send me in ONE mail or on ONE html page all resume info of all logs. I tried with logwatch and I didn't get this report I'm looking for. My question is? Is there any tool, script, app, etc which I run on the syslog server and give me the information of all servers in a way as simple as possible? Maybe in a single resume mail separated by a line for example? Thanks for your time. -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Sun Sep 6 21:00:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 6 Sep 2009 21:00:46 +0200 Subject: [rsyslog] syslog server and reports References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Probably not exactly what you look for, but maybe worth a try: http://www.phplogcon.org More reporting featueres are being tackled in the next couple of weeks. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Sunday, September 06, 2009 8:18 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] syslog server and reports > > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? > > Thanks for your time. > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 21:20:34 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 14:20:34 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Message-ID: <194a2c240909061220k25cf03e4ycb7dcf379d45ab8f@mail.gmail.com> Hi Rainer, thanks for your soon answer.. On 9/6/09, Rainer Gerhards wrote: > Probably not exactly what you look for, but maybe worth a try: > > http://www.phplogcon.org I have installed phplogcon but, it's not whay I'm looking for. I need an email, a simple daily email with the reports of all my servers. I've tried to setup logwatch and logcheck but I could not get what I want. regards, Israel. > > More reporting featueres are being tackled in the next couple of weeks. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia >> Sent: Sunday, September 06, 2009 8:18 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] syslog server and reports >> >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? >> >> Thanks for your time. >> >> -- >> Regards; >> Israel Garcia >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Sun Sep 6 23:19:35 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 14:19:35 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? there are a lot of products and projects out there to analyse logs and generate reports. the problem is that what I am interested in seeing in a report may or may not match what you are interested in seeing. also, most of this effort is taking place within originizations that have large volumes of logs, so distilling it down to a single report or e-mail requires that a lot of detail gets left out (and that goes back to exactly what you are interested in seeing) when you say you want one page that shows you 'everything', what is it that you want to see? are there particular messages that you want to see if they show up even once? or are you interested in simplifying log messages into categories and seeing how many messages in each category you have. do you only care about the logs showing up sometime during the day? or are you interested in the trending of how many logs you get each second throughout the day (or anything in between) unfortunantly the result of all these questions probably means that you will need to customize whatever you use to exactly the report that you want. large companies can spend millions of dollars on systems and software to alert, report, and query their logs. I am currently getting ~300M log messages/day and I distill it down to a single e-mail report that I look at (and generate additional reports with subsets of the data for other people to look at). the best advice I ever got was to use the approach termed 'artificial ignorance' start off with all your logs for any log type that you can categorize create a summary of that log type (even if it's an unimportant log, count it because the number of times an unimportant thing happens can be important) look at what's left and repeat the process after several iterations of this you end up with the vast majority of your logs summarized and a report of "what's left", any new messages that you have never seen before (which usually mean they are important) show up in the "what's left" bucket and tend to stand out you do need to keep on top of this, upgrades to systems, new installs, etc cause new logs to show up, if you categorize and summarize them your final report stays small, if you let things slide for several months the final report can end up very large (and therefor useless) David Lang From igalvarez at gmail.com Mon Sep 7 01:40:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 18:40:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? > > there are a lot of products and projects out there to analyse logs and > generate reports. > > the problem is that what I am interested in seeing in a report may or may > not match what you are interested in seeing. > > also, most of this effort is taking place within originizations that have > large volumes of logs, so distilling it down to a single report or e-mail > requires that a lot of detail gets left out (and that goes back to exactly > what you are interested in seeing) > > when you say you want one page that shows you 'everything', what is it > that you want to see? Hi, David I mean, a report like logwatch use to send me everyday from each server. As I said before, I'm collecting all servers logs (syslog and auth.log) into my central syslog, so I need some tool like logwatch running on the collector which send in one mail or in one html page. . I tried to configure logwatch in the collector without sucess. That's what I need. :-) thanks. regards, Israel > > are there particular messages that you want to see if they show up even > once? or are you interested in simplifying log messages into categories > and seeing how many messages in each category you have. > > do you only care about the logs showing up sometime during the day? or are > you interested in the trending of how many logs you get each second > throughout the day (or anything in between) > > unfortunantly the result of all these questions probably means that you > will need to customize whatever you use to exactly the report that you > want. > > large companies can spend millions of dollars on systems and software to > alert, report, and query their logs. > > I am currently getting ~300M log messages/day and I distill it down to a > single e-mail report that I look at (and generate additional reports with > subsets of the data for other people to look at). > > > the best advice I ever got was to use the approach termed 'artificial > ignorance' > > start off with all your logs > > for any log type that you can categorize create a summary of that log type > (even if it's an unimportant log, count it because the number of times an > unimportant thing happens can be important) > > look at what's left and repeat the process > > after several iterations of this you end up with the vast majority of your > logs summarized and a report of "what's left", any new messages that you > have never seen before (which usually mean they are important) show up in > the "what's left" bucket and tend to stand out > > you do need to keep on top of this, upgrades to systems, new installs, > etc cause new logs to show up, if you categorize and summarize them your > final report stays small, if you let things slide for several months the > final report can end up very large (and therefor useless) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 02:15:40 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 17:15:40 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> I have some debian lenny servers sending their logs (via TCP) to a >>> central rsyslog server. >>> Every remote servers has at /etc/rsyslog.conf: >>> >>> *.* @@IP_CENTRAL_SERVER >>> >>> So, I can see in the central syslog server all logs without problems. >>> I'm looking for a single and simple report, like logwatch for example >>> who process all logs and send me in ONE mail or on ONE html page all >>> resume info of all logs. I tried with logwatch and I didn't get this >>> report I'm looking for. >>> >>> My question is? >>> Is there any tool, script, app, etc which I run on the syslog server >>> and give me the information of all servers in a way as simple as >>> possible? Maybe in a single resume mail separated by a line for >>> example? >> >> there are a lot of products and projects out there to analyse logs and >> generate reports. >> >> the problem is that what I am interested in seeing in a report may or may >> not match what you are interested in seeing. >> >> also, most of this effort is taking place within originizations that have >> large volumes of logs, so distilling it down to a single report or e-mail >> requires that a lot of detail gets left out (and that goes back to exactly >> what you are interested in seeing) >> >> when you say you want one page that shows you 'everything', what is it >> that you want to see? > Hi, David > I mean, a report like logwatch use to send me everyday from each > server. As I said before, I'm collecting all servers logs (syslog and > auth.log) into my central syslog, so I need some tool like logwatch > running on the collector which send in one mail or in one html page. > . > I tried to configure logwatch in the collector without sucess. > > That's what I need. :-) ok, so you want the report that you get from logwatch, that simplifies things. when you say you can't get it to work on the collector box, more info is needed. does logwatch give you the info that you want about the collector box? do you put the logs from all servers in one file? or do you split them by host? (or split them in other ways) how does logwatch fail? does it crash? give you incorrect information? other? David Lang From igalvarez at gmail.com Mon Sep 7 03:47:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 20:47:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> I have some debian lenny servers sending their logs (via TCP) to a >>>> central rsyslog server. >>>> Every remote servers has at /etc/rsyslog.conf: >>>> >>>> *.* @@IP_CENTRAL_SERVER >>>> >>>> So, I can see in the central syslog server all logs without problems. >>>> I'm looking for a single and simple report, like logwatch for example >>>> who process all logs and send me in ONE mail or on ONE html page all >>>> resume info of all logs. I tried with logwatch and I didn't get this >>>> report I'm looking for. >>>> >>>> My question is? >>>> Is there any tool, script, app, etc which I run on the syslog server >>>> and give me the information of all servers in a way as simple as >>>> possible? Maybe in a single resume mail separated by a line for >>>> example? >>> >>> there are a lot of products and projects out there to analyse logs and >>> generate reports. >>> >>> the problem is that what I am interested in seeing in a report may or may >>> not match what you are interested in seeing. >>> >>> also, most of this effort is taking place within originizations that have >>> large volumes of logs, so distilling it down to a single report or e-mail >>> requires that a lot of detail gets left out (and that goes back to >>> exactly >>> what you are interested in seeing) >>> >>> when you say you want one page that shows you 'everything', what is it >>> that you want to see? >> Hi, David >> I mean, a report like logwatch use to send me everyday from each >> server. As I said before, I'm collecting all servers logs (syslog and >> auth.log) into my central syslog, so I need some tool like logwatch >> running on the collector which send in one mail or in one html page. >> . >> I tried to configure logwatch in the collector without sucess. >> >> That's what I need. :-) > > ok, so you want the report that you get from logwatch, that simplifies > things. > > when you say you can't get it to work on the collector box, more info is > needed. > > does logwatch give you the info that you want about the collector box? My scenario: I added this two lines in /etc/rsyslog.conf of all exporting servers: auth,authpriv.* @@xx.xx.xx.xx *.*;auth,authpriv.none @@xx.xx.xx.xx In the collector syslog and auth.log files I see logs coming from those servers. logwatch.conf file is the default. I run logwatch (testing mode) in the collector and it merge logs from all servers, so you can not identify which log output is belongs to. It looks like all logs are from the collector server. here you can see a part of logwatch output: In my case deb2 is the hostname of the collector and debian is the hostname of one exporter. deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) #################### Processing Initiated: Sun Sep 6 21:35:29 2009 Date Range Processed: today ( 2009-Sep-06 ) Period is day. Detail Level of Output: 0 Type of Output/Format: stdout / text Logfiles for Host: deb2 ################################################################## ###This logs are from deb2 Installed: libdate-manip-perl 5.54-1 lockfile-progs 0.1.11-0.1 logtail 1.2.69 logwatch 7.3.6.cvs20080702-2 postfix 2.5.5-1.1 . . . . . --------------------- pam_unix Begin ------------------------ ### All this logs entries from user test123 are from one exporter server (debian). sshd: Authentication Failures: root (localhost): 1 Time(s) su: Authentication Failures: test123(1003) -> root: 2 Time(s) Sessions Opened: root -> logcheck: 17 Time(s) root -> root: 9 Time(s) sudo: Authentication Failures: test123(0) -> test123: 1 Time(s) **Unmatched Entries** useradd: failed adding user `test', data deleted: 1 Time(s) ---------------------- Connections (secure-log) End ------------------------- ============================================================================== ### This is from exporter debian server. test123 => root --------------- /bin/su - 1 Times. ---------------------- Sudo (secure-log) End ------------------------- ## This df output is from deb2 (collector) --------------------- Disk Space Begin ------------------------ Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.5G 2.0G 5.2G 28% / ---------------------- Disk Space End ------------------------- ###################### Logwatch End ################### As you can see, it seems like the report belongs to deb2 server and it's not. I'd be happy if at least logwatch put some tags at the beginning of each line to identify the source. thanks again. regards, israel. > > do you put the logs from all servers in one file? or do you split them by > host? (or split them in other ways) > > how does logwatch fail? does it crash? give you incorrect information? > other? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 04:23:51 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 19:23:51 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> On 9/6/09, david at lang.hm wrote: >>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>> >>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>> central rsyslog server. >>>>> Every remote servers has at /etc/rsyslog.conf: >>>>> >>>>> *.* @@IP_CENTRAL_SERVER >>>>> >>>>> So, I can see in the central syslog server all logs without problems. >>>>> I'm looking for a single and simple report, like logwatch for example >>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>> report I'm looking for. >>>>> >>>>> My question is? >>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>> and give me the information of all servers in a way as simple as >>>>> possible? Maybe in a single resume mail separated by a line for >>>>> example? >>>> >>>> there are a lot of products and projects out there to analyse logs and >>>> generate reports. >>>> >>>> the problem is that what I am interested in seeing in a report may or may >>>> not match what you are interested in seeing. >>>> >>>> also, most of this effort is taking place within originizations that have >>>> large volumes of logs, so distilling it down to a single report or e-mail >>>> requires that a lot of detail gets left out (and that goes back to >>>> exactly >>>> what you are interested in seeing) >>>> >>>> when you say you want one page that shows you 'everything', what is it >>>> that you want to see? >>> Hi, David >>> I mean, a report like logwatch use to send me everyday from each >>> server. As I said before, I'm collecting all servers logs (syslog and >>> auth.log) into my central syslog, so I need some tool like logwatch >>> running on the collector which send in one mail or in one html page. >>> . >>> I tried to configure logwatch in the collector without sucess. >>> >>> That's what I need. :-) >> >> ok, so you want the report that you get from logwatch, that simplifies >> things. >> >> when you say you can't get it to work on the collector box, more info is >> needed. >> >> does logwatch give you the info that you want about the collector box? > > My scenario: > I added this two lines in /etc/rsyslog.conf of all exporting servers: > > auth,authpriv.* @@xx.xx.xx.xx > *.*;auth,authpriv.none @@xx.xx.xx.xx > > In the collector syslog and auth.log files I see logs coming from > those servers. > > logwatch.conf file is the default. > > I run logwatch (testing mode) in the collector and it merge logs from > all servers, so you can not identify which log output is belongs to. > It looks like all logs are from the collector server. ahh, that's the problem. unforutnantly fixing this would take some significant surgury to logwatch. it assumes that all the logs it is dealing with are from the local box and therefor it ignores the server tag in the output. you could use the rsyslog dynafiles feature to create a different file for each server, run logwatch against each of those files, and then combine the reports (including adding text to tell you which server is up next) David Lang > here you can see a part of logwatch output: > > In my case deb2 is the hostname of the collector and debian is the > hostname of one exporter. > > deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today > > ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) > #################### > Processing Initiated: Sun Sep 6 21:35:29 2009 > Date Range Processed: today > ( 2009-Sep-06 ) > Period is day. > Detail Level of Output: 0 > Type of Output/Format: stdout / text > Logfiles for Host: deb2 > ################################################################## > > ###This logs are from deb2 > Installed: > libdate-manip-perl 5.54-1 > lockfile-progs 0.1.11-0.1 > logtail 1.2.69 > logwatch 7.3.6.cvs20080702-2 > postfix 2.5.5-1.1 > . > . > . > . > . > --------------------- pam_unix Begin ------------------------ > ### All this logs entries from user test123 are from one exporter > server (debian). > sshd: > Authentication Failures: > root (localhost): 1 Time(s) > > su: > Authentication Failures: > test123(1003) -> root: 2 Time(s) > Sessions Opened: > root -> logcheck: 17 Time(s) > root -> root: 9 Time(s) > > sudo: > Authentication Failures: > test123(0) -> test123: 1 Time(s) > > **Unmatched Entries** > useradd: failed adding user `test', data deleted: 1 Time(s) > > ---------------------- Connections (secure-log) End ------------------------- > > > ============================================================================== > ### This is from exporter debian server. > test123 => root > --------------- > /bin/su - 1 Times. > > ---------------------- Sudo (secure-log) End ------------------------- > > ## This df output is from deb2 (collector) > --------------------- Disk Space Begin ------------------------ > > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 7.5G 2.0G 5.2G 28% / > > ---------------------- Disk Space End ------------------------- > > ###################### Logwatch End ################### > > As you can see, it seems like the report belongs to deb2 server and it's not. > > I'd be happy if at least logwatch put some tags at the beginning of > each line to identify the source. > > thanks again. > regards, > israel. > > > > > >> >> do you put the logs from all servers in one file? or do you split them by >> host? (or split them in other ways) >> >> how does logwatch fail? does it crash? give you incorrect information? >> other? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > From igalvarez at gmail.com Mon Sep 7 04:46:41 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 21:46:41 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> On 9/6/09, david at lang.hm wrote: >>>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>>> >>>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>>> central rsyslog server. >>>>>> Every remote servers has at /etc/rsyslog.conf: >>>>>> >>>>>> *.* @@IP_CENTRAL_SERVER >>>>>> >>>>>> So, I can see in the central syslog server all logs without problems. >>>>>> I'm looking for a single and simple report, like logwatch for example >>>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>>> report I'm looking for. >>>>>> >>>>>> My question is? >>>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>>> and give me the information of all servers in a way as simple as >>>>>> possible? Maybe in a single resume mail separated by a line for >>>>>> example? >>>>> >>>>> there are a lot of products and projects out there to analyse logs and >>>>> generate reports. >>>>> >>>>> the problem is that what I am interested in seeing in a report may or >>>>> may >>>>> not match what you are interested in seeing. >>>>> >>>>> also, most of this effort is taking place within originizations that >>>>> have >>>>> large volumes of logs, so distilling it down to a single report or >>>>> e-mail >>>>> requires that a lot of detail gets left out (and that goes back to >>>>> exactly >>>>> what you are interested in seeing) >>>>> >>>>> when you say you want one page that shows you 'everything', what is it >>>>> that you want to see? >>>> Hi, David >>>> I mean, a report like logwatch use to send me everyday from each >>>> server. As I said before, I'm collecting all servers logs (syslog and >>>> auth.log) into my central syslog, so I need some tool like logwatch >>>> running on the collector which send in one mail or in one html page. >>>> . >>>> I tried to configure logwatch in the collector without sucess. >>>> >>>> That's what I need. :-) >>> >>> ok, so you want the report that you get from logwatch, that simplifies >>> things. >>> >>> when you say you can't get it to work on the collector box, more info is >>> needed. >>> >>> does logwatch give you the info that you want about the collector box? >> >> My scenario: >> I added this two lines in /etc/rsyslog.conf of all exporting servers: >> >> auth,authpriv.* @@xx.xx.xx.xx >> *.*;auth,authpriv.none @@xx.xx.xx.xx >> >> In the collector syslog and auth.log files I see logs coming from >> those servers. >> >> logwatch.conf file is the default. >> >> I run logwatch (testing mode) in the collector and it merge logs from >> all servers, so you can not identify which log output is belongs to. >> It looks like all logs are from the collector server. > > ahh, that's the problem. > > unforutnantly fixing this would take some significant surgury to logwatch. > it assumes that all the logs it is dealing with are from the local box and > therefor it ignores the server tag in the output. > > you could use the rsyslog dynafiles feature to create a different file for > each server, run logwatch against each of those files, and then combine > the reports (including adding text to tell you which server is up next) Hi David, I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. > > David Lang > >> here you can see a part of logwatch output: >> >> In my case deb2 is the hostname of the collector and debian is the >> hostname of one exporter. >> >> deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today >> >> ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) >> #################### >> Processing Initiated: Sun Sep 6 21:35:29 2009 >> Date Range Processed: today >> ( 2009-Sep-06 ) >> Period is day. >> Detail Level of Output: 0 >> Type of Output/Format: stdout / text >> Logfiles for Host: deb2 >> ################################################################## >> >> ###This logs are from deb2 >> Installed: >> libdate-manip-perl 5.54-1 >> lockfile-progs 0.1.11-0.1 >> logtail 1.2.69 >> logwatch 7.3.6.cvs20080702-2 >> postfix 2.5.5-1.1 >> . >> . >> . >> . >> . >> --------------------- pam_unix Begin ------------------------ >> ### All this logs entries from user test123 are from one exporter >> server (debian). >> sshd: >> Authentication Failures: >> root (localhost): 1 Time(s) >> >> su: >> Authentication Failures: >> test123(1003) -> root: 2 Time(s) >> Sessions Opened: >> root -> logcheck: 17 Time(s) >> root -> root: 9 Time(s) >> >> sudo: >> Authentication Failures: >> test123(0) -> test123: 1 Time(s) >> >> **Unmatched Entries** >> useradd: failed adding user `test', data deleted: 1 Time(s) >> >> ---------------------- Connections (secure-log) End >> ------------------------- >> >> >> ============================================================================== >> ### This is from exporter debian server. >> test123 => root >> --------------- >> /bin/su - 1 Times. >> >> ---------------------- Sudo (secure-log) End ------------------------- >> >> ## This df output is from deb2 (collector) >> --------------------- Disk Space Begin ------------------------ >> >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda1 7.5G 2.0G 5.2G 28% / >> >> ---------------------- Disk Space End ------------------------- >> >> ###################### Logwatch End ################### >> >> As you can see, it seems like the report belongs to deb2 server and it's >> not. >> >> I'd be happy if at least logwatch put some tags at the beginning of >> each line to identify the source. >> >> thanks again. >> regards, >> israel. >> >> >> >> >> >>> >>> do you put the logs from all servers in one file? or do you split them by >>> host? (or split them in other ways) >>> >>> how does logwatch fail? does it crash? give you incorrect information? >>> other? >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Mon Sep 7 15:18:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Sep 2009 15:18:13 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE1F@GRFEXC.intern.adiscon.com> Hi all, after some struggle, a new status: Thanks to David's data sets, I think I have finally been able to find a code spot that may be troublesome. It also is in an area that we already had under suspicion. While it is too early to say if I finally found the issue, it looks very promising. If I am right, the problem is actually environment-induced, what would also explain why other users did not yet report anything and I did not see anything in my lab so far. The ultimate root cause may even be a formatting error in another rsyslogd instance further up in the relay chain. If so, I'll try to work upward from where to problem currently occurs to the above-it root cause. I just thought I share this new information with you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, September 03, 2009 12:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > Hi David, > > > I haven't gone back to the 3.x series, but I did several more runs > with > > 4.2.0 doing the folloiwng > > > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd > -r > > -h ; mv /core /core-4.2.0-10 > > > > I have several complete steps, as well as several partial sets of > data. > > I > > will gzip them and attempt to send them to you directly. > > Thanks for the data set, I am right now working on it. Unfortunately, > as I > feared, the core files do not really help. There is a big mismatch > between > your system environment and mine, and so gdb is not able to extract any > useful information. All I see is that there are six threads in the > system, > and the rest is almost only question marks. > > So it would be great if you could issue the gdb commands in your > environment > and let me know the outcome. > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From DGillies at fairfaxdigital.com.au Tue Sep 8 02:15:29 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 8 Sep 2009 10:15:29 +1000 Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: Hi Israel, Its been a while since I last used it, but I'm pretty sure that epylog can handle reporting on log files with multiple hosts: https://fedorahosted.org/epylog/ David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia Sent: Monday, 7 September 2009 12:47 PM To: rsyslog-users Subject: Re: [rsyslog] syslog server and reports I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From igalvarez at gmail.com Tue Sep 8 05:19:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Mon, 7 Sep 2009 22:19:50 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: <194a2c240909072019q618594cay38b7ca7202d13b10@mail.gmail.com> On 9/7/09, David Gillies wrote: > Hi Israel, Hi David, > > Its been a while since I last used it, but I'm pretty sure that epylog can > handle reporting on log files with multiple hosts: > > https://fedorahosted.org/epylog/ umm... sounds good.. I see my rsyslog collector has the latest version of epylog..I'll try right now..:-) thanks regards, israel. > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Monday, 7 September 2009 12:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] syslog server and reports > > > > I'll try this way.. but do you know if there another tool more simple to get > jmy report? > thanks in advance. > > regards, > israel. > The information contained in this e-mail message and any accompanying files > is or may be confidential. If you are not the intended recipient, any use, > dissemination, reliance, forwarding, printing or copying of this e-mail or > any attached files is unauthorised. This e-mail is subject to copyright. No > part of it should be reproduced, adapted or communicated without the written > consent of the copyright owner. If you have received this e-mail in error > please advise the sender immediately by return e-mail or telephone and > delete all copies. Fairfax does not guarantee the accuracy or completeness > of any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept legal > responsibility for the contents of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From henry78 at gmx.at Tue Sep 8 09:03:49 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 09:03:49 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <1252092330.924.24.camel@eberhe.office.chipkarte.at> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Message-ID: <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Hello! Tried it with various log locations (e.g. /tmp/my.log), neither worked. Is this worth ab bug? -- kind regards, Henry On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > Hi! > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > ubuntu: > > -----8<----- > $ModLoad imtcp > $InputTCPServerRun 514 > > # some dynamic templates > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > # log remote local1 to dynamic diretory > if $fromhost-ip != '127.0.0.1' and \ > $syslogfacility-text == 'local1' \ > then -?DYNlocal1 > ----->8----- > > I created /var/log/remote with sufficient privileges. > > Unfortunately this doesn't work. rsyslog crates a folder named after the > remote host (myhostname) and creates the file local1.log (again: > sufficient permissions: syslog:syslog 640). But it doesn't write to that > file, but logs the error: > > -----8<----- > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > discarding message > ----->8----- > > As you might guess my question is: Why isn't rsyslog able to open a file > it is able to create? Any help or hint is really appreciated. > From rgerhards at hq.adiscon.com Tue Sep 8 09:55:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 09:55:21 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> can you provide a debug log? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 9:04 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > Hello! > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > Is this worth ab bug? > > -- > kind regards, Henry > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > Hi! > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > ubuntu: > > > > -----8<----- > > $ModLoad imtcp > > $InputTCPServerRun 514 > > > > # some dynamic templates > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > # log remote local1 to dynamic diretory > > if $fromhost-ip != '127.0.0.1' and \ > > $syslogfacility-text == 'local1' \ > > then -?DYNlocal1 > > ----->8----- > > > > I created /var/log/remote with sufficient privileges. > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > the > > remote host (myhostname) and creates the file local1.log (again: > > sufficient permissions: syslog:syslog 640). But it doesn't write to > that > > file, but logs the error: > > > > -----8<----- > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > discarding message > > ----->8----- > > > > As you might guess my question is: Why isn't rsyslog able to open a > file > > it is able to create? Any help or hint is really appreciated. > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:30:17 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Hi, I got the debug log, it was too big to be sent via the list (but I got it as list admin). I see that you drop privileges to the user "syslog". This probably explains what happens. I think the file is created before you drop privileges, but can then no longer be written when running in the new security context. Could you verify that the user "syslog" can access this file? Also, could you temporarily remove the Privilege drop? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 9:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Tue Sep 8 12:31:35 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:31:35 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <1252405895.17741.41.camel@eberhe.office.chipkarte.at> Hmm... a simple '-d' debug doesn't seem to give enough information, see attached rsyslogd.debug.full. Attached log starts with processing of the remote logging because the full log is too large for this list. Note, that this was started with an empty /var/log/remote and the file /var/log/remote/myhostname/local1.log got created during debug run. rsysloghost='loghost', remotehost='remotehost'. Thanks for having a look at this, -- regards, Henry On Di, 2009-09-08 at 09:55 +0200, Rainer Gerhards wrote: > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -------------- next part -------------- 5054.727606022:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 5054.727609261:main thread: main queue: EnqueueMsg advised worker start 5054.727630769:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 5054.727662249:imtcp.c: -------- calling select, active fds (max 4): 4 5054.727668059:main thread: initialization completed, transitioning to regular run mode 5056.400110663:imtcp.c: New connect on NSD 0x65a690. 5056.400240204:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400255357:imtcp.c: netstream 0x683350 with new data 5056.400272148:imtcp.c: logmsg: flags 20, from 'remotehost', msg Sep 8 12:17:36 remotehost root: test by henry 5056.400274671:imtcp.c: Message has legacy syslog format. 5056.400279417:imtcp.c: main queue: entry added, size now 1 entries 5056.400282172:imtcp.c: wtpAdviseMaxWorkers signals busy 5056.400290169:imtcp.c: main queue: EnqueueMsg advised worker start 5056.400301360:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400311400:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400328690:main queue:Reg/w0: result of expression evaluation: 0 5056.400337430:main queue:Reg/w0: result of expression evaluation: 0 5056.400346474:main queue:Reg/w0: result of expression evaluation: 0 5056.400351644:main queue:Reg/w0: result of expression evaluation: 0 5056.400358215:main queue:Reg/w0: result of expression evaluation: 0 5056.400366212:main queue:Reg/w0: result of expression evaluation: 0 5056.400371208:main queue:Reg/w0: result of expression evaluation: 0 5056.400381921:main queue:Reg/w0: result of expression evaluation: 0 5056.400388679:main queue:Reg/w0: result of expression evaluation: 0 5056.400395611:main queue:Reg/w0: result of expression evaluation: 0 5056.400403792:main queue:Reg/w0: result of expression evaluation: 0 5056.400420697:main queue:Reg/w0: result of expression evaluation: 0 5056.400430773:main queue:Reg/w0: result of expression evaluation: 1 5056.400434138:main queue:Reg/w0: Called action, logging to builtin-file 5056.400440938:main queue:Reg/w0: (DYNlocal1) 5056.400507577:main queue:Reg/w0: Called LogError, msg: Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400531156:main queue:Reg/w0: logmsg: flags 1, from 'loghost', msg Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400537445:main queue:Reg/w0: Message has legacy syslog format. 5056.400540197:main queue:Reg/w0: main queue: entry added, size now 1 entries 5056.400542937:main queue:Reg/w0: wtpAdviseMaxWorkers signals busy 5056.400544844:main queue:Reg/w0: main queue: EnqueueMsg advised worker start 5056.400548374:main queue:Reg/w0: Removed entry 0 for file '[OPEN FAILED]' from dynaCache. 5056.400550633:main queue:Reg/w0: Action requested to be suspended, done that. 5056.400554546:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400562730:main queue:Reg/w0: result of expression evaluation: 0 5056.400572312:main queue:Reg/w0: result of expression evaluation: 1 5056.400575746:main queue:Reg/w0: Called action, logging to builtin-file 5056.400580715:main queue:Reg/w0: (/var/log/syslog) 5056.400591557:main queue:Reg/w0: result of expression evaluation: 0 5056.400607572:main queue:Reg/w0: result of expression evaluation: 0 5056.400612225:main queue:Reg/w0: result of expression evaluation: 0 5056.400619969:main queue:Reg/w0: result of expression evaluation: 0 5056.400627299:main queue:Reg/w0: result of expression evaluation: 0 5056.400633371:main queue:Reg/w0: result of expression evaluation: 0 5056.400639122:main queue:Reg/w0: result of expression evaluation: 0 5056.400644767:main queue:Reg/w0: result of expression evaluation: 0 5056.400651994:main queue:Reg/w0: result of expression evaluation: 1 5056.400656433:main queue:Reg/w0: Called action, logging to builtin-file 5056.400668930:main queue:Reg/w0: (/var/log/debug) 5056.400687895:main queue:Reg/w0: result of expression evaluation: 0 5056.400702536:main queue:Reg/w0: result of expression evaluation: 0 5056.400705763:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. From henry78 at gmx.at Tue Sep 8 12:41:37 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:41:37 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Message-ID: <1252406497.17741.48.camel@eberhe.office.chipkarte.at> The file (and folder) are created by the syslog user and definitely accessible. But it works if i don't drop privileges. So I'll investigate this further and report back. Thanks for pushing me that far. -- regard, Henry On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > Hi, > > I got the debug log, it was too big to be sent via the list (but I got it as > list admin). I see that you drop privileges to the user "syslog". This > probably explains what happens. I think the file is created before you drop > privileges, but can then no longer be written when running in the new > security context. Could you verify that the user "syslog" can access this > file? Also, could you temporarily remove the Privilege drop? > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 08, 2009 9:55 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > can you provide a debug log? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > Hello! > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > worked. > > > Is this worth ab bug? > > > > > > -- > > > kind regards, Henry > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > Hi! > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > > on > > > > ubuntu: > > > > > > > > -----8<----- > > > > $ModLoad imtcp > > > > $InputTCPServerRun 514 > > > > > > > > # some dynamic templates > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > # log remote local1 to dynamic diretory > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > $syslogfacility-text == 'local1' \ > > > > then -?DYNlocal1 > > > > ----->8----- > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > after > > > the > > > > remote host (myhostname) and creates the file local1.log (again: > > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > > that > > > > file, but logs the error: > > > > > > > > -----8<----- > > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > > - > > > > discarding message > > > > ----->8----- > > > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > > file > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:47:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:47:05 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at><9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> <1252406497.17741.48.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2E@GRFEXC.intern.adiscon.com> It is important to know that the PrivDrop directive set was a quick and dirty "let's implement it as far as possible, some is better than nothing" approach. It is expected that a couple of things break if it is used. Of course, if the users has proper rights, what you intend to do should work. I just wanted to alert you on the state of this feature (a mailing list search probably brings up more, but I have no time right now to do this). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 12:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > The file (and folder) are created by the syslog user and definitely > accessible. > > But it works if i don't drop privileges. So I'll investigate this > further and report back. > > Thanks for pushing me that far. > > -- > regard, Henry > > > On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > > Hi, > > > > I got the debug log, it was too big to be sent via the list (but I > got it as > > list admin). I see that you drop privileges to the user "syslog". > This > > probably explains what happens. I think the file is created before > you drop > > privileges, but can then no longer be written when running in the new > > security context. Could you verify that the user "syslog" can access > this > > file? Also, could you temporarily remove the Privilege drop? > > > > Thanks, > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > > Sent: Tuesday, September 08, 2009 9:55 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > can you provide a debug log? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] Could not open dynamic file ... - > discarding > > > > message > > > > > > > > Hello! > > > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > > worked. > > > > Is this worth ab bug? > > > > > > > > -- > > > > kind regards, Henry > > > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > > Hi! > > > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog > v4 > > > on > > > > > ubuntu: > > > > > > > > > > -----8<----- > > > > > $ModLoad imtcp > > > > > $InputTCPServerRun 514 > > > > > > > > > > # some dynamic templates > > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > > > # log remote local1 to dynamic diretory > > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > > $syslogfacility-text == 'local1' \ > > > > > then -?DYNlocal1 > > > > > ----->8----- > > > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > > after > > > > the > > > > > remote host (myhostname) and creates the file local1.log > (again: > > > > > sufficient permissions: syslog:syslog 640). But it doesn't > write to > > > > that > > > > > file, but logs the error: > > > > > > > > > > -----8<----- > > > > > Could not open dynamic file > '/var/log/remote/myhostname/local1.log' > > > - > > > > > discarding message > > > > > ----->8----- > > > > > > > > > > As you might guess my question is: Why isn't rsyslog able to > open a > > > > file > > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 13:23:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 08 Sep 2009 13:23:10 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <1252408990.17679.10.camel@rgf11> Hi all, I am really banging my head on a problem which sounds too easy. I have seen that my systems (and some others as well), seem to not provide more than 1024 bytes on a recvfrom() call. With wireshark, I see that the system itself, at the IP layer, receives more data. I am a bit puzzled, to phrase it lightly. I did not find any information on such a limitation. I have created a strip-down version of a receiver, even built it on top of the Linux man pages samples. Out of desperation, I even set the receivebuf size, which I think has no effect on datagram sockets. Still... I only get 1024 bytes. Code is after my sig. Does anybody have an idea what is going on OR a good place where to ask this question? Thanks, Rainer #include #include #include #include #include #include #include #define BUF_SIZE 2048 int main(int argc, char *argv[]) { struct addrinfo hints; struct addrinfo *result, *rp; int sfd, s; struct sockaddr_storage peer_addr; socklen_t peer_addr_len; ssize_t nread; char buf[BUF_SIZE]; if (argc != 2) { fprintf(stderr, "Usage: %s port\n", argv[0]); exit(EXIT_FAILURE); } memset(&hints, 0, sizeof(struct addrinfo)); hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ hints.ai_protocol = 0; /* Any protocol */ hints.ai_canonname = NULL; hints.ai_addr = NULL; hints.ai_next = NULL; s = getaddrinfo(NULL, argv[1], &hints, &result); if (s != 0) { fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); exit(EXIT_FAILURE); } /* getaddrinfo() returns a list of address structures. Try each address until we successfully bind(2). If socket(2) (or bind(2)) fails, we (close the socket and) try the next address. */ for (rp = result; rp != NULL; rp = rp->ai_next) { sfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (sfd == -1) continue; int result2; int bufSize = 2048; result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, sizeof(bufSize)); printf("result of setsockopt: %d\n", result2); if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) break; /* Success */ close(sfd); } if (rp == NULL) { /* No address succeeded */ fprintf(stderr, "Could not bind\n"); exit(EXIT_FAILURE); } freeaddrinfo(result); /* No longer needed */ /* Read datagrams and echo them back to sender */ for (;;) { peer_addr_len = sizeof(struct sockaddr_storage); memset(buf, 0, BUF_SIZE); nread = recvfrom(sfd, buf, BUF_SIZE, 0, (struct sockaddr *) &peer_addr, &peer_addr_len); if(nread > 1024) printf("NREAD > 1024!"); if (nread == -1) continue; /* Ignore failed request */ char host[NI_MAXHOST], service[NI_MAXSERV]; s = getnameinfo((struct sockaddr *) &peer_addr, peer_addr_len, host, NI_MAXHOST, service, NI_MAXSERV, NI_NUMERICSERV); if (s == 0) printf("Received %ld bytes from %s:%s, msg:'%s'\n", (long) nread, host, service, buf); else fprintf(stderr, "getnameinfo: %s\n", gai_strerror(s)); } } From rgerhards at hq.adiscon.com Tue Sep 8 14:17:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 14:17:05 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <1252408990.17679.10.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> oh my... Please disregard this question. I was working on a tcpdump file, and the message length actually *is* 1024 bytes. I was confused by Wireshark's (correct!) indication that the frame is 1066 octets in length. Of course, this is correct, if you take the 42 octets of UDP header into account... I guess the dump file was created with a max of 1K... Sometimes it is sooo easy ... and yet so hard to see ;) Sorry for the interruption, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 1:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Help requested: UDP max message size? > > Hi all, > > I am really banging my head on a problem which sounds too easy. I have > seen that my systems (and some others as well), seem to not provide > more > than 1024 bytes on a recvfrom() call. With wireshark, I see that the > system itself, at the IP layer, receives more data. I am a bit puzzled, > to phrase it lightly. I did not find any information on such a > limitation. > > I have created a strip-down version of a receiver, even built it on top > of the Linux man pages samples. Out of desperation, I even set the > receivebuf size, which I think has no effect on datagram sockets. > Still... I only get 1024 bytes. Code is after my sig. > > Does anybody have an idea what is going on OR a good place where to ask > this question? > > Thanks, > Rainer > > #include > #include > #include > #include > #include > #include > #include > > #define BUF_SIZE 2048 > > int > main(int argc, char *argv[]) > { > struct addrinfo hints; > struct addrinfo *result, *rp; > int sfd, s; > struct sockaddr_storage peer_addr; > socklen_t peer_addr_len; > ssize_t nread; > char buf[BUF_SIZE]; > > if (argc != 2) { > fprintf(stderr, "Usage: %s port\n", argv[0]); > exit(EXIT_FAILURE); > } > > memset(&hints, 0, sizeof(struct addrinfo)); > hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ > hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > hints.ai_flags = AI_PASSIVE; /* For wildcard IP address > */ > hints.ai_protocol = 0; /* Any protocol */ > hints.ai_canonname = NULL; > hints.ai_addr = NULL; > hints.ai_next = NULL; > > s = getaddrinfo(NULL, argv[1], &hints, &result); > if (s != 0) { > fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); > exit(EXIT_FAILURE); > } > > /* getaddrinfo() returns a list of address structures. > Try each address until we successfully bind(2). > If socket(2) (or bind(2)) fails, we (close the socket > and) try the next address. */ > > for (rp = result; rp != NULL; rp = rp->ai_next) { > sfd = socket(rp->ai_family, rp->ai_socktype, > rp->ai_protocol); > if (sfd == -1) > continue; > > > int result2; > int bufSize = 2048; > result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > sizeof(bufSize)); > printf("result of setsockopt: %d\n", result2); > > if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > break; /* Success */ > > close(sfd); > } > > if (rp == NULL) { /* No address succeeded */ > fprintf(stderr, "Could not bind\n"); > exit(EXIT_FAILURE); > } > > freeaddrinfo(result); /* No longer needed */ > > /* Read datagrams and echo them back to sender */ > for (;;) { > peer_addr_len = sizeof(struct sockaddr_storage); > memset(buf, 0, BUF_SIZE); > nread = recvfrom(sfd, buf, BUF_SIZE, 0, > (struct sockaddr *) &peer_addr, &peer_addr_len); > if(nread > 1024) > printf("NREAD > 1024!"); > if (nread == -1) > continue; /* Ignore failed request */ > > char host[NI_MAXHOST], service[NI_MAXSERV]; > > s = getnameinfo((struct sockaddr *) &peer_addr, > peer_addr_len, host, NI_MAXHOST, > service, NI_MAXSERV, NI_NUMERICSERV); > if (s == 0) > printf("Received %ld bytes from %s:%s, msg:'%s'\n", > (long) nread, host, service, buf); > else > fprintf(stderr, "getnameinfo: %s\n", > gai_strerror(s)); > } > } > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Tue Sep 8 17:24:59 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Tue, 08 Sep 2009 12:24:59 -0300 Subject: [rsyslog] FailoverSyslogServer: Write buffer immediatly to disk instead to memory option available? Message-ID: <4AA6774B.6070604@swiss-wireless.com.ar> Hello Short: rsyslog V3-V4: Can I write to disk ONLY if the remote rsyslog server is not reachable? Can it be done with the following? $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $ActionQueueType Disk $ActionQueueFileName srvrfwd # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure If the server is reachable there will be nothing written do disk (my problem is using CF card in embedded system see below) or is it written first to disk and than processed by the dispatcher? Long: I use rsyslog on AP which I try now to log remotely to a syslog server because CF card dies if you log often. Now the problem is, that I don't want to loss my syslog messages in the case the syslog server is not available. Now this messages are helt in the memory but if there is a power loss all messages will be lost. We have many power losses here :-( Greetings and thanks Beat From david at lang.hm Tue Sep 8 19:55:21 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 10:55:21 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> References: <1252408990.17679.10.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 8 20:38:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 20:38:12 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 08.09.09 19:55 Betreff: Re: [rsyslog] Help requested: UDP max message size? On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Sep 8 20:41:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 11:41:39 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) yes, as far as I know the none of the senders are rsyslog yet. I am working from the central server out. the central server is rsyslog with no problems all but this one relay box are rsyslog things sending to these relay boxes are whatever syslog sender was on the OS/appliance (there may be some acting as relays as well as sending for themselves) David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 08.09.09 19:55 > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > >> oh my... Please disregard this question. I was working on a tcpdump file, and >> the message length actually *is* 1024 bytes. I was confused by Wireshark's >> (correct!) indication that the frame is 1066 octets in length. Of course, >> this is correct, if you take the 42 octets of UDP header into account... >> >> I guess the dump file was created with a max of 1K... > > the dump file was set -s 0 (up to 64k packet size), but many/most syslog > senders will limit their outbound data to 1k > > David Lang > >> Sometimes it is sooo easy ... and yet so hard to see ;) >> >> Sorry for the interruption, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>> Sent: Tuesday, September 08, 2009 1:23 PM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] Help requested: UDP max message size? >>> >>> Hi all, >>> >>> I am really banging my head on a problem which sounds too easy. I have >>> seen that my systems (and some others as well), seem to not provide >>> more >>> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >>> system itself, at the IP layer, receives more data. I am a bit puzzled, >>> to phrase it lightly. I did not find any information on such a >>> limitation. >>> >>> I have created a strip-down version of a receiver, even built it on top >>> of the Linux man pages samples. Out of desperation, I even set the >>> receivebuf size, which I think has no effect on datagram sockets. >>> Still... I only get 1024 bytes. Code is after my sig. >>> >>> Does anybody have an idea what is going on OR a good place where to ask >>> this question? >>> >>> Thanks, >>> Rainer >>> >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> #define BUF_SIZE 2048 >>> >>> int >>> main(int argc, char *argv[]) >>> { >>> struct addrinfo hints; >>> struct addrinfo *result, *rp; >>> int sfd, s; >>> struct sockaddr_storage peer_addr; >>> socklen_t peer_addr_len; >>> ssize_t nread; >>> char buf[BUF_SIZE]; >>> >>> if (argc != 2) { >>> fprintf(stderr, "Usage: %s port\n", argv[0]); >>> exit(EXIT_FAILURE); >>> } >>> >>> memset(&hints, 0, sizeof(struct addrinfo)); >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >>> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >>> */ >>> hints.ai_protocol = 0; /* Any protocol */ >>> hints.ai_canonname = NULL; >>> hints.ai_addr = NULL; >>> hints.ai_next = NULL; >>> >>> s = getaddrinfo(NULL, argv[1], &hints, &result); >>> if (s != 0) { >>> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >>> exit(EXIT_FAILURE); >>> } >>> >>> /* getaddrinfo() returns a list of address structures. >>> Try each address until we successfully bind(2). >>> If socket(2) (or bind(2)) fails, we (close the socket >>> and) try the next address. */ >>> >>> for (rp = result; rp != NULL; rp = rp->ai_next) { >>> sfd = socket(rp->ai_family, rp->ai_socktype, >>> rp->ai_protocol); >>> if (sfd == -1) >>> continue; >>> >>> >>> int result2; >>> int bufSize = 2048; >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >>> sizeof(bufSize)); >>> printf("result of setsockopt: %d\n", result2); >>> >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >>> break; /* Success */ >>> >>> close(sfd); >>> } >>> >>> if (rp == NULL) { /* No address succeeded */ >>> fprintf(stderr, "Could not bind\n"); >>> exit(EXIT_FAILURE); >>> } >>> >>> freeaddrinfo(result); /* No longer needed */ >>> >>> /* Read datagrams and echo them back to sender */ >>> for (;;) { >>> peer_addr_len = sizeof(struct sockaddr_storage); >>> memset(buf, 0, BUF_SIZE); >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >>> (struct sockaddr *) &peer_addr, &peer_addr_len); >>> if(nread > 1024) >>> printf("NREAD > 1024!"); >>> if (nread == -1) >>> continue; /* Ignore failed request */ >>> >>> char host[NI_MAXHOST], service[NI_MAXSERV]; >>> >>> s = getnameinfo((struct sockaddr *) &peer_addr, >>> peer_addr_len, host, NI_MAXHOST, >>> service, NI_MAXSERV, NI_NUMERICSERV); >>> if (s == 0) >>> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >>> (long) nread, host, service, buf); >>> else >>> fprintf(stderr, "getnameinfo: %s\n", >>> gai_strerror(s)); >>> } >>> } >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 21:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 21:24:04 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE36@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 08, 2009 8:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > Was there an non-rsyslog relay in the relay chain? If not, > it points to the rsyslog forwarding module doing the > truncation (what recent v3+ i think should not do...) > > yes, as far as I know the none of the senders are rsyslog yet. Well, from what I see in the tcpdump logs, the initial sender is rsyslog and the messages originated from imklog. I can point you to the entries in question, but I don't have logs with me now. Rainer > > I am working from the central server out. > > the central server is rsyslog with no problems > > all but this one relay box are rsyslog > > things sending to these relay boxes are whatever syslog > sender was on the > OS/appliance (there may be some acting as relays as well as > sending for > themselves) > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 08.09.09 19:55 > > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > >> oh my... Please disregard this question. I was working on > a tcpdump file, and > >> the message length actually *is* 1024 bytes. I was > confused by Wireshark's > >> (correct!) indication that the frame is 1066 octets in > length. Of course, > >> this is correct, if you take the 42 octets of UDP header > into account... > >> > >> I guess the dump file was created with a max of 1K... > > > > the dump file was set -s 0 (up to 64k packet size), but > many/most syslog > > senders will limit their outbound data to 1k > > > > David Lang > > > >> Sometimes it is sooo easy ... and yet so hard to see ;) > >> > >> Sorry for the interruption, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>> Sent: Tuesday, September 08, 2009 1:23 PM > >>> To: rsyslog at lists.adiscon.com > >>> Subject: [rsyslog] Help requested: UDP max message size? > >>> > >>> Hi all, > >>> > >>> I am really banging my head on a problem which sounds too > easy. I have > >>> seen that my systems (and some others as well), seem to > not provide > >>> more > >>> than 1024 bytes on a recvfrom() call. With wireshark, I > see that the > >>> system itself, at the IP layer, receives more data. I am > a bit puzzled, > >>> to phrase it lightly. I did not find any information on such a > >>> limitation. > >>> > >>> I have created a strip-down version of a receiver, even > built it on top > >>> of the Linux man pages samples. Out of desperation, I even set the > >>> receivebuf size, which I think has no effect on datagram sockets. > >>> Still... I only get 1024 bytes. Code is after my sig. > >>> > >>> Does anybody have an idea what is going on OR a good > place where to ask > >>> this question? > >>> > >>> Thanks, > >>> Rainer > >>> > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> > >>> #define BUF_SIZE 2048 > >>> > >>> int > >>> main(int argc, char *argv[]) > >>> { > >>> struct addrinfo hints; > >>> struct addrinfo *result, *rp; > >>> int sfd, s; > >>> struct sockaddr_storage peer_addr; > >>> socklen_t peer_addr_len; > >>> ssize_t nread; > >>> char buf[BUF_SIZE]; > >>> > >>> if (argc != 2) { > >>> fprintf(stderr, "Usage: %s port\n", argv[0]); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> memset(&hints, 0, sizeof(struct addrinfo)); > >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 > or IPv6 */ > >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > >>> hints.ai_flags = AI_PASSIVE; /* For > wildcard IP address > >>> */ > >>> hints.ai_protocol = 0; /* Any protocol */ > >>> hints.ai_canonname = NULL; > >>> hints.ai_addr = NULL; > >>> hints.ai_next = NULL; > >>> > >>> s = getaddrinfo(NULL, argv[1], &hints, &result); > >>> if (s != 0) { > >>> fprintf(stderr, "getaddrinfo: %s\n", > gai_strerror(s)); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> /* getaddrinfo() returns a list of address structures. > >>> Try each address until we successfully bind(2). > >>> If socket(2) (or bind(2)) fails, we (close > the socket > >>> and) try the next address. */ > >>> > >>> for (rp = result; rp != NULL; rp = rp->ai_next) { > >>> sfd = socket(rp->ai_family, rp->ai_socktype, > >>> rp->ai_protocol); > >>> if (sfd == -1) > >>> continue; > >>> > >>> > >>> int result2; > >>> int bufSize = 2048; > >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > >>> sizeof(bufSize)); > >>> printf("result of setsockopt: %d\n", result2); > >>> > >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > >>> break; /* Success */ > >>> > >>> close(sfd); > >>> } > >>> > >>> if (rp == NULL) { /* No address > succeeded */ > >>> fprintf(stderr, "Could not bind\n"); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> freeaddrinfo(result); /* No longer needed */ > >>> > >>> /* Read datagrams and echo them back to sender */ > >>> for (;;) { > >>> peer_addr_len = sizeof(struct sockaddr_storage); > >>> memset(buf, 0, BUF_SIZE); > >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, > >>> (struct sockaddr *) &peer_addr, > &peer_addr_len); > >>> if(nread > 1024) > >>> printf("NREAD > 1024!"); > >>> if (nread == -1) > >>> continue; /* Ignore > failed request */ > >>> > >>> char host[NI_MAXHOST], service[NI_MAXSERV]; > >>> > >>> s = getnameinfo((struct sockaddr *) &peer_addr, > >>> peer_addr_len, host, NI_MAXHOST, > >>> service, NI_MAXSERV, > NI_NUMERICSERV); > >>> if (s == 0) > >>> printf("Received %ld bytes from %s:%s, > msg:'%s'\n", > >>> (long) nread, host, service, buf); > >>> else > >>> fprintf(stderr, "getnameinfo: %s\n", > >>> gai_strerror(s)); > >>> } > >>> } > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From corsmith at gmail.com Tue Sep 8 21:46:25 2009 From: corsmith at gmail.com (Corey Smith) Date: Tue, 8 Sep 2009 15:46:25 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> References: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909081246q6535c9d0s997f814c18e69c83@mail.gmail.com> On Thu, Sep 3, 2009 at 2:58 PM, Rainer Gerhards wrote: > Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) I come from a FreeBSD background so the Solaris package management system leaves much to be desired. The limitations of the default toolset in Solaris are amazing. That is why I started using pkgsrc - a portable package management system originally developed for netbsd. The way I got gcc44 working on Solaris 10/Sparc64: Download, install pkgsrc and bootstrap using the gcc from sunfreeware (3.4) # Check out: http://www.netbsd.org/docs/pkgsrc/platforms.html#solaris Install pkgsrc-wip using a cvs checkout # Check out: http://pkgsrc-wip.sourceforge.net/ Replace the wip/rsyslog port with the one I attached earlier on the thread. Build rsyslog and dependencies using gcc3.4 Install gcc44 from wip/gcc44 and make the changes I described in the first message of the thread cd /usr/pkgsrc/wip/rsyslog && make update # rebuild rsyslog with gcc44 On a side note: I tried building rsyslog-5 from git which compiled but would core every time I started it. BTW: Which virtual machine are you using to emulate sparc64? -Corey Smith From rgerhards at hq.adiscon.com Wed Sep 9 15:00:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Sep 2009 15:00:55 +0200 Subject: [rsyslog] epoll-supporting imudp Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE43@GRFEXC.intern.adiscon.com> Hi all, I have finally begun to add some new features concurrently to my analysis of the reported segfault (which seems to be environment-induced and will very seldom show up in practice). I have now created an imudp-epoll branch, based on current master, which provides an imudp module that utilizes epoll() instead of select(). This is my first move towards supporting epoll() where useful. Please note that imudp will not tremendously benefit - on busy servers, select() is very infrequently called, as we read the socket as long there is data. On non-busy servers, there are few calls and I don't expect that epoll vs. select makes any real difference then. Please note that the most benefit from epoll we will gain on tcp based traffic. However, moving to epoll there is far more complicated, because I need to remodel the netstream driver layer. Thus I wanted to gain some experience with easy things first. Probably imuxsock is my next target after I have waited some time for feedback. I would appreciate if some folks could try out the new branch and tell me their experience. I plan to include the new functionality with the next v5-devel release in a couple of days. Rainer From joshsystem at gmail.com Thu Sep 10 07:45:28 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 13:45:28 +0800 Subject: [rsyslog] does rsyslog supports data analytic Message-ID: hi all, i want to receive each syslog msg then input it into my special processing module.after processing the data,output the new data into database.of course,the raw data we must keep it into files. can anyone give me some suggestions? PS: i browse the git source code, but i can't understand why the Experimental-lockfree is not adopted? thanks From david at lang.hm Thu Sep 10 08:26:09 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 9 Sep 2009 23:26:09 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: Message-ID: On Thu, 10 Sep 2009, Josh Zhao wrote: > hi all, i want to receive each syslog msg then input it into my special > processing module.after processing the data,output the new data into > database.of course,the raw data we must keep it into files. can anyone give > me some suggestions? would you not just list two destinations, one to the place you want the raw data archived and one to the processing module? I have a very high volume of logs (>300M/day), so I roll the logs every 5 min with this script #!/bin/sh # PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin umask 022 year=`date +%Y` month=`date +%m` day=`date +%d` fdate=`date +%Y%m%d.%H%M` logroot=/var/log logroll=$logroot/oldlogs cd $logroot mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null mv messages messages.$fdate mv messages.$fdate $logroll/$year/$month/$day/messages.$fdate mv /usr/local/bin/ita/system/itascan1a-p/winlogs /usr/local/bin/ita/system/itascan1a-p/winlogs.0 pkill -HUP syslogd pkill -HUP syslog-ng #pkill win-dump gzip -9 $logroll/$year/$month/$day/messages.$fdate > PS: i browse the git source code, but i can't understand why the > Experimental-lockfree > is > not adopted? I believe that it boils down to complications in being sure that there are no bugs, and the fact that even without that there has been a LOT of room for improvement from the early 3.x timeframe to the current 5.x version. I expect that after the current round of improvements are settled that aspect of things will get reexamined. David Lang From rgerhards at hq.adiscon.com Thu Sep 10 08:32:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 08:32:08 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, September 10, 2009 8:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > PS: i browse the git source code, but i can't understand why the > > > Experimental-lockfree > > is > > not adopted? > > I believe that it boils down to complications in being sure > that there are > no bugs, and the fact that even without that there has been a > LOT of room > for improvement from the early 3.x timeframe to the current > 5.x version. > > I expect that after the current round of improvements are > settled that > aspect of things will get reexamined. That branch is mostly there for historical reasons. I keep that branch as a think-tank, but it is is obsoleted. Also, in less polite words than David used, it simply doesn't work. Getting this code with multiple producers and consumers correct is far from being trivial and the literature I browsed indicates that it is probably not possible given the other predicates the code must obey to. Still, optimization is high up on the todo list. Rainer From joshsystem at gmail.com Thu Sep 10 15:25:23 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 21:25:23 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: Thanks for David and Rainer's reply.I m sorry that I did not explain my question clearly.I m new to rsyslog and want to add a processing module in rsyslog.The rsyslog has input plugins(front-end) and output plugins(back-end).My processing module receives data from input plugins and output the processed data and raw data both into output plugins.So how I add it? 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, September 10, 2009 8:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > PS: i browse the git source code, but i can't understand why the > > > > > Experimental-lockfree shortlog;h=refs/heads/Experimental-lockfree> > > > is > > > not adopted? > > > > I believe that it boils down to complications in being sure > > that there are > > no bugs, and the fact that even without that there has been a > > LOT of room > > for improvement from the early 3.x timeframe to the current > > 5.x version. > > > > I expect that after the current round of improvements are > > settled that > > aspect of things will get reexamined. > > That branch is mostly there for historical reasons. I keep that branch as a > think-tank, but it is is obsoleted. Also, in less polite words than David > used, it simply doesn't work. Getting this code with multiple producers and > consumers correct is far from being trivial and the literature I browsed > indicates that it is probably not possible given the other predicates the > code must obey to. Still, optimization is high up on the todo list. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Sep 10 17:06:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 17:06:33 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Thursday, September 10, 2009 3:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module > in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins > and > output the processed data and raw data both into output plugins.So how > I add > it? What you are looking for is a library plugin. Unfortunaley, library plugins will work together with the scripting engine. In other words: there currently is no in-proc method available. What you can do, however, is chain two rsyslog instances, pipe data to your plugin and send that data to the other instance. Far from perfect and easy to do, but maybe a workable work-around... Rainer > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that > branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > used, it simply doesn't work. Getting this code with multiple > producers and > > consumers correct is far from being trivial and the literature I > browsed > > indicates that it is probably not possible given the other predicates > the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Thu Sep 10 22:50:30 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:50:30 +0200 Subject: [rsyslog] use snmp as source Message-ID: <4AA96696.9080906@irontec.com> Hi! I have a SNMP capable VoIP gateway, and I want to be able to log in syslog, the messages received by SNMP. Is this possible? I have read that in the other direction, it is possible. http://www.rsyslog.com/doc-omsnmp.html Thanks From mikel at irontec.com Thu Sep 10 22:52:50 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:52:50 +0200 Subject: [rsyslog] use snmp as source In-Reply-To: <4AA96696.9080906@irontec.com> References: <4AA96696.9080906@irontec.com> Message-ID: <4AA96722.4090307@irontec.com> The solution is snmptrapd Thanks!! Mikel Jimenez wrote: > Hi! > > I have a SNMP capable VoIP gateway, and I want to be able to log in > syslog, the messages received by SNMP. > > > Is this possible? > > I have read that in the other direction, it is possible. > http://www.rsyslog.com/doc-omsnmp.html > > Thanks > > From joshsystem at gmail.com Fri Sep 11 02:13:35 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 08:13:35 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: You mean I have to rewrite the processing module in rainerscript.where can i find the detailed documents related to the scripting engine? Thank you! 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Thursday, September 10, 2009 3:25 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > > and > > output the processed data and raw data both into output plugins.So how > > I add > > it? > > What you are looking for is a library plugin. Unfortunaley, library plugins > will work together with the scripting engine. In other words: there > currently > is no in-proc method available. > > What you can do, however, is chain two rsyslog instances, pipe data to your > plugin and send that data to the other instance. Far from perfect and easy > to > do, but maybe a workable work-around... > > Rainer > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that > > branch as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > > David > > > used, it simply doesn't work. Getting this code with multiple > > producers and > > > consumers correct is far from being trivial and the literature I > > browsed > > > indicates that it is probably not possible given the other predicates > > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 02:26:46 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 17:26:46 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > You mean I have to rewrite the processing module in rainerscript.where can i > find the detailed documents related to the scripting engine? right now rainerscript is as much an idea as an implementation. it can be used for a few things, but mostly just for filter 'does this log match X' type of things. David Lang > Thank you! > 2009/9/10 Rainer Gerhards > >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>> Sent: Thursday, September 10, 2009 3:25 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>> >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>> question clearly.I m new to rsyslog and want to add a processing module >>> in >>> rsyslog.The rsyslog has input plugins(front-end) and output >>> plugins(back-end).My processing module receives data from input plugins >>> and >>> output the processed data and raw data both into output plugins.So how >>> I add >>> it? >> >> What you are looking for is a library plugin. Unfortunaley, library plugins >> will work together with the scripting engine. In other words: there >> currently >> is no in-proc method available. >> >> What you can do, however, is chain two rsyslog instances, pipe data to your >> plugin and send that data to the other instance. Far from perfect and easy >> to >> do, but maybe a workable work-around... >> >> Rainer >> >>> >>> >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>> >>>>>> PS: i browse the git source code, but i can't understand why the >>>>>> >>>>> Experimental-lockfree>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>> is >>>>>> not adopted? >>>>> >>>>> I believe that it boils down to complications in being sure >>>>> that there are >>>>> no bugs, and the fact that even without that there has been a >>>>> LOT of room >>>>> for improvement from the early 3.x timeframe to the current >>>>> 5.x version. >>>>> >>>>> I expect that after the current round of improvements are >>>>> settled that >>>>> aspect of things will get reexamined. >>>> >>>> That branch is mostly there for historical reasons. I keep that >>> branch as a >>>> think-tank, but it is is obsoleted. Also, in less polite words than >>> David >>>> used, it simply doesn't work. Getting this code with multiple >>> producers and >>>> consumers correct is far from being trivial and the literature I >>> browsed >>>> indicates that it is probably not possible given the other predicates >>> the >>>> code must obey to. Still, optimization is high up on the todo list. >>>> >>>> Rainer >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 03:39:01 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 09:39:01 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Is rsyslog no way to reslove problem, What about syslog-ng? What I think about,rsyslog's multi-thread archititure is better for my mulit-core hardware. The logs data is very high volume too. Could you give me any suggestion on this matter? Thank you! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > You mean I have to rewrite the processing module in rainerscript.where > can i > > find the detailed documents related to the scripting engine? > > right now rainerscript is as much an idea as an implementation. it can be > used for a few things, but mostly just for filter 'does this log match X' > type of things. > > David Lang > > > Thank you! > > 2009/9/10 Rainer Gerhards > > > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>> Sent: Thursday, September 10, 2009 3:25 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>> > >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my > >>> question clearly.I m new to rsyslog and want to add a processing module > >>> in > >>> rsyslog.The rsyslog has input plugins(front-end) and output > >>> plugins(back-end).My processing module receives data from input plugins > >>> and > >>> output the processed data and raw data both into output plugins.So how > >>> I add > >>> it? > >> > >> What you are looking for is a library plugin. Unfortunaley, library > plugins > >> will work together with the scripting engine. In other words: there > >> currently > >> is no in-proc method available. > >> > >> What you can do, however, is chain two rsyslog instances, pipe data to > your > >> plugin and send that data to the other instance. Far from perfect and > easy > >> to > >> do, but maybe a workable work-around... > >> > >> Rainer > >> > >>> > >>> > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>> david at lang.hm > >>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>> > >>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>> > >>>>> Experimental-lockfree >>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>> is > >>>>>> not adopted? > >>>>> > >>>>> I believe that it boils down to complications in being sure > >>>>> that there are > >>>>> no bugs, and the fact that even without that there has been a > >>>>> LOT of room > >>>>> for improvement from the early 3.x timeframe to the current > >>>>> 5.x version. > >>>>> > >>>>> I expect that after the current round of improvements are > >>>>> settled that > >>>>> aspect of things will get reexamined. > >>>> > >>>> That branch is mostly there for historical reasons. I keep that > >>> branch as a > >>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>> David > >>>> used, it simply doesn't work. Getting this code with multiple > >>> producers and > >>>> consumers correct is far from being trivial and the literature I > >>> browsed > >>>> indicates that it is probably not possible given the other predicates > >>> the > >>>> code must obey to. Still, optimization is high up on the todo list. > >>>> > >>>> Rainer > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 06:28:59 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 21:28:59 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > about,rsyslog's multi-thread archititure is better for my mulit-core > hardware. The logs data is very high volume too. Could you give me any > suggestion on this matter? my experiance with syslog-ng was not good, so I'm not the right person to talk about doing this sort of thing with it. but I am not aware of any syslog daemon that lets you insert your own logic in the middle of the processing. rsyslog has the concept, but it has not been implemented (fixing bugs and speeding it up has taken priority) what sort of volume do you consider 'high'? (it's amazing the range that this can span, so I've learned to ask rather than assume ;-) since you are needing to get your final data into a database, I think that you will find that rsyslog will (or will soon) suit your needs far better than alternate approaches. the ability to process multiple messages in one transaction that is being developed will be a huge improvement in terms of database interaction. I would look at what rainer suggested for now. have one copy of rsyslog that receives the messages, does whatever formatting/cleanup is needed on them, then passes the logs to one or more instances of your code to do additional processing, which can then feed the results into another instance of rsyslog to forward them on, insert them into a database, etc. when rainerscript gains the capability to alter the fields (instead of just testing them), then there will be a lot more that can be done inside rsyslog. David Lang > Thank you! > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> You mean I have to rewrite the processing module in rainerscript.where >> can i >>> find the detailed documents related to the scripting engine? >> >> right now rainerscript is as much an idea as an implementation. it can be >> used for a few things, but mostly just for filter 'does this log match X' >> type of things. >> >> David Lang >> >>> Thank you! >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>> >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>>>> question clearly.I m new to rsyslog and want to add a processing module >>>>> in >>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>> plugins(back-end).My processing module receives data from input plugins >>>>> and >>>>> output the processed data and raw data both into output plugins.So how >>>>> I add >>>>> it? >>>> >>>> What you are looking for is a library plugin. Unfortunaley, library >> plugins >>>> will work together with the scripting engine. In other words: there >>>> currently >>>> is no in-proc method available. >>>> >>>> What you can do, however, is chain two rsyslog instances, pipe data to >> your >>>> plugin and send that data to the other instance. Far from perfect and >> easy >>>> to >>>> do, but maybe a workable work-around... >>>> >>>> Rainer >>>> >>>>> >>>>> >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>> david at lang.hm >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>> >>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>> >>>>>>> Experimental-lockfree>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>> is >>>>>>>> not adopted? >>>>>>> >>>>>>> I believe that it boils down to complications in being sure >>>>>>> that there are >>>>>>> no bugs, and the fact that even without that there has been a >>>>>>> LOT of room >>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>> 5.x version. >>>>>>> >>>>>>> I expect that after the current round of improvements are >>>>>>> settled that >>>>>>> aspect of things will get reexamined. >>>>>> >>>>>> That branch is mostly there for historical reasons. I keep that >>>>> branch as a >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>> David >>>>>> used, it simply doesn't work. Getting this code with multiple >>>>> producers and >>>>>> consumers correct is far from being trivial and the literature I >>>>> browsed >>>>>> indicates that it is probably not possible given the other predicates >>>>> the >>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>> >>>>>> Rainer >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 08:16:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 08:16:43 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE6D@GRFEXC.intern.adiscon.com> Hi Davd, I think you hit it right on the nail. But I have also thought a bit more about the idea. Actually, I think, one can implement processing modules right now. Especially the configuration is a bit tricky, but it should really work. The rough outline is to use an output module for that. Output modules may do whatever they want as long as they use the provided interfaces. As such, they can also inject messages. So the idea is to define an output module, that accepts the message, does any processing necessary, indicated RS_RET_DISCARD to the rule engine (to prevent the message from being further processed) and inject the "newly generated" message back into the main message queue. That would also be much faster than whatever RainerScript will have to offer, because RainerScript relies on VM execution. I just don't have time to elaborately talk someone through this approach... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, September 11, 2009 6:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? > What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you > give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the > right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the > concept, but it has > not been implemented (fixing bugs and speeding it up has > taken priority) > > what sort of volume do you consider 'high'? (it's amazing the > range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, > I think that > you will find that rsyslog will (or will soon) suit your > needs far better > than alternate approaches. the ability to process multiple > messages in one > transaction that is being developed will be a huge > improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to > one or more > instances of your code to do additional processing, which can > then feed > the results into another instance of rsyslog to forward them > on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields > (instead of > just testing them), then there will be a lot more that can be > done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in > rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an > implementation. it can be > >> used for a few things, but mostly just for filter 'does > this log match X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I > did not explain my > >>>>> question clearly.I m new to rsyslog and want to add a > processing module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data > from input plugins > >>>>> and > >>>>> output the processed data and raw data both into output > plugins.So how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. > Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other > words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog > instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far > from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't > understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less > polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the > literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the > other predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on > the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 10:17:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 10:17:12 +0200 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <1252657032.17679.12.camel@rgf11> Now that I got an idea of how this could be implemented with current rsyslog technology, I would be interested in some more details of what you intend to do with the processing module. What exactly will it do with the message? I am asking because I would like to see a real use case. Thinking about the scenario I have proposed in my last mail, I think I see some pitfalls and I am not sure if they will cause any trouble in real projects. So I would appreciate if you could provide more in-depth info. Thanks, Rainer On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins and > output the processed data and raw data both into output plugins.So how I add > it? > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than David > > used, it simply doesn't work. Getting this code with multiple producers and > > consumers correct is far from being trivial and the literature I browsed > > indicates that it is probably not possible given the other predicates the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From thomas.mieslinger at 1und1.de Fri Sep 11 11:47:28 2009 From: thomas.mieslinger at 1und1.de (Thomas Mieslinger) Date: Fri, 11 Sep 2009 11:47:28 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelp transports In-Reply-To: <1236002254.28865.46.camel@rf10up.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com> <1236002254.28865.46.camel@rf10up.intern.adiscon.com> Message-ID: <4AAA1CB0.90106@1und1.de> Hi, I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. One machine (logsender) has: $ModLoad omrelp.so $WorkDirectory /var/spool/rsyslog $ActionQueueType LinkedList $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on user.* :omrelp:loghost:2514 or user.* @@loghost:1514 and the other machine (loghost) has $ModLoad imrelp.so $UDPServerRun 514 $InputTCPServerRun 1514 $InputRELPServerRun 2514 *.* -/some/logfile If I restart rsyslog on loghost without restarting rsyslog on logsender, the logs produced on logsender never appear on loghost. Is this working as designed? Is there a kind of syslog.debug facility where I can monitor the reconnect activity of rsyslog? Thanks in advance Thomas From rgerhards at hq.adiscon.com Fri Sep 11 12:13:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 12:13:12 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelptransports References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com><1236002254.28865.46.camel@rf10up.intern.adiscon.com> <4AAA1CB0.90106@1und1.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE79@GRFEXC.intern.adiscon.com> I suggest to turn on debug logging on both the client and sender: http://www.rsyslog.com/doc-troubleshoot.html Often, the debug log points to an obvious problem source. If it does not, feel free to mail me the logs to rgerhards at gmail.com BUT let me know you did so - I usually do not monitor this account. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger > Sent: Friday, September 11, 2009 11:47 AM > To: rsyslog-users > Subject: [rsyslog] rsyslogd not reconnecting when using tcp or > omrelptransports > > Hi, > > I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. > One > machine (logsender) has: > > $ModLoad omrelp.so > $WorkDirectory /var/spool/rsyslog > $ActionQueueType LinkedList > $ActionQueueFileName srvrfwd > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > > user.* :omrelp:loghost:2514 > or > user.* @@loghost:1514 > > and the other machine (loghost) > has > > $ModLoad imrelp.so > $UDPServerRun 514 > $InputTCPServerRun 1514 > $InputRELPServerRun 2514 > > *.* -/some/logfile > > If I restart rsyslog on loghost without restarting rsyslog on > logsender, > the logs produced on logsender never appear on loghost. Is this working > as designed? > > Is there a kind of syslog.debug facility where I can monitor the > reconnect activity of rsyslog? > > Thanks in advance Thomas > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Sep 11 15:21:34 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 11 Sep 2009 15:21:34 +0200 Subject: [rsyslog] rsyslog 5.1.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE82@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.5. This is the first public beta of the v5 branch. As such, it is an important milestone on the way to an even more powerful rsyslogd. As of our usual policies, this means that the first v5-stable will probably available within two to three month, so before the end of the year. Please note that this also means we are shifting our development efforts primarily to v5 for any new functionality (but we keep the option open to add some enhancements to v4-devel). Feedback and bug reports on the new v5-bea branch would be deeply appreciated. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-174.phtml Changelog: http://www.rsyslog.com/Article400.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From joshsystem at gmail.com Fri Sep 11 16:17:14 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 22:17:14 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Our raw data is "high" volume that means to prcocess data about 100M/min. Yes, I want to improve the system performance as soon as possibe. As you said,rsyslog has a concept that inserts my logic module into it ,but it was not been implemented. Could you point out in detail? The rainerscript seems not that strong,otherwise, it is a good idea for user interface. 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the concept, but it has > not been implemented (fixing bugs and speeding it up has taken priority) > > what sort of volume do you consider 'high'? (it's amazing the range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, I think that > you will find that rsyslog will (or will soon) suit your needs far better > than alternate approaches. the ability to process multiple messages in one > transaction that is being developed will be a huge improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to one or more > instances of your code to do additional processing, which can then feed > the results into another instance of rsyslog to forward them on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields (instead of > just testing them), then there will be a lot more that can be done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an implementation. it can > be > >> used for a few things, but mostly just for filter 'does this log match > X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain > my > >>>>> question clearly.I m new to rsyslog and want to add a processing > module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data from input > plugins > >>>>> and > >>>>> output the processed data and raw data both into output plugins.So > how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the other > predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 17:09:32 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 23:09:32 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <1252657032.17679.12.camel@rgf11> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> Message-ID: Thanks Rainer. The basic purpose is statistics,which can accumulate some fields of msgs,but I think the customers have more weird requirements. 2009/9/11 Rainer Gerhards > Now that I got an idea of how this could be implemented with current > rsyslog technology, I would be interested in some more details of what > you intend to do with the processing module. What exactly will it do > with the message? I am asking because I would like to see a real use > case. Thinking about the scenario I have proposed in my last mail, I > think I see some pitfalls and I am not sure if they will cause any > trouble in real projects. > > So I would appreciate if you could provide more in-depth info. > > Thanks, > Rainer > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > and > > output the processed data and raw data both into output plugins.So how I > add > > it? > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that branch > as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > > used, it simply doesn't work. Getting this code with multiple producers > and > > > consumers correct is far from being trivial and the literature I > browsed > > > indicates that it is probably not possible given the other predicates > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:18:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:18:27 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><1252657032.17679.12.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> OK, that's not very precise, but I have also thought a bit about this. What I have proposed this morning should be possible. But you should be warned, it requires a lot of reading and understanding the source code. A good place to start is the template input and output modules as well as some actual output modules. I think imdiag would be useful (because it is simple) and probably also either omstdout (simple) and omoracle (complex, but utilizes the vector interface which may be the best choice for what you intend to acomplish). As a side-note, if this is paid work you may want to think about purchasing some development help from Adiscon, which may dramatically reduce the time you need to get started (just a thought, omoracle was crafted very well without any such help - thanks again!). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Friday, September 11, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks Rainer. The basic purpose is statistics,which can accumulate > some > fields of msgs,but I think the customers have more weird requirements. > > 2009/9/11 Rainer Gerhards > > > Now that I got an idea of how this could be implemented with current > > rsyslog technology, I would be interested in some more details of > what > > you intend to do with the processing module. What exactly will it do > > with the message? I am asking because I would like to see a real use > > case. Thinking about the scenario I have proposed in my last mail, I > > think I see some pitfalls and I am not sure if they will cause any > > trouble in real projects. > > > > So I would appreciate if you could provide more in-depth info. > > > > Thanks, > > Rainer > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > Thanks for David and Rainer's reply.I m sorry that I did not > explain my > > > question clearly.I m new to rsyslog and want to add a processing > module > > in > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > plugins(back-end).My processing module receives data from input > plugins > > and > > > output the processed data and raw data both into output plugins.So > how I > > add > > > it? > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > PS: i browse the git source code, but i can't understand why > the > > > > > > > > > > > Experimental-lockfree > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > is > > > > > > not adopted? > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > that there are > > > > > no bugs, and the fact that even without that there has been a > > > > > LOT of room > > > > > for improvement from the early 3.x timeframe to the current > > > > > 5.x version. > > > > > > > > > > I expect that after the current round of improvements are > > > > > settled that > > > > > aspect of things will get reexamined. > > > > > > > > That branch is mostly there for historical reasons. I keep that > branch > > as a > > > > think-tank, but it is is obsoleted. Also, in less polite words > than > > David > > > > used, it simply doesn't work. Getting this code with multiple > producers > > and > > > > consumers correct is far from being trivial and the literature I > > browsed > > > > indicates that it is probably not possible given the other > predicates > > the > > > > code must obey to. Still, optimization is high up on the todo > list. > > > > > > > > Rainer > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Sep 11 17:24:04 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 08:24:04 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Our raw data is "high" volume that means to prcocess data about 100M/min. is this 100M log records, or 100MB of log data (if the latter, approximatly how large are the recors, of how many log records/min) I'm currently processing ~300K messages averaging ~256 bytes/message for a total of ~75MB of logs/min. in my testing v4 will support up to about 6x this volume before it runs into problems (it can receive them faster, up to gig-E wire speed, the limit is in the output, which is ~80K records a sec if doing trivial work like writing them to disk or ~30K records/sec if doing more complex things like forwarding them elsewhere) improvements in V5 include a batch mode that lets an output module process up to N records at a time. I expect this to provide close to a Nx speedup to the output capabilities (with single log per action much of the overhead is in the queue locking, so multiple output workers doesn't help much, with batches not only is much more getting done per pass, but you have the possibility of each output thread taking long enough to get it's work done that it's effective to run more of them without locking contention being the bottleneck) this batch mode will be especially useful for database work as it will let you insert multiple messages in the database in a single transaction. what transport are you using to deliver the logs to your server? > Yes, I want to improve the system performance as soon as possibe. what is the bottleneck you are running into today (what syslog system are you using, etc)? > As you > said,rsyslog has a concept that inserts my logic module into it ,but it was > not been implemented. Could you point out in detail? The rainerscript seems > not that strong,otherwise, it is a good idea for user interface. if you are looking at the source look for imtemplate and omtemplate, basicly he is suggesting creating a custom output module that rsyslog thinks is delivering the messages somewhere, have it be given the log, do it's processing, then acting like an input module and delivering the result to rsyslog as if it was a new message that just arrived. you will need to put some filters in rsyslog to keep your output module from seeing the logs that it creates, and either use discard or filters to keep the other output modules from seeing the raw input that your module is looking for. David Lang > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I think >>> about,rsyslog's multi-thread archititure is better for my mulit-core >>> hardware. The logs data is very high volume too. Could you give me any >>> suggestion on this matter? >> >> my experiance with syslog-ng was not good, so I'm not the right person to >> talk about doing this sort of thing with it. >> >> but I am not aware of any syslog daemon that lets you insert your own >> logic in the middle of the processing. rsyslog has the concept, but it has >> not been implemented (fixing bugs and speeding it up has taken priority) >> >> what sort of volume do you consider 'high'? (it's amazing the range that >> this can span, so I've learned to ask rather than assume ;-) >> >> since you are needing to get your final data into a database, I think that >> you will find that rsyslog will (or will soon) suit your needs far better >> than alternate approaches. the ability to process multiple messages in one >> transaction that is being developed will be a huge improvement in terms of >> database interaction. >> >> I would look at what rainer suggested for now. >> >> have one copy of rsyslog that receives the messages, does whatever >> formatting/cleanup is needed on them, then passes the logs to one or more >> instances of your code to do additional processing, which can then feed >> the results into another instance of rsyslog to forward them on, insert >> them into a database, etc. >> >> when rainerscript gains the capability to alter the fields (instead of >> just testing them), then there will be a lot more that can be done inside >> rsyslog. >> >> David Lang >> >>> Thank you! >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> You mean I have to rewrite the processing module in rainerscript.where >>>> can i >>>>> find the detailed documents related to the scripting engine? >>>> >>>> right now rainerscript is as much an idea as an implementation. it can >> be >>>> used for a few things, but mostly just for filter 'does this log match >> X' >>>> type of things. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>> >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain >> my >>>>>>> question clearly.I m new to rsyslog and want to add a processing >> module >>>>>>> in >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>> plugins(back-end).My processing module receives data from input >> plugins >>>>>>> and >>>>>>> output the processed data and raw data both into output plugins.So >> how >>>>>>> I add >>>>>>> it? >>>>>> >>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>> plugins >>>>>> will work together with the scripting engine. In other words: there >>>>>> currently >>>>>> is no in-proc method available. >>>>>> >>>>>> What you can do, however, is chain two rsyslog instances, pipe data to >>>> your >>>>>> plugin and send that data to the other instance. Far from perfect and >>>> easy >>>>>> to >>>>>> do, but maybe a workable work-around... >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>> david at lang.hm >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>> >>>>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>>>> >>>>>>>>> Experimental-lockfree>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>> is >>>>>>>>>> not adopted? >>>>>>>>> >>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>> that there are >>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>> LOT of room >>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>> 5.x version. >>>>>>>>> >>>>>>>>> I expect that after the current round of improvements are >>>>>>>>> settled that >>>>>>>>> aspect of things will get reexamined. >>>>>>>> >>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>> branch as a >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>>>> David >>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>> producers and >>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>> browsed >>>>>>>> indicates that it is probably not possible given the other >> predicates >>>>>>> the >>>>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>>>> >>>>>>>> Rainer >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:30:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:30:25 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE89@GRFEXC.intern.adiscon.com> > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, > do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. I think I did not state one important fact: this is not a dirty trick, but something that the engine was designed for. This mechanism was originally designed and is (somewhat) actually used to report back error conditions. It's used sparsely, because of the circular loop potential. But it is something the engine can handle and is designed to - so no abuse. Actually, I have begun to think if for some feature requests (string replacements before finally writing to an output) this may be good alternative approach. But it seems to involve more overhead than necessary for the job. > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters > to > keep the other output modules from seeing the raw input that your > module > is looking for. Returning RS_RET_DISCARD would solve this, as it stops processing. You just need to make sure that the newly injected messages don't go back into the same rule. With multiple rulesets we now have, this is trivial. But while all this is interesting, I unfortunately have more pressing things to do ;) Rainer > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit- > core > >>> hardware. The logs data is very high volume too. Could you give me > any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right > person to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your > own > >> logic in the middle of the processing. rsyslog has the concept, but > it has > >> not been implemented (fixing bugs and speeding it up has taken > priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range > that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I > think that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages > in one > >> transaction that is being developed will be a huge improvement in > terms of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then > feed > >> the results into another instance of rsyslog to forward them on, > insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead > of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it > can > >> be > >>>> used for a few things, but mostly just for filter 'does this log > match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a > processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output > plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, > library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: > there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe > data to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from > perfect and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand > why the > >>>>>>>>>> > >>>>>>>>> Experimental- > lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep > that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature > I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:39:30 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:39:30 -0600 Subject: [rsyslog] How to increase message size maximum Message-ID: Hello, I replied to an older thread on the forums but wanted to bring this up here. I have an application logging to rsyslog (version 3.22.0) which sends very large messages. We are trying to migrate logging from syslog-ng to rsyslog and I'm running into a problem where messages appear to be truncated or split across lines (I'm seeing both behaviors but I'm not sure if they are both the same problem). In syslog-ng we had to increase the maximum message size with the parameter "log_msg_size(65536);" within the options section. I'm trying to do the equivalent in rsyslog. I saw mention in the forums that this was possibly configurable via a #define but no mention of where I might find this. I realize this is probably outside the typical syslog spec but unfortunately it's a situation I have to deal with for rsyslog to be suitable in our environment. Unfortunately I cannot post the log messages publicly but I can probably provide sanitized samples if an individual was willing to help. Thank you, Aaron From rgerhards at hq.adiscon.com Fri Sep 11 18:48:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 18:48:39 +0200 Subject: [rsyslog] How to increase message size maximum References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, search for MAXLINE inside the code, change that, and recompile. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, September 11, 2009 6:40 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How to increase message size maximum > > Hello, > I replied to an older thread on the forums but wanted to bring this > up > here. I have an application logging to rsyslog (version 3.22.0) which > sends > very large messages. We are trying to migrate logging from syslog-ng to > rsyslog and I'm running into a problem where messages appear to be > truncated > or split across lines (I'm seeing both behaviors but I'm not sure if > they > are both the same problem). In syslog-ng we had to increase the maximum > message size with the parameter "log_msg_size(65536);" within the > options > section. I'm trying to do the equivalent in rsyslog. I saw mention in > the > forums that this was possibly configurable via a #define but no mention > of > where I might find this. > > I realize this is probably outside the typical syslog spec but > unfortunately > it's a situation I have to deal with for rsyslog to be suitable in our > environment. Unfortunately I cannot post the log messages publicly but > I can > probably provide sanitized samples if an individual was willing to > help. > > Thank you, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:59:38 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:59:38 -0600 Subject: [rsyslog] How to increase message size maximum In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> Message-ID: On Fri, Sep 11, 2009 at 10:48 AM, Rainer Gerhards wrote: > $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, > search for MAXLINE inside the code, change that, and recompile. > > Excellent - thank you, that is supported as of 3.21.4 per ChangeLog. From anichols at trumped.org Fri Sep 11 23:29:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 15:29:39 -0600 Subject: [rsyslog] Beginning of log messages being removed Message-ID: Hello, Log messages are now being delivered correctly after raising the messagesize value - now I seem to be having some issue with parsing. I am trying to log only the %msg% portion of the log message however the beginning of that message seems to be removed. Below are the two templates I used to log in both the rawmsg and then the value of %msg% so you can see what is being removed. I cannot post the entire %msg% value, but the two are the same with the exception of the beginning value. I just need to be able to log the message portion without the timestamp which is being delivered from the client. Thinking this may have been fixed with some of the parsing problems I have updated to the latest 4.x stable release - this problem has been observed on 3.22.1 & 4.4.1. I am currently running against 4.4.1. Two templates: $template ServerXML, "%timestamp% || %hostname% || %msg%\n" $template ServerXMLraw, "%rawmsg%\n" Using the first template the message looks like this: Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder of message removed for brevity, but it is intact in the logs) Using the second template the raw message looks like this: <142>Sep 11 21:15:01 localhost References: Message-ID: On Fri, 11 Sep 2009, Aaron Nichols wrote: > Hello, > Log messages are now being delivered correctly after raising the > messagesize value - now I seem to be having some issue with parsing. I am > trying to log only the %msg% portion of the log message however the > beginning of that message seems to be removed. Below are the two templates I > used to log in both the rawmsg and then the value of %msg% so you can see > what is being removed. I cannot post the entire %msg% value, but the two are > the same with the exception of the beginning value. I just need to be able > to log the message portion without the timestamp which is being delivered > from the client. > > Thinking this may have been fixed with some of the parsing problems I have > updated to the latest 4.x stable release - this problem has been observed on > 3.22.1 & 4.4.1. I am currently running against 4.4.1. > > Two templates: > $template ServerXML, "%timestamp% || %hostname% || %msg%\n" > $template ServerXMLraw, "%rawmsg%\n" > > Using the first template the message looks like this: > Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder > of message removed for brevity, but it is intact in the logs) > > Using the second template the raw message looks like this: > <142>Sep 11 21:15:01 localhost userId= > > I'm trying to understand why the value " from %msg%. because it's being put in %syslogtag% as the program name. David Lang From joshsystem at gmail.com Sat Sep 12 07:36:11 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:36:11 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/11 Rainer Gerhards > OK, that's not very precise, but I have also thought a bit about this. What > I > have proposed this morning should be possible. But you should be warned, it > requires a lot of reading and understanding the source code. A good place > to > start is the template input and output modules as well as some actual > output > modules. I think imdiag would be useful (because it is simple) and probably > also either omstdout (simple) and omoracle (complex, but utilizes the > vector > interface which may be the best choice for what you intend to acomplish). > > As a side-note, if this is paid work you may want to think about purchasing > some development help from Adiscon, which may dramatically reduce the time > you need to get started (just a thought, omoracle was crafted very well > without any such help - thanks again!). > Where can I find the purchased development help in details? Thanks > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Friday, September 11, 2009 5:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks Rainer. The basic purpose is statistics,which can accumulate > > some > > fields of msgs,but I think the customers have more weird requirements. > > > > 2009/9/11 Rainer Gerhards > > > > > Now that I got an idea of how this could be implemented with current > > > rsyslog technology, I would be interested in some more details of > > what > > > you intend to do with the processing module. What exactly will it do > > > with the message? I am asking because I would like to see a real use > > > case. Thinking about the scenario I have proposed in my last mail, I > > > think I see some pitfalls and I am not sure if they will cause any > > > trouble in real projects. > > > > > > So I would appreciate if you could provide more in-depth info. > > > > > > Thanks, > > > Rainer > > > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > > Thanks for David and Rainer's reply.I m sorry that I did not > > explain my > > > > question clearly.I m new to rsyslog and want to add a processing > > module > > > in > > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > > plugins(back-end).My processing module receives data from input > > plugins > > > and > > > > output the processed data and raw data both into output plugins.So > > how I > > > add > > > > it? > > > > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > > > -----Original Message----- > > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > david at lang.hm > > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > > To: rsyslog-users > > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > > > PS: i browse the git source code, but i can't understand why > > the > > > > > > > > > > > > > Experimental-lockfree > > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > > is > > > > > > > not adopted? > > > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > > that there are > > > > > > no bugs, and the fact that even without that there has been a > > > > > > LOT of room > > > > > > for improvement from the early 3.x timeframe to the current > > > > > > 5.x version. > > > > > > > > > > > > I expect that after the current round of improvements are > > > > > > settled that > > > > > > aspect of things will get reexamined. > > > > > > > > > > That branch is mostly there for historical reasons. I keep that > > branch > > > as a > > > > > think-tank, but it is is obsoleted. Also, in less polite words > > than > > > David > > > > > used, it simply doesn't work. Getting this code with multiple > > producers > > > and > > > > > consumers correct is far from being trivial and the literature I > > > browsed > > > > > indicates that it is probably not possible given the other > > predicates > > > the > > > > > code must obey to. Still, optimization is high up on the todo > > list. > > > > > > > > > > Rainer > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Sat Sep 12 07:39:36 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:39:36 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck is output module.it relies on storage disk. I currently test the splunk,but not strong enough.of course,the client delivers the message to server via ethernet. >if you are looking at the source look for imtemplate and omtemplate, >basicly he is suggesting creating a custom output module that rsyslog >thinks is delivering the messages somewhere, have it be given the log, do >it's processing, then acting like an input module and delivering the >result to rsyslog as if it was a new message that just arrived. . This approach as Rainer may be overhead :( ; >That would also be much faster than whatever RainerScript will have to offer, >because RainerScript relies on VM execution. As Rainer said that RainerScrpt is not easy to be extended,but I think it is the perfect approach. I can't find any documents about it:(; It's really hard to start it! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Our raw data is "high" volume that means to prcocess data about 100M/min. > > is this 100M log records, or 100MB of log data (if the latter, > approximatly how large are the recors, of how many log records/min) > > I'm currently processing ~300K messages averaging ~256 bytes/message for a > total of ~75MB of logs/min. > > in my testing v4 will support up to about 6x this volume before it runs > into problems (it can receive them faster, up to gig-E wire speed, the > limit is in the output, which is ~80K records a sec if doing trivial work > like writing them to disk or ~30K records/sec if doing more complex things > like forwarding them elsewhere) > > improvements in V5 include a batch mode that lets an output module process > up to N records at a time. I expect this to provide close to a Nx speedup > to the output capabilities (with single log per action much of the > overhead is in the queue locking, so multiple output workers doesn't help > much, with batches not only is much more getting done per pass, but you > have the possibility of each output thread taking long enough to get it's > work done that it's effective to run more of them without locking > contention being the bottleneck) > > this batch mode will be especially useful for database work as it will let > you insert multiple messages in the database in a single transaction. > > what transport are you using to deliver the logs to your server? > > > Yes, I want to improve the system performance as soon as possibe. > > what is the bottleneck you are running into today (what syslog system are > you using, etc)? > > > As you > > said,rsyslog has a concept that inserts my logic module into it ,but it > was > > not been implemented. Could you point out in detail? The rainerscript > seems > > not that strong,otherwise, it is a good idea for user interface. > > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. > > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters to > keep the other output modules from seeing the raw input that your module > is looking for. > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit-core > >>> hardware. The logs data is very high volume too. Could you give me any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right person > to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your own > >> logic in the middle of the processing. rsyslog has the concept, but it > has > >> not been implemented (fixing bugs and speeding it up has taken priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I think > that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages in > one > >> transaction that is being developed will be a huge improvement in terms > of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then feed > >> the results into another instance of rsyslog to forward them on, insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it can > >> be > >>>> used for a few things, but mostly just for filter 'does this log match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe data > to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from perfect > and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand why > the > >>>>>>>>>> > >>>>>>>>> Experimental-lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Sep 12 08:18:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 23:18:17 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Sat, 12 Sep 2009, Josh Zhao wrote: > It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck > is output module.it relies on storage disk. I currently test the splunk,but > not strong enough. I am also using splunk. at that volume I would expect to be able to handle everything with a single server. My old system has one box receiving logs from many sources, archiving them, and forwarding them on to several other systems for event correlation, reporting, etc (one of which is my old splunk box). it is comfortably handling about your volume (averaged over a few min, my peak seconds top 10K logs). things are spread across multiple systems less due to current load than in preperation for increasing the load (I am gearing up to handle ~10x my current load) I've done a fair bit of stress testing of the various components and applications. what sort of problems are you having? > of course,the client delivers the message to server via > ethernet. I was meaning are you using TCP syslog, UDP syslog, or something else? David Lang >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. > . > This approach as Rainer may be overhead :( ; > >> That would also be much faster than whatever RainerScript will have to > offer, >> because RainerScript relies on VM execution. > > As Rainer said that RainerScrpt is not easy to be extended,but I think it is > the perfect approach. I can't find any documents about it:(; It's really > hard to start it! > > > > > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Our raw data is "high" volume that means to prcocess data about 100M/min. >> >> is this 100M log records, or 100MB of log data (if the latter, >> approximatly how large are the recors, of how many log records/min) >> >> I'm currently processing ~300K messages averaging ~256 bytes/message for a >> total of ~75MB of logs/min. >> >> in my testing v4 will support up to about 6x this volume before it runs >> into problems (it can receive them faster, up to gig-E wire speed, the >> limit is in the output, which is ~80K records a sec if doing trivial work >> like writing them to disk or ~30K records/sec if doing more complex things >> like forwarding them elsewhere) >> >> improvements in V5 include a batch mode that lets an output module process >> up to N records at a time. I expect this to provide close to a Nx speedup >> to the output capabilities (with single log per action much of the >> overhead is in the queue locking, so multiple output workers doesn't help >> much, with batches not only is much more getting done per pass, but you >> have the possibility of each output thread taking long enough to get it's >> work done that it's effective to run more of them without locking >> contention being the bottleneck) >> >> this batch mode will be especially useful for database work as it will let >> you insert multiple messages in the database in a single transaction. >> >> what transport are you using to deliver the logs to your server? >> >>> Yes, I want to improve the system performance as soon as possibe. >> >> what is the bottleneck you are running into today (what syslog system are >> you using, etc)? >> >>> As you >>> said,rsyslog has a concept that inserts my logic module into it ,but it >> was >>> not been implemented. Could you point out in detail? The rainerscript >> seems >>> not that strong,otherwise, it is a good idea for user interface. >> >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. >> >> you will need to put some filters in rsyslog to keep your output module >> from seeing the logs that it creates, and either use discard or filters to >> keep the other output modules from seeing the raw input that your module >> is looking for. >> >> David Lang >> >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> Is rsyslog no way to reslove problem, What about syslog-ng? What I >> think >>>>> about,rsyslog's multi-thread archititure is better for my mulit-core >>>>> hardware. The logs data is very high volume too. Could you give me any >>>>> suggestion on this matter? >>>> >>>> my experiance with syslog-ng was not good, so I'm not the right person >> to >>>> talk about doing this sort of thing with it. >>>> >>>> but I am not aware of any syslog daemon that lets you insert your own >>>> logic in the middle of the processing. rsyslog has the concept, but it >> has >>>> not been implemented (fixing bugs and speeding it up has taken priority) >>>> >>>> what sort of volume do you consider 'high'? (it's amazing the range that >>>> this can span, so I've learned to ask rather than assume ;-) >>>> >>>> since you are needing to get your final data into a database, I think >> that >>>> you will find that rsyslog will (or will soon) suit your needs far >> better >>>> than alternate approaches. the ability to process multiple messages in >> one >>>> transaction that is being developed will be a huge improvement in terms >> of >>>> database interaction. >>>> >>>> I would look at what rainer suggested for now. >>>> >>>> have one copy of rsyslog that receives the messages, does whatever >>>> formatting/cleanup is needed on them, then passes the logs to one or >> more >>>> instances of your code to do additional processing, which can then feed >>>> the results into another instance of rsyslog to forward them on, insert >>>> them into a database, etc. >>>> >>>> when rainerscript gains the capability to alter the fields (instead of >>>> just testing them), then there will be a lot more that can be done >> inside >>>> rsyslog. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> >>>>> 2009/9/11 >>>>> >>>>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>>>> >>>>>>> You mean I have to rewrite the processing module in >> rainerscript.where >>>>>> can i >>>>>>> find the detailed documents related to the scripting engine? >>>>>> >>>>>> right now rainerscript is as much an idea as an implementation. it can >>>> be >>>>>> used for a few things, but mostly just for filter 'does this log match >>>> X' >>>>>> type of things. >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Thank you! >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>> >>>>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not >> explain >>>> my >>>>>>>>> question clearly.I m new to rsyslog and want to add a processing >>>> module >>>>>>>>> in >>>>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>>>> plugins(back-end).My processing module receives data from input >>>> plugins >>>>>>>>> and >>>>>>>>> output the processed data and raw data both into output plugins.So >>>> how >>>>>>>>> I add >>>>>>>>> it? >>>>>>>> >>>>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>>>> plugins >>>>>>>> will work together with the scripting engine. In other words: there >>>>>>>> currently >>>>>>>> is no in-proc method available. >>>>>>>> >>>>>>>> What you can do, however, is chain two rsyslog instances, pipe data >> to >>>>>> your >>>>>>>> plugin and send that data to the other instance. Far from perfect >> and >>>>>> easy >>>>>>>> to >>>>>>>> do, but maybe a workable work-around... >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2009/9/10 Rainer Gerhards >>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>>>> david at lang.hm >>>>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>>> >>>>>>>>>>>> PS: i browse the git source code, but i can't understand why >> the >>>>>>>>>>>> >>>>>>>>>>> Experimental-lockfree>>>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>>>> is >>>>>>>>>>>> not adopted? >>>>>>>>>>> >>>>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>>>> that there are >>>>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>>>> LOT of room >>>>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>>>> 5.x version. >>>>>>>>>>> >>>>>>>>>>> I expect that after the current round of improvements are >>>>>>>>>>> settled that >>>>>>>>>>> aspect of things will get reexamined. >>>>>>>>>> >>>>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>>>> branch as a >>>>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words >> than >>>>>>>>> David >>>>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>>>> producers and >>>>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>>>> browsed >>>>>>>>>> indicates that it is probably not possible given the other >>>> predicates >>>>>>>>> the >>>>>>>>>> code must obey to. Still, optimization is high up on the todo >> list. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Sep 14 10:14:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Sep 2009 10:14:39 +0200 Subject: [rsyslog] DNS Cache Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE9D@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that in parallel to my bughunt (which involves a lot of waiting for lab results), I will now begin to implement a real DNS cache. That will be a v5-exclusive feature (too much trouble to do it in v4 and v5, code base has changed too much). Together with the case, I will probably also implement a feature to override reverse DNS resolution via a file - simply by loading non-expiring entries from that file. I just thought I share this plan, if someone has feature requests in that regard. It would be good to know them, as now is a good time to integrate them into the design. Tech side-note: I'll be using AVL trees for the cache, as I don't outrule many entries and this hopefully speeds up cache searches for larger caches. Once the avl tree class is there, I can probably speed up a few other things that currently rely on simple linked lists). I will probably do two or even three releases until the full functionality is there. I also plan to do a "pre-cache" v5-devel today, so that all new features (including imudp epoll) are rolled out and can be tested. Rainer From tbergfeld at hq.adiscon.com Mon Sep 14 17:05:35 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 14 Sep 2009 17:05:35 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.0. This release starts a new v5-development branch. This release offers a lot of new features like the use of epoll, when possible, in imudp, which provides greater performance and is a pilot to more such enhancements. Further more there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml Changelog: http://www.rsyslog.com/Article402.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld From david at lang.hm Tue Sep 15 02:51:19 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 14 Sep 2009 17:51:19 -0700 (PDT) Subject: [rsyslog] rsyslog 5.3.0 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: I do not see this tagged in git. David Lang On Mon, 14 Sep 2009, Tom Bergfeld wrote: > Date: Mon, 14 Sep 2009 17:05:35 +0200 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.0 (devel) released > > Hi all, > > We have just released rsyslog 5.3.0. This release starts a new v5-development > branch. > This release offers a lot of new features like the use of epoll, when > possible, in imudp, which provides greater performance and is a pilot to more > such enhancements. Further more there are also some bug fixes. See Changelog > for more details. This is a recommended update for all users of the devel > branch. > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml > > Changelog: > > http://www.rsyslog.com/Article402.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mikel at irontec.com Tue Sep 15 10:52:37 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 10:52:37 +0200 Subject: [rsyslog] server frozen when remote logging Message-ID: <4AAF55D5.20807@irontec.com> Hi!! I have 80 servers logging to a centralized rsyslog, and I have experimented the kaos!! Accidentaly the central server shutdowns, and one hour later, all the 80 servers frezze. Can not access ssh, ping... I use Debian in central server, and suse in nodes. Thanks! -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 10:56:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:56:28 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> This sounds like you are overdoing "reliable delivery". But I need configs and version information to tell you what may be the case. If it is an older v3 version, this may also be a bug in rsyslog. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 10:53 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] server frozen when remote logging > > Hi!! > > I have 80 servers logging to a centralized rsyslog, and I have > experimented the kaos!! > > Accidentaly the central server shutdowns, and one hour later, all the > 80 > servers frezze. > > Can not access ssh, ping... > > I use Debian in central server, and suse in nodes. > > Thanks! > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 10:58:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:58:44 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECC@GRFEXC.intern.adiscon.com> Thanks - I had forgotten to push the tags (but thankfully this time not the tagging itself ;)) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 15, 2009 2:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.0 (devel) released > > I do not see this tagged in git. From mikel at irontec.com Tue Sep 15 11:56:26 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:56:26 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF64CA.8020002@irontec.com> Ok Rainer In the clients: OS= opensuse 10.0 rsyslog version: 3.19.7 In the server OS=Debian 4.0 rsyslog version: 3.18.2 I attach the configuration files of the clients and the servers. The remote server is 192.1.4.215. Thanks Rainer Gerhards wrote: > This sounds like you are overdoing "reliable delivery". But I need configs > and version information to tell you what may be the case. If it is an older > v3 version, this may also be a bug in rsyslog. > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 10:53 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] server frozen when remote logging >> >> Hi!! >> >> I have 80 servers logging to a centralized rsyslog, and I have >> experimented the kaos!! >> >> Accidentaly the central server shutdowns, and one hour later, all the >> 80 >> servers frezze. >> >> Can not access ssh, ping... >> >> I use Debian in central server, and suse in nodes. >> >> Thanks! >> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-client.conf URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-server.conf URL: From mikel at irontec.com Tue Sep 15 11:58:52 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:58:52 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9E6A72.8080202@irontec.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <4AAF655C.6050601@irontec.com> Hi!! We are very interested, how much do you estimate? Thanks Mikel Jimenez wrote: > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of > January 27 ... > > Thanks > > Mikel Jimenez wrote: >> hi >> >> Some news about this? >> >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >> Maybe with a bounty? >> >> thanks >> > > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:01:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:01:02 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:59 AM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Hi!! > > We are very interested, how much do you estimate? Thanks. I've just asked the right people, please expect a reply either today or (depending on discussion) tomorrow, must probably via private mail. Rainer > Thanks > > Mikel Jimenez wrote: > > Ok, I will comunicate you if we decide. > > > > Is the development of phplogcon frezzed? the last version is of > > January 27 ... > > > > Thanks > > > > Mikel Jimenez wrote: > >> hi > >> > >> Some news about this? > >> > >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > >> Maybe with a bounty? > >> > >> thanks > >> > > > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:05:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:05:40 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Ok, there are errors in the config files. I've stopped looking at them when I saw EST=... ... @@$EST This does not work in rsyslog (yet). Please make sure that your configs are OK. With the versions you have, you either need to start rsyslogd interactively in debug mode OR simply look at the syslogd logs (those with syslog facility). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:56 AM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Ok Rainer > > In the clients: > > OS= opensuse 10.0 > rsyslog version: 3.19.7 > > In the server > OS=Debian 4.0 > rsyslog version: 3.18.2 > > I attach the configuration files of the clients and the servers. > > The remote server is 192.1.4.215. > > Thanks > > Rainer Gerhards wrote: > > This sounds like you are overdoing "reliable delivery". But I need > configs > > and version information to tell you what may be the case. If it is an > older > > v3 version, this may also be a bug in rsyslog. > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 10:53 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] server frozen when remote logging > >> > >> Hi!! > >> > >> I have 80 servers logging to a centralized rsyslog, and I have > >> experimented the kaos!! > >> > >> Accidentaly the central server shutdowns, and one hour later, all > the > >> 80 > >> servers frezze. > >> > >> Can not access ssh, ping... > >> > >> I use Debian in central server, and suse in nodes. > >> > >> Thanks! > >> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > From mikel at irontec.com Tue Sep 15 12:10:16 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:10:16 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> Message-ID: <4AAF6808.2080402@irontec.com> Yeah!!! :) Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:59 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Hi!! >> >> We are very interested, how much do you estimate? >> > > Thanks. I've just asked the right people, please expect a reply either today > or (depending on discussion) tomorrow, must probably via private mail. > > Rainer > > > > >> Thanks >> >> Mikel Jimenez wrote: >> >>> Ok, I will comunicate you if we decide. >>> >>> Is the development of phplogcon frezzed? the last version is of >>> January 27 ... >>> >>> Thanks >>> >>> Mikel Jimenez wrote: >>> >>>> hi >>>> >>>> Some news about this? >>>> >>>> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >>>> Maybe with a bounty? >>>> >>>> thanks >>>> >>>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From mikel at irontec.com Tue Sep 15 12:14:20 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:14:20 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Message-ID: <4AAF68FC.1040205@irontec.com> But rsyslog starts... If I use UDP instead TCP? Rainer Gerhards wrote: > Ok, there are errors in the config files. I've stopped looking at them when I > saw > > EST=... > > ... @@$EST > > This does not work in rsyslog (yet). Please make sure that your configs are > OK. With the versions you have, you either need to start rsyslogd > interactively in debug mode OR simply look at the syslogd logs (those with > syslog facility). > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:56 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> Ok Rainer >> >> In the clients: >> >> OS= opensuse 10.0 >> rsyslog version: 3.19.7 >> >> In the server >> OS=Debian 4.0 >> rsyslog version: 3.18.2 >> >> I attach the configuration files of the clients and the servers. >> >> The remote server is 192.1.4.215. >> >> Thanks >> >> Rainer Gerhards wrote: >> >>> This sounds like you are overdoing "reliable delivery". But I need >>> >> configs >> >>> and version information to tell you what may be the case. If it is an >>> >> older >> >>> v3 version, this may also be a bug in rsyslog. >>> >>> HTH >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] server frozen when remote logging >>>> >>>> Hi!! >>>> >>>> I have 80 servers logging to a centralized rsyslog, and I have >>>> experimented the kaos!! >>>> >>>> Accidentaly the central server shutdowns, and one hour later, all >>>> >> the >> >>>> 80 >>>> servers frezze. >>>> >>>> Can not access ssh, ping... >>>> >>>> I use Debian in central server, and suse in nodes. >>>> >>>> Thanks! >>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:18:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:18:30 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > But rsyslog starts... > > If I use UDP instead TCP? sure, because the messages are thrown away, at least I think. The point is: it doesn't make sense to hunt for a problem as long as we know that the config is incorrect. Better get the config clean first, then see if the problem even persists and then look at it. Bluntly and not meant to be embarrassing: I've set aside some time to do this kind of support, but if you need more "full service" help, it would probably be a good idea to purchase one of the support packages. They exists so that we can look at issues in depth. This is often an excellent values, as it may safe you hours and hours of work. And, really, I can't develop all this and provide this kind of full-service support ;) Rainer > > Rainer Gerhards wrote: > > Ok, there are errors in the config files. I've stopped looking at > them when I > > saw > > > > EST=... > > > > ... @@$EST > > > > This does not work in rsyslog (yet). Please make sure that your > configs are > > OK. With the versions you have, you either need to start rsyslogd > > interactively in debug mode OR simply look at the syslogd logs (those > with > > syslog facility). > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 11:56 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> Ok Rainer > >> > >> In the clients: > >> > >> OS= opensuse 10.0 > >> rsyslog version: 3.19.7 > >> > >> In the server > >> OS=Debian 4.0 > >> rsyslog version: 3.18.2 > >> > >> I attach the configuration files of the clients and the servers. > >> > >> The remote server is 192.1.4.215. > >> > >> Thanks > >> > >> Rainer Gerhards wrote: > >> > >>> This sounds like you are overdoing "reliable delivery". But I need > >>> > >> configs > >> > >>> and version information to tell you what may be the case. If it is > an > >>> > >> older > >> > >>> v3 version, this may also be a bug in rsyslog. > >>> > >>> HTH > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: [rsyslog] server frozen when remote logging > >>>> > >>>> Hi!! > >>>> > >>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>> experimented the kaos!! > >>>> > >>>> Accidentaly the central server shutdowns, and one hour later, all > >>>> > >> the > >> > >>>> 80 > >>>> servers frezze. > >>>> > >>>> Can not access ssh, ping... > >>>> > >>>> I use Debian in central server, and suse in nodes. > >>>> > >>>> Thanks! > >>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 12:47:31 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:47:31 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> Message-ID: <4AAF70C3.1000800@irontec.com> Hi I will delete this line of the config. I will make probes. About the comercial support, I think that this issue is "basic" for the proper working of a production and seriour enviroment of rsyslog. In the future, if we want an especialezed support we call you for support, sure!! :) So any solution for this? UDP? Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 12:14 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> But rsyslog starts... >> >> If I use UDP instead TCP? >> > > sure, because the messages are thrown away, at least I think. The point is: > it doesn't make sense to hunt for a problem as long as we know that the > config is incorrect. Better get the config clean first, then see if the > problem even persists and then look at it. > > Bluntly and not meant to be embarrassing: I've set aside some time to do this > kind of support, but if you need more "full service" help, it would probably > be a good idea to purchase one of the support packages. They exists so that > we can look at issues in depth. This is often an excellent values, as it may > safe you hours and hours of work. And, really, I can't develop all this and > provide this kind of full-service support ;) > > Rainer > > >> Rainer Gerhards wrote: >> >>> Ok, there are errors in the config files. I've stopped looking at >>> >> them when I >> >>> saw >>> >>> EST=... >>> >>> ... @@$EST >>> >>> This does not work in rsyslog (yet). Please make sure that your >>> >> configs are >> >>> OK. With the versions you have, you either need to start rsyslogd >>> interactively in debug mode OR simply look at the syslogd logs (those >>> >> with >> >>> syslog facility). >>> >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] server frozen when remote logging >>>> >>>> Ok Rainer >>>> >>>> In the clients: >>>> >>>> OS= opensuse 10.0 >>>> rsyslog version: 3.19.7 >>>> >>>> In the server >>>> OS=Debian 4.0 >>>> rsyslog version: 3.18.2 >>>> >>>> I attach the configuration files of the clients and the servers. >>>> >>>> The remote server is 192.1.4.215. >>>> >>>> Thanks >>>> >>>> Rainer Gerhards wrote: >>>> >>>> >>>>> This sounds like you are overdoing "reliable delivery". But I need >>>>> >>>>> >>>> configs >>>> >>>> >>>>> and version information to tell you what may be the case. If it is >>>>> >> an >> >>>> older >>>> >>>> >>>>> v3 version, this may also be a bug in rsyslog. >>>>> >>>>> HTH >>>>> Rainer >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>> To: rsyslog at lists.adiscon.com >>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>> >>>>>> Hi!! >>>>>> >>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>> experimented the kaos!! >>>>>> >>>>>> Accidentaly the central server shutdowns, and one hour later, all >>>>>> >>>>>> >>>> the >>>> >>>> >>>>>> 80 >>>>>> servers frezze. >>>>>> >>>>>> Can not access ssh, ping... >>>>>> >>>>>> I use Debian in central server, and suse in nodes. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> Mikel Jimenez Fernandez >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>> >>>>>> >>>> http://www.irontec.com >>>> >>>> >>>>>> +34 94.404.81.82 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:52:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:52:27 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> <4AAF70C3.1000800@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Hi > > I will delete this line of the config. > I will make probes. > About the comercial support, I think that this issue is "basic" for > the > proper working of a production and seriour enviroment of rsyslog. Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in bug reports is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. > > In the future, if we want an especialezed support we call you for > support, sure!! :) > > > So any solution for this? > UDP? Anyhow, does that mean your config is now error-free and the problem still persists? Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 12:14 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> But rsyslog starts... > >> > >> If I use UDP instead TCP? > >> > > > > sure, because the messages are thrown away, at least I think. The > point is: > > it doesn't make sense to hunt for a problem as long as we know that > the > > config is incorrect. Better get the config clean first, then see if > the > > problem even persists and then look at it. > > > > Bluntly and not meant to be embarrassing: I've set aside some time to > do this > > kind of support, but if you need more "full service" help, it would > probably > > be a good idea to purchase one of the support packages. They exists > so that > > we can look at issues in depth. This is often an excellent values, as > it may > > safe you hours and hours of work. And, really, I can't develop all > this and > > provide this kind of full-service support ;) > > > > Rainer > > > > > >> Rainer Gerhards wrote: > >> > >>> Ok, there are errors in the config files. I've stopped looking at > >>> > >> them when I > >> > >>> saw > >>> > >>> EST=... > >>> > >>> ... @@$EST > >>> > >>> This does not work in rsyslog (yet). Please make sure that your > >>> > >> configs are > >> > >>> OK. With the versions you have, you either need to start rsyslogd > >>> interactively in debug mode OR simply look at the syslogd logs > (those > >>> > >> with > >> > >>> syslog facility). > >>> > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] server frozen when remote logging > >>>> > >>>> Ok Rainer > >>>> > >>>> In the clients: > >>>> > >>>> OS= opensuse 10.0 > >>>> rsyslog version: 3.19.7 > >>>> > >>>> In the server > >>>> OS=Debian 4.0 > >>>> rsyslog version: 3.18.2 > >>>> > >>>> I attach the configuration files of the clients and the servers. > >>>> > >>>> The remote server is 192.1.4.215. > >>>> > >>>> Thanks > >>>> > >>>> Rainer Gerhards wrote: > >>>> > >>>> > >>>>> This sounds like you are overdoing "reliable delivery". But I > need > >>>>> > >>>>> > >>>> configs > >>>> > >>>> > >>>>> and version information to tell you what may be the case. If it > is > >>>>> > >> an > >> > >>>> older > >>>> > >>>> > >>>>> v3 version, this may also be a bug in rsyslog. > >>>>> > >>>>> HTH > >>>>> Rainer > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>>>> To: rsyslog at lists.adiscon.com > >>>>>> Subject: [rsyslog] server frozen when remote logging > >>>>>> > >>>>>> Hi!! > >>>>>> > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>>>> experimented the kaos!! > >>>>>> > >>>>>> Accidentaly the central server shutdowns, and one hour later, > all > >>>>>> > >>>>>> > >>>> the > >>>> > >>>> > >>>>>> 80 > >>>>>> servers frezze. > >>>>>> > >>>>>> Can not access ssh, ping... > >>>>>> > >>>>>> I use Debian in central server, and suse in nodes. > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> -- > >>>>>> Mikel Jimenez Fernandez > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>>>> > >>>>>> > >>>> http://www.irontec.com > >>>> > >>>> > >>>>>> +34 94.404.81.82 > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>>> > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>>> > >>>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:54:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:54:24 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Used the wrong words ;) Of course, this should read: Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in support is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. And: why is my interest limited? Support to get someone else going contributes almost nothing back to the project... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > Sent: Tuesday, September 15, 2009 12:48 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > Hi > > > > I will delete this line of the config. > > I will make probes. > > About the comercial support, I think that this issue is "basic" for > > the > > proper working of a production and seriour enviroment of rsyslog. > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in bug reports is *extremely limited*. Still, there is the > rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > > > > In the future, if we want an especialezed support we call you for > > support, sure!! :) > > > > > > So any solution for this? > > UDP? > > Anyhow, does that mean your config is now error-free and the problem > still > persists? > > Rainer > > > > Rainer Gerhards wrote: > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] server frozen when remote logging > > >> > > >> But rsyslog starts... > > >> > > >> If I use UDP instead TCP? > > >> > > > > > > sure, because the messages are thrown away, at least I think. The > > point is: > > > it doesn't make sense to hunt for a problem as long as we know that > > the > > > config is incorrect. Better get the config clean first, then see if > > the > > > problem even persists and then look at it. > > > > > > Bluntly and not meant to be embarrassing: I've set aside some time > to > > do this > > > kind of support, but if you need more "full service" help, it would > > probably > > > be a good idea to purchase one of the support packages. They exists > > so that > > > we can look at issues in depth. This is often an excellent values, > as > > it may > > > safe you hours and hours of work. And, really, I can't develop all > > this and > > > provide this kind of full-service support ;) > > > > > > Rainer > > > > > > > > >> Rainer Gerhards wrote: > > >> > > >>> Ok, there are errors in the config files. I've stopped looking at > > >>> > > >> them when I > > >> > > >>> saw > > >>> > > >>> EST=... > > >>> > > >>> ... @@$EST > > >>> > > >>> This does not work in rsyslog (yet). Please make sure that your > > >>> > > >> configs are > > >> > > >>> OK. With the versions you have, you either need to start rsyslogd > > >>> interactively in debug mode OR simply look at the syslogd logs > > (those > > >>> > > >> with > > >> > > >>> syslog facility). > > >>> > > >>> Rainer > > >>> > > >>> > > >>> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > >>>> To: rsyslog-users > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > >>>> > > >>>> Ok Rainer > > >>>> > > >>>> In the clients: > > >>>> > > >>>> OS= opensuse 10.0 > > >>>> rsyslog version: 3.19.7 > > >>>> > > >>>> In the server > > >>>> OS=Debian 4.0 > > >>>> rsyslog version: 3.18.2 > > >>>> > > >>>> I attach the configuration files of the clients and the servers. > > >>>> > > >>>> The remote server is 192.1.4.215. > > >>>> > > >>>> Thanks > > >>>> > > >>>> Rainer Gerhards wrote: > > >>>> > > >>>> > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > need > > >>>>> > > >>>>> > > >>>> configs > > >>>> > > >>>> > > >>>>> and version information to tell you what may be the case. If it > > is > > >>>>> > > >> an > > >> > > >>>> older > > >>>> > > >>>> > > >>>>> v3 version, this may also be a bug in rsyslog. > > >>>>> > > >>>>> HTH > > >>>>> Rainer > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > >>>>>> To: rsyslog at lists.adiscon.com > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > >>>>>> > > >>>>>> Hi!! > > >>>>>> > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > > >>>>>> experimented the kaos!! > > >>>>>> > > >>>>>> Accidentaly the central server shutdowns, and one hour later, > > all > > >>>>>> > > >>>>>> > > >>>> the > > >>>> > > >>>> > > >>>>>> 80 > > >>>>>> servers frezze. > > >>>>>> > > >>>>>> Can not access ssh, ping... > > >>>>>> > > >>>>>> I use Debian in central server, and suse in nodes. > > >>>>>> > > >>>>>> Thanks! > > >>>>>> > > >>>>>> -- > > >>>>>> Mikel Jimenez Fernandez > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>>>> > > >>>>>> > > >>>> http://www.irontec.com > > >>>> > > >>>> > > >>>>>> +34 94.404.81.82 > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>> _______________________________________________ > > >>>>> rsyslog mailing list > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>> http://www.rsyslog.com > > >>>>> > > >>>>> > > >>>>> > > >>>> -- > > >>>> Mikel Jimenez Fernandez > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>> > > >> http://www.irontec.com > > >> > > >>>> +34 94.404.81.82 > > >>>> > > >>>> > > >>>> > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >>> > > >> -- > > >> Mikel Jimenez Fernandez > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > http://www.irontec.com > > >> +34 94.404.81.82 > > >> > > >> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > > > > > -- > > Mikel Jimenez Fernandez > > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > > +34 94.404.81.82 > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:55:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:55:29 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDB@GRFEXC.intern.adiscon.com> and a bit more on the philosophy, I knew I wrote it down ;) http://www.rsyslog.com/doc-free_support.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 15, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > Sent: Tuesday, September 15, 2009 12:48 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > > Hi > > > > > > I will delete this line of the config. > > > I will make probes. > > > About the comercial support, I think that this issue is "basic" > for > > > the > > > proper working of a production and seriour enviroment of rsyslog. > > > > Sure, but let me phrase it that way: My interest is finding bugs, > > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in bug reports is *extremely limited*. Still, there is the > > rest of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > > save > > time while doing so. > > > > > > > > In the future, if we want an especialezed support we call you for > > > support, sure!! :) > > > > > > > > > So any solution for this? > > > UDP? > > > > Anyhow, does that mean your config is now error-free and the problem > > still > > persists? > > > > Rainer > > > > > > Rainer Gerhards wrote: > > > >> -----Original Message----- > > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > > >> To: rsyslog-users > > > >> Subject: Re: [rsyslog] server frozen when remote logging > > > >> > > > >> But rsyslog starts... > > > >> > > > >> If I use UDP instead TCP? > > > >> > > > > > > > > sure, because the messages are thrown away, at least I think. The > > > point is: > > > > it doesn't make sense to hunt for a problem as long as we know > that > > > the > > > > config is incorrect. Better get the config clean first, then see > if > > > the > > > > problem even persists and then look at it. > > > > > > > > Bluntly and not meant to be embarrassing: I've set aside some > time > > to > > > do this > > > > kind of support, but if you need more "full service" help, it > would > > > probably > > > > be a good idea to purchase one of the support packages. They > exists > > > so that > > > > we can look at issues in depth. This is often an excellent > values, > > as > > > it may > > > > safe you hours and hours of work. And, really, I can't develop > all > > > this and > > > > provide this kind of full-service support ;) > > > > > > > > Rainer > > > > > > > > > > > >> Rainer Gerhards wrote: > > > >> > > > >>> Ok, there are errors in the config files. I've stopped looking > at > > > >>> > > > >> them when I > > > >> > > > >>> saw > > > >>> > > > >>> EST=... > > > >>> > > > >>> ... @@$EST > > > >>> > > > >>> This does not work in rsyslog (yet). Please make sure that your > > > >>> > > > >> configs are > > > >> > > > >>> OK. With the versions you have, you either need to start > rsyslogd > > > >>> interactively in debug mode OR simply look at the syslogd logs > > > (those > > > >>> > > > >> with > > > >> > > > >>> syslog facility). > > > >>> > > > >>> Rainer > > > >>> > > > >>> > > > >>> > > > >>>> -----Original Message----- > > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > > >>>> To: rsyslog-users > > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > > >>>> > > > >>>> Ok Rainer > > > >>>> > > > >>>> In the clients: > > > >>>> > > > >>>> OS= opensuse 10.0 > > > >>>> rsyslog version: 3.19.7 > > > >>>> > > > >>>> In the server > > > >>>> OS=Debian 4.0 > > > >>>> rsyslog version: 3.18.2 > > > >>>> > > > >>>> I attach the configuration files of the clients and the > servers. > > > >>>> > > > >>>> The remote server is 192.1.4.215. > > > >>>> > > > >>>> Thanks > > > >>>> > > > >>>> Rainer Gerhards wrote: > > > >>>> > > > >>>> > > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > > need > > > >>>>> > > > >>>>> > > > >>>> configs > > > >>>> > > > >>>> > > > >>>>> and version information to tell you what may be the case. If > it > > > is > > > >>>>> > > > >> an > > > >> > > > >>>> older > > > >>>> > > > >>>> > > > >>>>> v3 version, this may also be a bug in rsyslog. > > > >>>>> > > > >>>>> HTH > > > >>>>> Rainer > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > > >>>>>> To: rsyslog at lists.adiscon.com > > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > > >>>>>> > > > >>>>>> Hi!! > > > >>>>>> > > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I > have > > > >>>>>> experimented the kaos!! > > > >>>>>> > > > >>>>>> Accidentaly the central server shutdowns, and one hour > later, > > > all > > > >>>>>> > > > >>>>>> > > > >>>> the > > > >>>> > > > >>>> > > > >>>>>> 80 > > > >>>>>> servers frezze. > > > >>>>>> > > > >>>>>> Can not access ssh, ping... > > > >>>>>> > > > >>>>>> I use Debian in central server, and suse in nodes. > > > >>>>>> > > > >>>>>> Thanks! > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Mikel Jimenez Fernandez > > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>>>> > > > >>>>>> > > > >>>> http://www.irontec.com > > > >>>> > > > >>>> > > > >>>>>> +34 94.404.81.82 > > > >>>>>> > > > >>>>>> > > > >>>>>> _______________________________________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>> http://www.rsyslog.com > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> _______________________________________________ > > > >>>>> rsyslog mailing list > > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>> http://www.rsyslog.com > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>> -- > > > >>>> Mikel Jimenez Fernandez > > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>> > > > >> http://www.irontec.com > > > >> > > > >>>> +34 94.404.81.82 > > > >>>> > > > >>>> > > > >>>> > > > >>> _______________________________________________ > > > >>> rsyslog mailing list > > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>> http://www.rsyslog.com > > > >>> > > > >>> > > > >> -- > > > >> Mikel Jimenez Fernandez > > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > http://www.irontec.com > > > >> +34 94.404.81.82 > > > >> > > > >> > > > >> _______________________________________________ > > > >> rsyslog mailing list > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> http://www.rsyslog.com > > > >> > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > > > > > > > -- > > > Mikel Jimenez Fernandez > > > Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > > > +34 94.404.81.82 > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 13:09:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:09:02 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF75CE.7090605@irontec.com> Rainer Gerhards wrote: > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > Im going to make probes with deleting the config line EST=... When we have coclusion I will tell you, and I will back my concluison to this magic project. :) Thanks > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 15, 2009 12:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>> Sent: Tuesday, September 15, 2009 12:48 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] server frozen when remote logging >>> >>> Hi >>> >>> I will delete this line of the config. >>> I will make probes. >>> About the comercial support, I think that this issue is "basic" for >>> the >>> proper working of a production and seriour enviroment of rsyslog. >>> >> Sure, but let me phrase it that way: My interest is finding bugs, >> support >> questions often lead to that. If there is no bug involved, my personal >> interest in bug reports is *extremely limited*. Still, there is the >> rest of >> the community, and they often provide advice. So Adiscon created the >> commercial support for corporations that want to have a solution and >> save >> time while doing so. >> >> >>> In the future, if we want an especialezed support we call you for >>> support, sure!! :) >>> >>> >>> So any solution for this? >>> UDP? >>> >> Anyhow, does that mean your config is now error-free and the problem >> still >> persists? >> >> Rainer >> >>> Rainer Gerhards wrote: >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>> Sent: Tuesday, September 15, 2009 12:14 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>> >>>>> But rsyslog starts... >>>>> >>>>> If I use UDP instead TCP? >>>>> >>>>> >>>> sure, because the messages are thrown away, at least I think. The >>>> >>> point is: >>> >>>> it doesn't make sense to hunt for a problem as long as we know that >>>> >>> the >>> >>>> config is incorrect. Better get the config clean first, then see if >>>> >>> the >>> >>>> problem even persists and then look at it. >>>> >>>> Bluntly and not meant to be embarrassing: I've set aside some time >>>> >> to >> >>> do this >>> >>>> kind of support, but if you need more "full service" help, it would >>>> >>> probably >>> >>>> be a good idea to purchase one of the support packages. They exists >>>> >>> so that >>> >>>> we can look at issues in depth. This is often an excellent values, >>>> >> as >> >>> it may >>> >>>> safe you hours and hours of work. And, really, I can't develop all >>>> >>> this and >>> >>>> provide this kind of full-service support ;) >>>> >>>> Rainer >>>> >>>> >>>> >>>>> Rainer Gerhards wrote: >>>>> >>>>> >>>>>> Ok, there are errors in the config files. I've stopped looking at >>>>>> >>>>>> >>>>> them when I >>>>> >>>>> >>>>>> saw >>>>>> >>>>>> EST=... >>>>>> >>>>>> ... @@$EST >>>>>> >>>>>> This does not work in rsyslog (yet). Please make sure that your >>>>>> >>>>>> >>>>> configs are >>>>> >>>>> >>>>>> OK. With the versions you have, you either need to start rsyslogd >>>>>> interactively in debug mode OR simply look at the syslogd logs >>>>>> >>> (those >>> >>>>> with >>>>> >>>>> >>>>>> syslog facility). >>>>>> >>>>>> Rainer >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>>>> >>>>>>> Ok Rainer >>>>>>> >>>>>>> In the clients: >>>>>>> >>>>>>> OS= opensuse 10.0 >>>>>>> rsyslog version: 3.19.7 >>>>>>> >>>>>>> In the server >>>>>>> OS=Debian 4.0 >>>>>>> rsyslog version: 3.18.2 >>>>>>> >>>>>>> I attach the configuration files of the clients and the servers. >>>>>>> >>>>>>> The remote server is 192.1.4.215. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Rainer Gerhards wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> This sounds like you are overdoing "reliable delivery". But I >>>>>>>> >>> need >>> >>>>>>>> >>>>>>> configs >>>>>>> >>>>>>> >>>>>>> >>>>>>>> and version information to tell you what may be the case. If it >>>>>>>> >>> is >>> >>>>> an >>>>> >>>>> >>>>>>> older >>>>>>> >>>>>>> >>>>>>> >>>>>>>> v3 version, this may also be a bug in rsyslog. >>>>>>>> >>>>>>>> HTH >>>>>>>> Rainer >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>>>>> To: rsyslog at lists.adiscon.com >>>>>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>>>>> >>>>>>>>> Hi!! >>>>>>>>> >>>>>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>>>>> experimented the kaos!! >>>>>>>>> >>>>>>>>> Accidentaly the central server shutdowns, and one hour later, >>>>>>>>> >>> all >>> >>>>>>>>> >>>>>>> the >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> 80 >>>>>>>>> servers frezze. >>>>>>>>> >>>>>>>>> Can not access ssh, ping... >>>>>>>>> >>>>>>>>> I use Debian in central server, and suse in nodes. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Mikel Jimenez Fernandez >>>>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> http://www.irontec.com >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> +34 94.404.81.82 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Mikel Jimenez Fernandez >>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>> >>>>>>> >>>>> http://www.irontec.com >>>>> >>>>> >>>>>>> +34 94.404.81.82 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Mikel Jimenez Fernandez >>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>> >>> http://www.irontec.com >>> >>>>> +34 94.404.81.82 >>>>> >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> -- >>> Mikel Jimenez Fernandez >>> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >>> +34 94.404.81.82 >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 13:44:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 13:44:06 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> > Rainer Gerhards wrote: > > Used the wrong words ;) Of course, this should read: > > > > Sure, but let me phrase it that way: My interest is finding bugs, > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in support is *extremely limited*. Still, there is the rest > of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > save > > time while doing so. > > > > And: why is my interest limited? Support to get someone else going > > contributes almost nothing back to the project... > > > Im going to make probes with deleting the config line EST=... > > When we have coclusion I will tell you, and I will back my concluison > to > this magic project. :) I have taken another look at the log files in the meantime, assuming that $EST were not present ;) However, I do not see anything obviously wrong. But I think I remember there was a condition that caused messages to be processed to slowly. Probably the best idea is to see if the issue persists with the current v3-stable release. If it does, we should go to v4 and if it still persists we need to obtain debug logs. But I think chances are extremely high that the current v3-stable will solve it. however, those forwarding rules to $EST can never have worked, and may actually be overruning the retry mechanism after a while... Rainer From mikel at irontec.com Tue Sep 15 13:46:27 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:46:27 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> Message-ID: <4AAF7E93.1080309@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: >> Rainer Gerhards wrote: >> >>> Used the wrong words ;) Of course, this should read: >>> >>> Sure, but let me phrase it that way: My interest is finding bugs, >>> >> support >> >>> questions often lead to that. If there is no bug involved, my >>> >> personal >> >>> interest in support is *extremely limited*. Still, there is the rest >>> >> of >> >>> the community, and they often provide advice. So Adiscon created the >>> commercial support for corporations that want to have a solution and >>> >> save >> >>> time while doing so. >>> >>> And: why is my interest limited? Support to get someone else going >>> contributes almost nothing back to the project... >>> >>> >> Im going to make probes with deleting the config line EST=... >> >> When we have coclusion I will tell you, and I will back my concluison >> to >> this magic project. :) >> > > I have taken another look at the log files in the meantime, assuming that > $EST were not present ;) However, I do not see anything obviously wrong. But > I think I remember there was a condition that caused messages to be processed > to slowly. Probably the best idea is to see if the issue persists with the > current v3-stable release. If it does, we should go to v4 and if it still > persists we need to obtain debug logs. But I think chances are extremely high > that the current v3-stable will solve it. > > however, those forwarding rules to $EST can never have worked, and may > actually be overruning the retry mechanism after a while... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From tbergfeld at hq.adiscon.com Mon Sep 21 08:12:22 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 21 Sep 2009 08:12:22 +0200 Subject: [rsyslog] rsyslog 4.5.3 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FF73@GRFEXC.intern.adiscon.com> Rsyslog 4.5.3, a member of the v4- beta branch, has been released. It is a bug-fixing release. Most importantly, a bug that repeated messages were incorrectly processed by what it could lead to loss of the repeated message content, was fixed. As a side- effect, it could probably also be possible that some segfault occurs (quite unlikely). The root cause was that some counters introduced during the malloc optimizations were not properly duplicated in MsgDup(). Note that repeated message processing is not enabled by default. See Changelog for more details. This is a recommended update for all users of the beta branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-176.phtml Changelog: http://www.rsyslog.com/Article404.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From anichols at trumped.org Tue Sep 22 03:50:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 21 Sep 2009 19:50:39 -0600 Subject: [rsyslog] Improving filter performance & general performance Message-ID: Hi Everyone, I have rsyslog 4.4.1 chugging along reasonably well but am looking for ways to improve performance and optimize the filter ruleset. Unfortunately I have to create fairly extensive rulesets to filter on hostname, programname, facility, priority, etc. Some log sources generate a high volume of logs (a few Mbytes/sec) across multiple machines and others generate a fairly routine amount of log data - maybe 5 meg per day. Many filters have duplicate conditions for some values but there is always variance. I have tried to order the rules so that the highest volume logs match first and then are discarded. I've included a sample of the rules used for my highest volume logs (names changed to protect the innocent). If there are ways to chain or nest rules so that I can take advantage of matches already made against a log entry to filter it minimally that would be great. For example, most of the below rules filter on the same facility & list of hostnames but look for different values in the 'rawmsg'. If I could filter on the facility & hostname once and then rawmsg to sort to different destinations I'm guessing it would be lower overhead but I don't really know how the processing logic works. Also - if a condition is not met, are other parts of the filter evaluated? For example, if a message was received on local0, would any conditions beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more efficient to filter on the undecoded value syslogfacility vs. syslogfacility-text? I'm looking for suggestions or general techniques for optimizing rule performance under these circumstances. $template XMLFormat, "%syslogtag%%msg%\n" if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'protocolLogRecord' then -/log/syslog/collated/server/protocol.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'messageLogRecord' then -/log/syslog/collated/server/message.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'clientLogRecord' then -/log/syslog/collated/server/client.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local2' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) then -/log/syslog/collated/server/usage.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'WAP Page Service ID' then -/log/syslog/collated/server/customer-service;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'locationlogrecord' then -/log/syslog/collated/server/lbs.log;XMLFormat & ~ # discard after match From rgerhards at hq.adiscon.com Tue Sep 22 07:23:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Sep 2009 07:23:43 +0200 Subject: [rsyslog] Improving filter performance & general performance References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FFA1@GRFEXC.intern.adiscon.com> Sorry, I am swamped with fixing an important segfault issue we see in one environment, so I do not have time for an more in-depth answer (other list members may have). But I suggest to look into multiple ruleset support, which is in its infancy, but may help. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, September 22, 2009 3:51 AM > To: rsyslog-users > Subject: [rsyslog] Improving filter performance & general performance > > Hi Everyone, > > I have rsyslog 4.4.1 chugging along reasonably well but am > looking for ways > to improve performance and optimize the filter ruleset. > Unfortunately I have > to create fairly extensive rulesets to filter on hostname, > programname, > facility, priority, etc. Some log sources generate a high > volume of logs (a > few Mbytes/sec) across multiple machines and others generate a fairly > routine amount of log data - maybe 5 meg per day. Many filters have > duplicate conditions for some values but there is always > variance. I have > tried to order the rules so that the highest volume logs > match first and > then are discarded. I've included a sample of the rules used > for my highest > volume logs (names changed to protect the innocent). > > If there are ways to chain or nest rules so that I can take > advantage of > matches already made against a log entry to filter it > minimally that would > be great. For example, most of the below rules filter on the > same facility & > list of hostnames but look for different values in the > 'rawmsg'. If I could > filter on the facility & hostname once and then rawmsg to > sort to different > destinations I'm guessing it would be lower overhead but I > don't really know > how the processing logic works. > > Also - if a condition is not met, are other parts of the > filter evaluated? > For example, if a message was received on local0, would any conditions > beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more > efficient to filter on the undecoded value syslogfacility vs. > syslogfacility-text? > > I'm looking for suggestions or general techniques for optimizing rule > performance under these circumstances. > > $template XMLFormat, "%syslogtag%%msg%\n" > > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'protocolLogRecord' then > -/log/syslog/collated/server/protocol.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'messageLogRecord' then > -/log/syslog/collated/server/message.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'clientLogRecord' then > -/log/syslog/collated/server/client.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local2' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) then -/log/syslog/collated/server/usage.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'WAP Page Service ID' then > -/log/syslog/collated/server/customer-service;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'locationlogrecord' then > -/log/syslog/collated/server/lbs.log;XMLFormat > & ~ # discard after match > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 14:54:38 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 07:54:38 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error Message-ID: <20090925125437.GA28679@it.is.rice.edu> I just looked at our PostgreSQL DB for our rsyslog system and the following error was logged: ERROR: value too long for type character varying(60) STATEMENT: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, '////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////') The problem is not so much the error but that it stopped logging to the database. I had to restart rsyslog to get it to start logging once more. Should rsyslog check that its values match the schema or should I need to setup a trigger in the DB to handle off-the-wall input. Regards, Ken From rgerhards at hq.adiscon.com Fri Sep 25 15:30:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:30:58 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Actually, it should have dropped this message, but that depends on the configuration. In general, rsyslog does not know about the schema. And to be more precise, we are not really talking about rsyslogd itself but rather the output plugin. Every output plugin can perform its own checks. But the best answer probably is to use a trigger ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 2:55 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > I just looked at our PostgreSQL DB for our rsyslog system and > the following error was logged: > > ERROR: value too long for type character varying(60) > STATEMENT: insert into SystemEvents (Message, Facility, > FromHost, Priority, DeviceReportedTime, ReceivedAt, > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > '///////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > /////////////////////////////////////////////////////////////////') > > The problem is not so much the error but that it stopped logging > to the database. I had to restart rsyslog to get it to start logging > once more. Should rsyslog check that its values match the schema or > should I need to setup a trigger in the DB to handle off-the-wall > input. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 15:41:28 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 08:41:28 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> References: <20090925125437.GA28679@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Message-ID: <20090925134128.GB28679@it.is.rice.edu> Okay, I will take a look at the output plugin to see where it makes the most sense to fix this. A trigger will always work, but would require every DB to setup and maybe having the plugin perform the truncation would be better. Thank you for the recommendation. Regards, Ken On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > Actually, it should have dropped this message, but that depends on the > configuration. In general, rsyslog does not know about the schema. And to be > more precise, we are not really talking about rsyslogd itself but rather the > output plugin. Every output plugin can perform its own checks. > > But the best answer probably is to use a trigger ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > Kenneth Marshall > > Sent: Friday, September 25, 2009 2:55 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > the following error was logged: > > > > ERROR: value too long for type character varying(60) > > STATEMENT: insert into SystemEvents (Message, Facility, > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > '///////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > /////////////////////////////////////////////////////////////////') > > > > The problem is not so much the error but that it stopped logging > > to the database. I had to restart rsyslog to get it to start logging > > once more. Should rsyslog check that its values match the schema or > > should I need to setup a trigger in the DB to handle off-the-wall > > input. > > > > Regards, > > Ken > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 25 15:43:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:43:36 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> <20090925134128.GB28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BE@GRFEXC.intern.adiscon.com> Ken, The postgres output is quite simple. You may also want to have a look at omoracle, just to see how flexible an output plugin is (postgres was contributed, as was oracle, btw). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 3:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog bug - logging stops after a DB error > > Okay, I will take a look at the output plugin to see where it > makes the most sense to fix this. A trigger will always work, > but would require every DB to setup and maybe having the plugin > perform the truncation would be better. Thank you for the > recommendation. > > Regards, > Ken > > On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > > Actually, it should have dropped this message, but that > depends on the > > configuration. In general, rsyslog does not know about the > schema. And to be > > more precise, we are not really talking about rsyslogd > itself but rather the > > output plugin. Every output plugin can perform its own checks. > > > > But the best answer probably is to use a trigger ;) > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > Kenneth Marshall > > > Sent: Friday, September 25, 2009 2:55 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > > the following error was logged: > > > > > > ERROR: value too long for type character varying(60) > > > STATEMENT: insert into SystemEvents (Message, Facility, > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > > '///////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > > /////////////////////////////////////////////////////////////////') > > > > > > The problem is not so much the error but that it stopped logging > > > to the database. I had to restart rsyslog to get it to > start logging > > > once more. Should rsyslog check that its values match the > schema or > > > should I need to setup a trigger in the DB to handle off-the-wall > > > input. > > > > > > Regards, > > > Ken > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 29 09:59:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Sep 2009 09:59:14 +0200 Subject: [rsyslog] rsyslog 4.5.4 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030E3@GRFEXC.intern.adiscon.com> Hi all, I have just released 4.5.4, a member of the v4-beta branch. This beta contains an important fix that can lead to a segfault when the gzip output writer is used. It also contains some other fixes. Users of v4-beta are strongly advised to upgrade to that version. ChangeLog: http://www.rsyslog.com/Article406.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-177.phtml I hope this release is useful, Rainer From rgerhards at hq.adiscon.com Wed Sep 30 16:31:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 16:31:27 +0200 Subject: [rsyslog] DNS cache and expiration Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Hi all, I think we had discussed this some time in the past, but I cannot find a record of it. So I thought I ask (again?): After my bughunt looks almost completed, I have come back to implementing the name lookup cache. However, I just found out that obtaining the expiration period of the name lookup seems not to be covered by the "usual" socket calls. Or did I just miss them? Any advise, comments and hints regarding name caching and expiration would deeply be appreciated. Rainer From aland at freeradius.org Wed Sep 30 17:44:26 2009 From: aland at freeradius.org (Alan T DeKok) Date: Wed, 30 Sep 2009 17:44:26 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4AC37CDA.9040707@freeradius.org> Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? You didn't miss anything. They're not available. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. You'll have to use a more powerful DNS library, like adns. Alan DeKok. From aoz.syn at gmail.com Wed Sep 30 17:55:09 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 09:55:09 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? Unfortunately not - most resolver libraries provide only what the programmer usually wants - the symbolic (name) or numeric (IP) result of a query. I've not looked carefully at APIs like res_query, though, and that might bring what you need. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. This was my greatest concern with doing *good* internal caching in rsyslog - you're almost guaranteed to use and/or implement a large chunk of proper resolver functionality. Depending on how readable you find Perl, the Net::DNS infrastructure may provide some good pointers on implementing custom resolution toolkits. The djbdns 'dnscache' program (and perhaps the djbdns client resolver library itself) could also be good pointers. From rgerhards at hq.adiscon.com Wed Sep 30 18:52:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 18:52:44 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Thanks for the quick responses, this was what I feared. In the mean time, I have thought a bit about the design. I think I will start not with the cache, but rather by checking to see if I can move the reverse name resolution further down in the processing flow AND move it to one central location. That makes it easier and more efficient to do caching. One drawback when doing so is that the name resolution potentially happens much later than the message reception. Just think about a busy system, or even one waiting for an upstream server to come online again, that lacks behind some minutes or even some hours. When I do the name resolution in the backend thread, the reverse entries may have changed since the message was received :( For many cases, this may be acceptable, for some not. I will probably need to at least define a config value which enables direct queries vs. deferred ones. Any comments on that issue would also be most welcome. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, September 30, 2009 5:55 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards > wrote: > > After my bughunt looks almost completed, I have come back to > implementing the > > name lookup cache. However, I just found out that obtaining the > expiration > > period of the name lookup seems not to be covered by the "usual" > socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration > would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 30 19:00:58 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:00:58 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Hi all, > > I think we had discussed this some time in the past, but I cannot find a > record of it. So I thought I ask (again?): > > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? no they don't, they are name resolution calls, not DNS calls. there are many sources of name resolution (/etc/hosts, LDAP, wins, NIS, etc) and most of them do not have a concept of expiration, and those that do have specific rules for how expiration works (for DNS you have a time after which you are supposed to try and re-resolve it, but can continue to use the name, and a different time after which you are not supposed to use the name for example) > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. going back to basics here. Why is this feature desired? why not just use a caching nameserver listening on localhost? 1. doing a name lookup can cause lost logs as you saw when you were doing testing recently, doing name lookups on each received UDP log can cause you to loose log messages when the OS can no longer queue them up. 2. throughput in a high volume site, the cost of doing a name lookup for each log message can be high enough to be a problem. even a local nameserver can be expensive if you are dealing with 10's of thousands of messages/sec what if you were to move the name resolution from the input module to the output module? that would solve problem #1 immediatly by just eliminating any lookups as the messages are received. note: this may not be possible due to name based rules for what hosts to accept logs from, although the answer here may be to lookup the names when you startup and do the filtering by IP while running. if you delay the name resolution until the output module, you may be able to only do it if the output module needs it (if it uses a name property in the template or ruleset), and if it doesn't you skip the work entirely. in anything short of a very high volume site a local caching nameserver will satisfy the throughput issue nicely (especially if the name resolution is delayed to the output as I mentioned above). in a high volume site I really think that it can be good enough to just throw away the name cache when you do a HUP. a high volume site is going to be doing a HUP on a frequent basis anyway to rotate the logs. This avoids a LOT of overhead and complications in managing expirations. In my site I send rsyslog a HUP every 5 min currently (and have some cases where I plan to change this to every 1 min in the near future) David Lang From david at lang.hm Wed Sep 30 19:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:05:04 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the quick responses, this was what I feared. > > In the mean time, I have thought a bit about the design. I think I will start > not with the cache, but rather by checking to see if I can move the reverse > name resolution further down in the processing flow AND move it to one > central location. That makes it easier and more efficient to do caching. > > One drawback when doing so is that the name resolution potentially happens > much later than the message reception. Just think about a busy system, or > even one waiting for an upstream server to come online again, that lacks > behind some minutes or even some hours. When I do the name resolution in the > backend thread, the reverse entries may have changed since the message was > received :( > > For many cases, this may be acceptable, for some not. I will probably need to > at least define a config value which enables direct queries vs. deferred > ones. > > Any comments on that issue would also be most welcome. it is actually pretty unusual for the source of logs to change it's name. remember that DNS takes time to propogate changes, so even if you do queries immediatly the data may be out of date if you are in an environemnt where it changes. David Lang > Thanks, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, September 30, 2009 5:55 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards >> wrote: >>> After my bughunt looks almost completed, I have come back to >> implementing the >>> name lookup cache. However, I just found out that obtaining the >> expiration >>> period of the name lookup seems not to be covered by the "usual" >> socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration >> would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbiebl at gmail.com Wed Sep 30 19:36:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 30 Sep 2009 19:36:20 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/30 : > > going back to basics here. > > Why is this feature desired? why not just use a caching nameserver > listening on localhost? > Was wondering about this myself. There are small caching nameservers like dnsmasq which will do all the hard work for you. Rainer, have you evaluated such an option? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Sep 30 19:52:07 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:52:07 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Michael Biebl wrote: > 2009/9/30 : >> >> going back to basics here. >> >> Why is this feature desired? why not just use a caching nameserver >> listening on localhost? >> > > Was wondering about this myself. There are small caching nameservers > like dnsmasq which will do all the hard work for you. > Rainer, have you evaluated such an option? by itself, this would not solve the problems 1. with the current situation where the lookups are done as the message is being received, the time taken to do the lookup (especially in the case where the lookup is not yet in the cache) can take long enough that log messages get lost 2. when you are talking message rates of 100K logs/sec and up the overhead of doing a DNS query, even to a server running on localhost that has the info cached in it, becomes a signficant amount of the total time you have to process that message before the next message arrives. David Lang From ktm at rice.edu Wed Sep 30 19:54:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 12:54:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <20090930175456.GD6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > > After my bughunt looks almost completed, I have come back to implementing the > > name lookup cache. However, I just found out that obtaining the expiration > > period of the name lookup seems not to be covered by the "usual" socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. I do not think that the goal of this feature in rsyslog is to re-implement resolver functionality but to provide a fast-path mechanism to map IP addresses to names for the purposes of logging error messages. As such, pretty much the only piece that needs to be tracked within rsyslog is the TTL for the entry and the ip -> name mapping. A thread would be responsible for expiring entries from the cache (or refreshing the timeout) after validating the correctness of the mapping. I think the DNS lookups should be handled by a good resolver like pdns-recursor, djbdns,... The goal here is to allow names in the log entries and not just IP addresses and in a very high performance logging environment. Regards, Ken From david at lang.hm Wed Sep 30 20:15:42 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:15:42 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: >>> After my bughunt looks almost completed, I have come back to implementing the >>> name lookup cache. However, I just found out that obtaining the expiration >>> period of the name lookup seems not to be covered by the "usual" socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. > > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. As such, pretty much the only piece that needs to > be tracked within rsyslog is the TTL for the entry and the ip -> > name mapping. A thread would be responsible for expiring entries > from the cache (or refreshing the timeout) after validating the > correctness of the mapping. I think the DNS lookups should be > handled by a good resolver like pdns-recursor, djbdns,... The > goal here is to allow names in the log entries and not just IP > addresses and in a very high performance logging environment. the trouble is that doing _proper_ TTL expiration isn't as simple as it sounds. and if you are willing to back away from 'proper' expiration to something that will work in practice, why not go much further (as I have detailed in the other messages) David Lang From aoz.syn at gmail.com Wed Sep 30 20:18:37 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 12:18:37 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. Although I agree with your assessment of the goal, the only difference I see between the two is wording semantics. An RFC-compliant DNS cache will, for all intents and purposes, look an awful lot like any other caching, recursive-only DNS resolver (like dnscache). The only major difference would be that it would accept requests via an API as opposed to through a socket interface. Regardless, I have to sit on the same side as David and Michael - in very high-performance environments, I doubt the difference between an internal cache and an external one is going to be significant. From ktm at rice.edu Wed Sep 30 20:25:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 13:25:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <20090930182556.GE6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 11:15:42AM -0700, david at lang.hm wrote: > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > >>> After my bughunt looks almost completed, I have come back to implementing the > >>> name lookup cache. However, I just found out that obtaining the expiration > >>> period of the name lookup seems not to be covered by the "usual" socket > >>> calls. Or did I just miss them? > >> > >> Unfortunately not - most resolver libraries provide only what the > >> programmer usually wants - the symbolic (name) or numeric (IP) result > >> of a query. I've not looked carefully at APIs like res_query, though, > >> and that might bring what you need. > >> > >>> Any advise, comments and hints regarding name caching and expiration would > >>> deeply be appreciated. > >> > >> This was my greatest concern with doing *good* internal caching in > >> rsyslog - you're almost guaranteed to use and/or implement a large > >> chunk of proper resolver functionality. Depending on how readable you > >> find Perl, the Net::DNS infrastructure may provide some good pointers > >> on implementing custom resolution toolkits. The djbdns 'dnscache' > >> program (and perhaps the djbdns client resolver library itself) could > >> also be good pointers. > > > > I do not think that the goal of this feature in rsyslog is to > > re-implement resolver functionality but to provide a fast-path > > mechanism to map IP addresses to names for the purposes of logging > > error messages. As such, pretty much the only piece that needs to > > be tracked within rsyslog is the TTL for the entry and the ip -> > > name mapping. A thread would be responsible for expiring entries > > from the cache (or refreshing the timeout) after validating the > > correctness of the mapping. I think the DNS lookups should be > > handled by a good resolver like pdns-recursor, djbdns,... The > > goal here is to allow names in the log entries and not just IP > > addresses and in a very high performance logging environment. > > the trouble is that doing _proper_ TTL expiration isn't as simple as it > sounds. > > and if you are willing to back away from 'proper' expiration to something > that will work in practice, why not go much further (as I have detailed in > the other messages) > > David Lang I agree. I only mention TTL values as a reasonable upperbound of the refresh check. The advantage is to remove the large pause in logging due to a DNS refresh after a HUP to rsyslog, if that were the only method to flush/refresh. Like you mention, the IPs/names of systems being logged change rarely so we should tune this for speed and not worry about expiration correctness. I do not agree so other statements that an external cache will perform as well as an internal cache. Too many software products that I work with have needed exactly that functionality to support very high levels of performance. Regards, Ken From david at lang.hm Wed Sep 30 20:51:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:51:28 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, RB wrote: > On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: >> I do not think that the goal of this feature in rsyslog is to >> re-implement resolver functionality but to provide a fast-path >> mechanism to map IP addresses to names for the purposes of logging >> error messages. > > Although I agree with your assessment of the goal, the only difference > I see between the two is wording semantics. An RFC-compliant DNS > cache will, for all intents and purposes, look an awful lot like any > other caching, recursive-only DNS resolver (like dnscache). The only > major difference would be that it would accept requests via an API as > opposed to through a socket interface. > > Regardless, I have to sit on the same side as David and Michael - in > very high-performance environments, I doubt the difference between an > internal cache and an external one is going to be significant. actually, I am thinking that in a high-performance environment, the difference between an internal name cache and an external one _is_ significant I just don't think the internal one should be a DNS RFC complient one. if you put everything in /etc/hosts it is faster than doing a query against a local caching server, but it's still significantly slower than looking it up in memory. remember that when you make the gethostbyname() call it has to do a lot of checking to see which name resolver libraries you have configured (which includes checking for the existance of multiple files), then call them in order until it finds the name. if you do a strace of this sometime you will see how much stuff goes on under the covers. skipping all of this is significant at high log rates. David Lang From david at lang.hm Wed Sep 30 20:54:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:54:12 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930182556.GE6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <20090930182556.GE6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>> mechanism to map IP addresses to names for the purposes of logging >>> error messages. As such, pretty much the only piece that needs to >>> be tracked within rsyslog is the TTL for the entry and the ip -> >>> name mapping. A thread would be responsible for expiring entries >>> from the cache (or refreshing the timeout) after validating the >>> correctness of the mapping. I think the DNS lookups should be >>> handled by a good resolver like pdns-recursor, djbdns,... The >>> goal here is to allow names in the log entries and not just IP >>> addresses and in a very high performance logging environment. >> >> the trouble is that doing _proper_ TTL expiration isn't as simple as it >> sounds. >> >> and if you are willing to back away from 'proper' expiration to something >> that will work in practice, why not go much further (as I have detailed in >> the other messages) >> >> David Lang > > I agree. I only mention TTL values as a reasonable upperbound of the > refresh check. The advantage is to remove the large pause in logging > due to a DNS refresh after a HUP to rsyslog, if that were the only > method to flush/refresh. Like you mention, the IPs/names of systems > being logged change rarely so we should tune this for speed and > not worry about expiration correctness. I do not agree so other > statements that an external cache will perform as well as an internal > cache. Too many software products that I work with have needed exactly > that functionality to support very high levels of performance. actually, you could have the cache be configurable in three modes 1. no caching 2. blank the cache on HUP 3. never blank the cache (i.e. require a full restart to clear it) David Lang From rgerhards at hq.adiscon.com Wed Sep 30 21:45:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:45:00 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Thanks for the good discussion. I am lacking somewhat behind, but will review it in depth tomorrow morning. I just wanted to stress the point that an external cache does not really help, much for the reason David mentioned: if you process messages at very high data rates, the context switch overhead involved with any external solution is extremely costly. Also, in the usual cases, I may do several million queries within a few seconds for just a handful of hosts. With an internal cache, the overhead in doing so is very minimal. With an external solution, the overhead in calling the external cache causes a lot of performance degredation, what in the case of UDP also implies (heavy!) message loss. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 8:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >>> mechanism to map IP addresses to names for the purposes of logging > >>> error messages. As such, pretty much the only piece that needs to > >>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>> name mapping. A thread would be responsible for expiring entries > >>> from the cache (or refreshing the timeout) after validating the > >>> correctness of the mapping. I think the DNS lookups should be > >>> handled by a good resolver like pdns-recursor, djbdns,... The > >>> goal here is to allow names in the log entries and not just IP > >>> addresses and in a very high performance logging environment. > >> > >> the trouble is that doing _proper_ TTL expiration isn't as > simple as it > >> sounds. > >> > >> and if you are willing to back away from 'proper' > expiration to something > >> that will work in practice, why not go much further (as I > have detailed in > >> the other messages) > >> > >> David Lang > > > > I agree. I only mention TTL values as a reasonable upperbound of the > > refresh check. The advantage is to remove the large pause in logging > > due to a DNS refresh after a HUP to rsyslog, if that were the only > > method to flush/refresh. Like you mention, the IPs/names of systems > > being logged change rarely so we should tune this for speed and > > not worry about expiration correctness. I do not agree so other > > statements that an external cache will perform as well as > an internal > > cache. Too many software products that I work with have > needed exactly > > that functionality to support very high levels of performance. > > actually, you could have the cache be configurable in three modes > > 1. no caching > > 2. blank the cache on HUP > > 3. never blank the cache (i.e. require a full restart to clear it) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Sep 30 21:53:00 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 12:53:00 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the good discussion. I am lacking somewhat behind, but will review > it in depth tomorrow morning. > > I just wanted to stress the point that an external cache does not really > help, much for the reason David mentioned: if you process messages at very > high data rates, the context switch overhead involved with any external > solution is extremely costly. Also, in the usual cases, I may do several > million queries within a few seconds for just a handful of hosts. With an > internal cache, the overhead in doing so is very minimal. With an external > solution, the overhead in calling the external cache causes a lot of > performance degredation, what in the case of UDP also implies (heavy!) > message loss. the message loss problem with UDP will not be solved completely by an internal cache. when the source is not in the cache and you have to go out to find it the lookup can take several seconds. moving the lookup out of the input module and into the output module would address this, anything else would leave you with losses as the cache gets populated. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, September 30, 2009 8:54 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >> >>>>> mechanism to map IP addresses to names for the purposes of logging >>>>> error messages. As such, pretty much the only piece that needs to >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> >>>>> name mapping. A thread would be responsible for expiring entries >>>>> from the cache (or refreshing the timeout) after validating the >>>>> correctness of the mapping. I think the DNS lookups should be >>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>> goal here is to allow names in the log entries and not just IP >>>>> addresses and in a very high performance logging environment. >>>> >>>> the trouble is that doing _proper_ TTL expiration isn't as >> simple as it >>>> sounds. >>>> >>>> and if you are willing to back away from 'proper' >> expiration to something >>>> that will work in practice, why not go much further (as I >> have detailed in >>>> the other messages) >>>> >>>> David Lang >>> >>> I agree. I only mention TTL values as a reasonable upperbound of the >>> refresh check. The advantage is to remove the large pause in logging >>> due to a DNS refresh after a HUP to rsyslog, if that were the only >>> method to flush/refresh. Like you mention, the IPs/names of systems >>> being logged change rarely so we should tune this for speed and >>> not worry about expiration correctness. I do not agree so other >>> statements that an external cache will perform as well as >> an internal >>> cache. Too many software products that I work with have >> needed exactly >>> that functionality to support very high levels of performance. >> >> actually, you could have the cache be configurable in three modes >> >> 1. no caching >> >> 2. blank the cache on HUP >> >> 3. never blank the cache (i.e. require a full restart to clear it) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 30 21:56:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:56:33 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 9:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > Thanks for the good discussion. I am lacking somewhat > behind, but will review > > it in depth tomorrow morning. > > > > I just wanted to stress the point that an external cache > does not really > > help, much for the reason David mentioned: if you process > messages at very > > high data rates, the context switch overhead involved with > any external > > solution is extremely costly. Also, in the usual cases, I > may do several > > million queries within a few seconds for just a handful of > hosts. With an > > internal cache, the overhead in doing so is very minimal. > With an external > > solution, the overhead in calling the external cache causes a lot of > > performance degredation, what in the case of UDP also > implies (heavy!) > > message loss. > > the message loss problem with UDP will not be solved completely by an > internal cache. when the source is not in the cache and you > have to go out > to find it the lookup can take several seconds. > > moving the lookup out of the input module and into the output > module would > address this, anything else would leave you with losses as > the cache gets > populated. That's right and that's one reason why I intend to move this (optionally) over to the "backend" processing. However, even that does not completely solve the message loss problem, as we, in extreme cases, may loose messages when the queue is full - and for a myriad of other reasons, like routers discarding frames and such. Of course, you know that, but I'd like to mention if for those folks that at some time find our conversation via Google ;) Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Wednesday, September 30, 2009 8:54 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] DNS cache and expiration > >> > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > >> > >>>>> mechanism to map IP addresses to names for the purposes > of logging > >>>>> error messages. As such, pretty much the only piece > that needs to > >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>>>> name mapping. A thread would be responsible for expiring entries > >>>>> from the cache (or refreshing the timeout) after validating the > >>>>> correctness of the mapping. I think the DNS lookups should be > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > >>>>> goal here is to allow names in the log entries and not just IP > >>>>> addresses and in a very high performance logging environment. > >>>> > >>>> the trouble is that doing _proper_ TTL expiration isn't as > >> simple as it > >>>> sounds. > >>>> > >>>> and if you are willing to back away from 'proper' > >> expiration to something > >>>> that will work in practice, why not go much further (as I > >> have detailed in > >>>> the other messages) > >>>> > >>>> David Lang > >>> > >>> I agree. I only mention TTL values as a reasonable > upperbound of the > >>> refresh check. The advantage is to remove the large pause > in logging > >>> due to a DNS refresh after a HUP to rsyslog, if that were the only > >>> method to flush/refresh. Like you mention, the IPs/names > of systems > >>> being logged change rarely so we should tune this for speed and > >>> not worry about expiration correctness. I do not agree so other > >>> statements that an external cache will perform as well as > >> an internal > >>> cache. Too many software products that I work with have > >> needed exactly > >>> that functionality to support very high levels of performance. > >> > >> actually, you could have the cache be configurable in three modes > >> > >> 1. no caching > >> > >> 2. blank the cache on HUP > >> > >> 3. never blank the cache (i.e. require a full restart to clear it) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 1 10:51:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 10:51:35 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> > As a side-note: I think that my UDP message loss may partly be related > to DNS > resolution. I will try this in a lab tomorrow. But I still think a lot > of > packets never leave the source system. This may be related to the > virtual > environment I am currently using for the lab. I hope to be able to > generate > the traffic by a program, because that offers me the flexibility (now > and in > the future) to test complex messages scenarios (what, granted, does not > help > if it does not expose the problem...). Very interesting - I just did a couple of tests with UDP and various DNS resolution settings. The message loss I see is definitely related to DNS resolution. This is especially interesting as in my lab setup there should be no need to do more than the initial query. This points into some area that either is buggy or needs to be optimized. When I turn off DNS resolution, I have far fewer lost message. Still, there is between 1% and 10% loss for reasonable high traffic, but that is OK from my expectations given the lab environment I use. With DNS resolution, I have > 90% loss, and this difference is clearly not acceptable. I will look into this issue, but will try to find the segfault first (better not change the environment so that the bug moves to some other region). In the light of this, I'll probably rerun some of my tests today without reverse DNS resolution - the higher rate will hopefully trigger the bug in my lab. Rainer From rgerhards at hq.adiscon.com Tue Sep 1 12:26:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 12:26:31 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 01, 2009 10:52 AM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 / UDP message loss > > > As a side-note: I think that my UDP message loss may partly be > related > > to DNS > > resolution. I will try this in a lab tomorrow. But I still think a > lot > > of > > packets never leave the source system. This may be related to the > > virtual > > environment I am currently using for the lab. I hope to be able to > > generate > > the traffic by a program, because that offers me the flexibility (now > > and in > > the future) to test complex messages scenarios (what, granted, does > not > > help > > if it does not expose the problem...). > > Very interesting - I just did a couple of tests with UDP and various > DNS > resolution settings. The message loss I see is definitely related to > DNS > resolution. This is especially interesting as in my lab setup there > should be > no need to do more than the initial query. This points into some area > that > either is buggy or needs to be optimized. ... my simplistic requery-avoidance logic does not take the source port into account. So a requery is also done if the host is the same as before, but the port changes. Thus the difference. Needs to be optimized ;) While this does not point to an obvious bug, I'll still try to get a segfault without DNS resolution. Rainer From mikel at irontec.com Tue Sep 1 14:20:35 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 01 Sep 2009 14:20:35 +0200 Subject: [rsyslog] milliseconds timestamp Message-ID: <4A9D1193.4090806@irontec.com> hi Some news about this? http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html Maybe with a bounty? thanks From rgerhards at hq.adiscon.com Tue Sep 1 14:25:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 14:25:12 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD4@GRFEXC.intern.adiscon.com> Hi, Andre has just gone on vacation, expect a real answer in two weeks ;) But I don't think he had time to look at this (too many paid projects in the way...). So a bounty may be useful ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 01, 2009 2:21 PM > To: rsyslog-users > Subject: [rsyslog] milliseconds timestamp > > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Tue Sep 1 18:59:25 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Tue, 01 Sep 2009 18:59:25 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> Message-ID: <4A9D52ED.4090103@redhat.com> On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas From rgerhards at hq.adiscon.com Tue Sep 1 19:55:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:55:57 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001501ca2b2d$7e16309d$100013ac@intern.adiscon.com> Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 1 19:58:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:58:56 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001601ca2b2d$e8e76725$100013ac@intern.adiscon.com> I was too quick. I should have said "backported with ease **to v4**". v2 obviously is so outdated, that this will require a totally different effort. ----- Urspr?ngliche Nachricht ----- Von: "Rainer Gerhards" An: "rsyslog-users" Gesendet: 01.09.09 19:56 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Sep 2 01:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 16:06:17 -0700 (PDT) Subject: [rsyslog] -cX command line option Message-ID: with version 3+ do we really need to change the X in this option? if you run v5 with -c4 is it really going to do something different with the config file than if you use -c5? yes, there are new config options in the newer versions, and once in a while some depriciated config options stop working, but does changing from -c3 to -c4 to -c5 actually fix any of these? in my testing I keep switching between the v4 series and the v5 series and having to change the startup to give the correct -c flag has tripped me up more than once. it would also be helpful if rsyslog would spit out errors about unknown config files (either to the console or as syslog messages) without needing to be in debug mode. it may that it tries to do this, but I don't see them (either with the debian startup scripts or when starting it directly on the command line) David Lang From david at lang.hm Wed Sep 2 03:11:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 18:11:56 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251715849.4897.13.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: I got a core file with 4.2.0 I did git checkout -f v4.2.0 configure --enable-imfile and installed the result. I will go through the core file either later tonight or in the morning. in this case it did take a while for it to die. (over an hour) David Lang On Mon, 31 Aug 2009, Rainer Gerhards wrote: > Date: Mon, 31 Aug 2009 12:50:49 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > On Fri, 2009-08-28 at 14:55 -0700, david at lang.hm wrote: >> On Fri, 28 Aug 2009, Rainer Gerhards wrote: >>> Also, it would be good if you could --enable-rtinst --enable-debug and try >>> out that version on your machine. I am a bit concerned about the speed of the >>> resulting executable, it may be too slow. You do not need to run it in debug >>> mode itself. These option (especially--enable-debug) will activate in-depth >>> runtime checks (assert, will abort when something wrong happens) and my hope >>> is that they will catch the bug closer to the root cause. If so, I would need >>> the gdb abort info (actually enabling debug output would be an option some >>> time later). >>> >>> Please let me know what would be OK with you. >> >> I will give this a try. >> >> I was going to suggest that since we have the message getting corrupted it >> may make sense to make a temporary branch that has multiple message >> buffers and at various times through the message processing it makes a >> copy of the emssage to the buffer. when the system crashes I will be able >> to look at the core and see where the message is getting corrupted. > > David, I fear it is even more complicated than that. It looks like not > only the message got corrupted but the message object itself. There are > already two copies of some of the message elements, and they also look > inconsistent - except, if we really had a null message, that is one with > no content at all (and generating a message object from a null message, > I think, would be a bug in itself - but I am sure there are no such > messages in your actual traffic). If you think there could be a real > null message, I'd follow that path (will probably do so in any case...). > > I think that what really happens is that some part of the code runs > wild, thus invalidating some random part of the main memory. At some > times, it hits queue structures (or the message object that is held by > them) and if so, we will see the abort you experience. With that > scenario, duplicating the message buffer does not really help, because > looking at the corrupted message object would not provide any additional > information. > > However, if that's easy enough to reproduce, it would probably be good > if you could send me the core analysis (the backtrace and the print > statements) from a few (five maybe?) independent aborts. Maybe they show > a pattern. It would probably best to send them via private mail, as I am > not sure if they disclose more than they should. > >> >> I will see about doing a tcpdump at the time that I do this and send it to >> you (I'll need to check with management, but since we have a contract in >> place for other reasons I think we can do this) >> > > That would probably be a good thing. I've made some progress with my > testing tool, and I have created a basic version right now. Probably not > good enough to mimic your traffic pattern, but closer. I am doing a test > run for quite some time now, unfortunately so far without abort. > > Note that I run into the trouble with UDP - even though I've put some > one-ms sleeps into the code, I lose a lot of messages, as it looks even > before they hit the wire. It's always real trobulesome to test with > UDP... > > Rainer >> I can't do this late on a friday, but I should be able to do this monday >> afternoon. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 12:14:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:14:21 +0200 Subject: [rsyslog] abort in 4.2.1 In-Reply-To: References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: <1251886461.5821.8.camel@rgf11> David, thanks for this test. The outcome is obviously other than I hoped/expected, but that makes it very useful. Obviously I have been looking for the wrong root cause. Any abort information you can provide would be useful. Even more useful would be if you could try out some earlier releases. Not sure if that is possible from a feature point of view. If it is, I would appreciate if you could give v3-stable a try and, if and only if that fails, too, checkout v3.18.6 and try that one. The 3.18.6 is the version that Debian ships and so I know it has a lot of testers and received a lot of bug-finding attention (I thankfully receive lots of very qualified bug reports from the Debian community :)). Please let me know what is possible. In any case, the 4.2.0 failure even more points to environment-specific problems. Rainer On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: > I got a core file with 4.2.0 > > I did git checkout -f v4.2.0 configure --enable-imfile and installed the > result. > > I will go through the core file either later tonight or in the morning. > > in this case it did take a while for it to die. (over an hour) > > David Lang From rgerhards at hq.adiscon.com Wed Sep 2 12:23:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:23:02 +0200 Subject: [rsyslog] -cX command line option In-Reply-To: References: Message-ID: <1251886982.5821.17.camel@rgf11> On Tue, 2009-09-01 at 16:06 -0700, david at lang.hm wrote: > with version 3+ do we really need to change the X in this option? > if you run v5 with -c4 is it really going to do something different with > the config file than if you use -c5? > > yes, there are new config options in the newer versions, and once in a > while some depriciated config options stop working, but does changing from > -c3 to -c4 to -c5 actually fix any of these? The -cX is more a vehicle to change things like *defaults*, that is something that breaks existing configurations. So far, there is no difference between v4 and v5 in this regard. However, I would not like to give up this vehicle. That would actually force me to never change any defaults. > > in my testing I keep switching between the v4 series and the v5 series and > having to change the startup to give the correct -c flag has tripped me up > more than once. > > it would also be helpful if rsyslog would spit out errors about unknown > config files (either to the console or as syslog messages) without needing > to be in debug mode. The current versions already does this. I think they go to stderr (maybe stdout). > > it may that it tries to do this, but I don't see them (either with the > debian startup scripts or when starting it directly on the command line) > I could offer the follwing solution for what you describe: I could permit (in newer v3/v4 builds) to specify a higher version (-c5) and only sending an alert. Doing so, of course, means "I know what I do and I can live with any consequences from it" what should be fine for your use case. Please let me know if that would be helpful for you. Rainer From tbergfeld at hq.adiscon.com Wed Sep 2 14:41:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 2 Sep 2009 14:41:49 +0200 Subject: [rsyslog] rsyslog 4.4.1 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDDF@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.1., a member of the v4-development branch. This is a bug-fixing release, providing some important fixes for issues that have only been detected after the beta phase. Some of them are serious (like a segfault when UDP messageforwarding is activated), so users of 4.4.0 are urged to upgrade to this release. Have a look at the change log to see all new features included in this release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-173.phtml Changelog: http://www.rsyslog.com/Article398.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From mikel at irontec.com Wed Sep 2 14:52:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 14:52:02 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9D1193.4090806@irontec.com> References: <4A9D1193.4090806@irontec.com> Message-ID: <4A9E6A72.8080202@irontec.com> Ok, I will comunicate you if we decide. Is the development of phplogcon frezzed? the last version is of January 27 ... Thanks Mikel Jimenez wrote: > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > From rgerhards at hq.adiscon.com Wed Sep 2 14:56:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 14:56:34 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 2:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of January > 27 ... Definitely not, it is active. But it looks like the web site did not receive proper attention. I'll check what's going on... See the git log for what's going on: http://git.adiscon.com/?p=phplogcon.git;a=summary The pace of changes is somewhat lower than in the initial phase, because there have been more pressing projects. But I have talked with Andre over big reporting features, which he will (hopefully) be able to tackle once he is back from his vacation. Rainer From mikel at irontec.com Wed Sep 2 15:00:57 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 15:00:57 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> Message-ID: <4A9E6C89.4060309@irontec.com> Ahhh!! Ok Ok I see that it is active... so in near future the web page would be syncronized with the real state of the development? I usually use the web for news about phplogcon. (www.phplogcon.org) Thanks Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Wednesday, September 02, 2009 2:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Ok, I will comunicate you if we decide. >> >> Is the development of phplogcon frezzed? the last version is of January >> 27 ... >> > > Definitely not, it is active. But it looks like the web site did not receive > proper attention. I'll check what's going on... > > See the git log for what's going on: > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > The pace of changes is somewhat lower than in the initial phase, because > there have been more pressing projects. But I have talked with Andre over big > reporting features, which he will (hopefully) be able to tackle once he is > back from his vacation. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 15:03:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 15:03:46 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> <4A9E6C89.4060309@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ahhh!! Ok Ok > > I see that it is active... so in near future the web page would be > syncronized with the real state of the development? > > I usually use the web for news about phplogcon. (www.phplogcon.org) > Thanks I've already pinged the web folks. I agree, I also go to the sites. I think there also have been no release annoucements (actually my primary source of new release info). Interestingly, I just saw that freshmeat has announcements: http://freshmeat.net/projects/phplogcon/ ... strange ;) Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Wednesday, September 02, 2009 2:52 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] milliseconds timestamp > >> > >> Ok, I will comunicate you if we decide. > >> > >> Is the development of phplogcon frezzed? the last version is of > January > >> 27 ... > >> > > > > Definitely not, it is active. But it looks like the web site did not > receive > > proper attention. I'll check what's going on... > > > > See the git log for what's going on: > > > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > > > The pace of changes is somewhat lower than in the initial phase, > because > > there have been more pressing projects. But I have talked with Andre > over big > > reporting features, which he will (hopefully) be able to tackle once > he is > > back from his vacation. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Wed Sep 2 16:45:39 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 11:45:39 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" Message-ID: <4A9E8513.8020107@swiss-wireless.com.ar> Hello I'm pretty new to rsyslog. I know that you can specifiy 2 server for remote logging which will be handled "independent" i.e. rsyslog will log to the 2 server in parallel. What I want is a primary rsyslog server and a secondary rsyslog server and only if the primary is not avaiable the secondary should be used. Is this possible with rsyslog? Thanks for any hints Beat From rgerhards at hq.adiscon.com Wed Sep 2 19:06:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 19:06:14 +0200 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" References: <4A9E8513.8020107@swiss-wireless.com.ar> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Beat Meier > Sent: Wednesday, September 02, 2009 4:46 PM > To: rsyslog-users > Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with > "priority" > > Hello > > I'm pretty new to rsyslog. > I know that you can specifiy 2 server for remote logging which will be > handled "independent" > i.e. rsyslog will log to the 2 server in parallel. > What I want is a primary rsyslog server and a secondary rsyslog server > and only if the primary > is not avaiable the secondary should be used. Is this possible with > rsyslog? > > Thanks for any hints > > Beat > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 2 19:55:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 2 Sep 2009 10:55:32 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251886461.5821.8.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> <1251886461.5821.8.camel@rgf11> Message-ID: On Wed, 2 Sep 2009, Rainer Gerhards wrote: > David, > > thanks for this test. The outcome is obviously other than I > hoped/expected, but that makes it very useful. Obviously I have been > looking for the wrong root cause. > > Any abort information you can provide would be useful. Even more useful > would be if you could try out some earlier releases. Not sure if that is > possible from a feature point of view. > > If it is, I would appreciate if you could give v3-stable a try and, if > and only if that fails, too, checkout v3.18.6 and try that one. The > 3.18.6 is the version that Debian ships and so I know it has a lot of > testers and received a lot of bug-finding attention (I thankfully > receive lots of very qualified bug reports from the Debian > community :)). > > Please let me know what is possible. In any case, the 4.2.0 failure even > more points to environment-specific problems. I haven't gone back to the 3.x series, but I did several more runs with 4.2.0 doing the folloiwng killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r -h ; mv /core /core-4.2.0-10 I have several complete steps, as well as several partial sets of data. I will gzip them and attempt to send them to you directly. David Lang > Rainer > > On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: >> I got a core file with 4.2.0 >> >> I did git checkout -f v4.2.0 configure --enable-imfile and installed the >> result. >> >> I will go through the core file either later tonight or in the morning. >> >> in this case it did take a while for it to die. (over an hour) >> >> David Lang > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbe_ml at swiss-wireless.com.ar Wed Sep 2 22:18:09 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 17:18:09 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> References: <4A9E8513.8020107@swiss-wireless.com.ar> <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> Message-ID: <4A9ED301.2020500@swiss-wireless.com.ar> Thanks Rainer That's exactly what I looked for. One more question I use templates of the form: $template DynFileAuth,"/var/log/%HOSTNAME%/auth.log" $template DynFileSyslog,"/var/log/%HOSTNAME%/syslog" $template DynFileCron,"/var/log/%HOSTNAME%/cron.log" How can I use variables to replace the path /var/log So I can use something like: path="/var/log" $template DynFileAuth,"$path/%HOSTNAME%/auth.log" $template DynFileSyslog,"$path/%HOSTNAME%/syslog" $template DynFileCron,"$path/%HOSTNAME%/cron.log" Is the template way the only one? I think that template is expanded at runtime, isn't it? Is there a variable method that is expanded when daemon starts, for efficiency? I have nothing found in the wiki with search, no in the docu index, nor the man page of rsyslog.conf. Greetings and thanks Beat Rainer Gerhards wrote: >please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > > > From mbe_ml at swiss-wireless.com.ar Thu Sep 3 00:19:19 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 19:19:19 -0300 Subject: [rsyslog] Which librelp does work with rsyslog-3.18.6 ? Message-ID: <4A9EEF67.20502@swiss-wireless.com.ar> Hello I'm using rsyslog 3.18.6 on debian 4.0 (backport of rsyslog) There is no backport of a newer version with librelp support :-( so I downloaded librelp-0.1.3 and compiled it on debian 4.0. This will install librelp.so I have moved this to /usr/lib/rsyslog but rsyslog is complaining that he cannot find imrelp.so. I have configured the module imrelp as noted some ware else. I have now seen that debian version of relp (for debian 5.0) has imrelp.so and omrelp.so defined. Has the name of the shared object changed from an old release to 0.1.3? Nor renaming librelp.so nor changing module name did work. (renaming library results in an undedfined symbol: modInit) Which release of lbrelp can I use with rsyslog V3.18.6 or can I use librelp only with rsyslog-4.4? Greetings and thanks Beat From joe at joetify.com Thu Sep 3 05:11:28 2009 From: joe at joetify.com (Joe Williams) Date: Wed, 2 Sep 2009 20:11:28 -0700 Subject: [rsyslog] case sensitivity in templates Message-ID: <20090902201128.507f2449@der-dieb> Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 07:47:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 07:47:14 +0200 Subject: [rsyslog] case sensitivity in templates Message-ID: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> You need to check the property replacer documentation. There are options for case conversion. I don't know the exact syntax out of my head, but it is along the lines of %field:::ucase%. Hth rainer ----- Urspr?ngliche Nachricht ----- Von: "Joe Williams" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 05:19 Betreff: [rsyslog] case sensitivity in templates Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 3 12:23:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 12:23:45 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Hi David, > I haven't gone back to the 3.x series, but I did several more runs with > 4.2.0 doing the folloiwng > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r > -h ; mv /core /core-4.2.0-10 > > I have several complete steps, as well as several partial sets of data. > I > will gzip them and attempt to send them to you directly. Thanks for the data set, I am right now working on it. Unfortunately, as I feared, the core files do not really help. There is a big mismatch between your system environment and mine, and so gdb is not able to extract any useful information. All I see is that there are six threads in the system, and the rest is almost only question marks. So it would be great if you could issue the gdb commands in your environment and let me know the outcome. Thanks, Rainer From joe at joetify.com Thu Sep 3 17:51:23 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 08:51:23 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> Message-ID: <20090903085123.0bcea4b0@der-dieb> Thanks, that worked perfectly. -Joe On Thu, 3 Sep 2009 07:47:14 +0200 "Rainer Gerhards" wrote: > You need to check the property replacer documentation. There are > options for case conversion. I don't know the exact syntax out of my > head, but it is along the lines of %field:::ucase%. > > Hth > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Joe Williams" > An: "rsyslog at lists.adiscon.com" > Gesendet: 03.09.09 05:19 > Betreff: [rsyslog] case sensitivity in templates > > > Hello, I am new to the list sorry if this has been covered already. I > am logging using a per-host template like: > > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > The services that log directly to the rsyslog server (haproxy, etc) > are using all lower case hostname directories where as the logs that > use the rsyslog client daemon to log to the server are using the case > specified in the hostname which in my case have capital letters in > them. > > Is there any way to specify which to use? I would like to have a > single directory for each host regardless of the case used in the > hostname. It doesn't matter to me which case is used as long as it's > the same for all logs. > > Thanks. > > -Joe > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From joe at joetify.com Thu Sep 3 18:02:58 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 09:02:58 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <20090903085123.0bcea4b0@der-dieb> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> <20090903085123.0bcea4b0@der-dieb> Message-ID: <20090903090258.384a9214@der-dieb> BTW, if anyone else has this problem or something similar the doc is at: http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.phtml The fix was changing the template to be: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase%/debug" Thanks again. -Joe On Thu, 3 Sep 2009 08:51:23 -0700 Joe Williams wrote: > Thanks, that worked perfectly. > > -Joe > > > On Thu, 3 Sep 2009 07:47:14 +0200 > "Rainer Gerhards" wrote: > > > You need to check the property replacer documentation. There are > > options for case conversion. I don't know the exact syntax out of my > > head, but it is along the lines of %field:::ucase%. > > > > Hth > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Joe Williams" > > An: "rsyslog at lists.adiscon.com" > > Gesendet: 03.09.09 05:19 > > Betreff: [rsyslog] case sensitivity in templates > > > > > > Hello, I am new to the list sorry if this has been covered already. > > I am logging using a per-host template like: > > > > $template > > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > > > The services that log directly to the rsyslog server (haproxy, etc) > > are using all lower case hostname directories where as the logs that > > use the rsyslog client daemon to log to the server are using the > > case specified in the hostname which in my case have capital > > letters in them. > > > > Is there any way to specify which to use? I would like to have a > > single directory for each host regardless of the case used in the > > hostname. It doesn't matter to me which case is used as long as it's > > the same for all logs. > > > > Thanks. > > > > -Joe > > > > > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 18:05:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 18:05:42 +0200 Subject: [rsyslog] case sensitivity in templates References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com><20090903085123.0bcea4b0@der-dieb> <20090903090258.384a9214@der-dieb> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDF5@GRFEXC.intern.adiscon.com> > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercas > e%/debug" Now that you say it: it would probably make sense to use one of the "secpath" (or so) options to make this file writer more secure - see the doc you quoted for details. It will then look something along these lines: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase,secp ath-...%/debug" > > Thanks again. my pleasure :) Rainer From corsmith at gmail.com Thu Sep 3 19:39:15 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 13:39:15 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <8061fbee0909031039y637bdf52j72b1322cf2538f55@mail.gmail.com> I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:01:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:01:12 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Thats an very interesting effort. Do you needed to patch the source? Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 19:45 Betreff: [rsyslog] rsyslog 4.4.1 and solaris I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From corsmith at gmail.com Thu Sep 3 20:22:09 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 14:22:09 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> References: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909031122x4e6469fo426368e6b7363a84@mail.gmail.com> On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:58:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:58:27 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog-users" Gesendet: 03.09.09 20:22 Betreff: Re: [rsyslog] rsyslog 4.4.1 and solaris On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From srinivasan.sreenivasan at gmail.com Fri Sep 4 01:19:05 2009 From: srinivasan.sreenivasan at gmail.com (Srinivasan Sreenivasan) Date: Thu, 3 Sep 2009 18:19:05 -0500 Subject: [rsyslog] rsyslog on Solaris Message-ID: Hi, We are trying to run rsyslog version 4.4.1 on Solaris 2.8. We cannot get it do any logging. Rainer has a blog entry (its a bit dated) that says that rsyslogd does not do local logging on Solaris. Is that still valid? -Srini From joe at joetify.com Fri Sep 4 01:48:18 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 16:48:18 -0700 Subject: [rsyslog] logging wildcards Message-ID: <20090903164818.00c7bc9d@der-dieb> Hello again, I am trying to log everything (*.*) to /var/log/syslog but local*.*. I tried a couple different ways to do this but didn't find a solution. Is this possible? I have a couple services I want to log to their own file rather than syslog, messages, etc. Any help is appreciated. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:05:04 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903164818.00c7bc9d@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > Hello again, > > I am trying to log everything (*.*) to /var/log/syslog but local*.*. I > tried a couple different ways to do this but didn't find a solution. Is > this possible? I have a couple services I want to log to their own file > rather than syslog, messages, etc. if you just do *.* /var/log/syslog that will write everything to that file I'm not sure what you are trying to say when you say 'but local*.*' above. David Lang From joe at joetify.com Fri Sep 4 02:10:22 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 17:10:22 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> Message-ID: <20090903171022.5f37eade@der-dieb> I do not want everything to log to a single file, the local facilities I would like to log to there own file and not be caught by a wildcard. Thanks. -Joe On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > Hello again, > > > > I am trying to log everything (*.*) to /var/log/syslog but > > local*.*. I tried a couple different ways to do this but didn't > > find a solution. Is this possible? I have a couple services I want > > to log to their own file rather than syslog, messages, etc. > > if you just do > > *.* /var/log/syslog > > that will write everything to that file > > I'm not sure what you are trying to say when you say 'but local*.*' > above. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:24:06 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:24:06 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903171022.5f37eade@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > I do not want everything to log to a single file, the local facilities > I would like to log to there own file and not be caught by a wildcard. ahh, ok, you cannot say local*.* you would have to list local0.*,local1.*,.. to cover them all there are 16 facility numbers, and by filtering out local0-local7 you are wanting to eliminate exactly half of them as such it's probably just as easy to list all the ones you want to record as it is to say *.* and subtract half of them. David Lang > Thanks. > -Joe > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > david at lang.hm wrote: > >> On Thu, 3 Sep 2009, Joe Williams wrote: >> >>> Hello again, >>> >>> I am trying to log everything (*.*) to /var/log/syslog but >>> local*.*. I tried a couple different ways to do this but didn't >>> find a solution. Is this possible? I have a couple services I want >>> to log to their own file rather than syslog, messages, etc. >> >> if you just do >> >> *.* /var/log/syslog >> >> that will write everything to that file >> >> I'm not sure what you are trying to say when you say 'but local*.*' >> above. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > > From joe at joetify.com Fri Sep 4 05:44:31 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 20:44:31 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: <20090903204431.5a4a2008@der-dieb> Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > > david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From DGillies at fairfaxdigital.com.au Fri Sep 4 05:51:38 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Fri, 4 Sep 2009 13:51:38 +1000 Subject: [rsyslog] logging wildcards In-Reply-To: <20090903204431.5a4a2008@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: I think something like this should work: if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != 'local2' ) then file3 David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital Level 2, 1 Darling Island Road Pyrmont NSW 2009 -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com Subject: Re: [rsyslog] logging wildcards Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From oribani at gmail.com Fri Sep 4 07:25:47 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:25:47 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <378058110909032225i10d9eb60g3039134f21eb2d64@mail.gmail.com> On 9/3/09, Ori Bani wrote: >>> I'm sorry if this isn't quite the right place to ask, since maybe no >>> one here created the RPM that's in the CentOS base repository. But I >>> am guessing people here have installed RPMs like this before and can >>> help anyway.... >>> >>> When I ask yum on CentOS 5 about rsyslog, I get this (note older >>> version - too bad): >>> >>> Available Packages >>> Name : rsyslog >>> Arch : i386 >>> Version: 2.0.6 >>> Release: 1.el5 >>> Size : 198 k >>> Repo : base >>> Summary: Enhanced system logging and kernel message trapping daemons >>> Description: >>> Rsyslog is an enhanced multi-threaded syslogd supporting, among >>> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >>> filtering on any message part, and fine grain output format control. >>> It is quite compatible to stock sysklogd and can be used as a drop- >>> in replacement. Its advanced features make it suitable for >>> enterprise-class, encryption protected syslog relay chains while at >>> the same time being very easy to setup for the novice user. >> >> I use Scientific Linux 5.x and because they are RHEL derivatives I see >> the >> same thing in the SL repo's. >> >> I have used the rsyslog from the repo's yet, all my rsyslog servers are >> based >> on EL4, but I'll try to help below. > > Thank you for your help. > >>> My questions are a little bit newbie... before I try installing >>> this, I want to know what it's going to do to my system: >>> >>> 1) Will it disable syslogd and/or klogd? Or will it add itself using >>> the "alternatives" paradigm so I can switch between them that way? >>> If neither, does it include startup scripts at all? If they are there >>> but not used by default, is there a recommended way to make the >>> switch and not really screw things up? >> >> You should try this on a test box. I haven't tried it but I think it >> should >> remove syslog RPM's from your installation and then install rsyslog. It >> should >> also make a /etc/syslog.conf.rpmsave file which you can reference for use >> in >> /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog I don't think that last line is needed; rsyslog is already added for you during the install process by yum. > And use yum to remove ksyslogd/syslog > >>> 2) Will it add itself to my cron jobs? Specifically, I don't mind >>> (for now) leaving the log rotation alone (don't let rsyslog manage my >>> rotations). If it adds itself to my cron jobs, does that mean it >>> will remove the logrotate cron job? >> >> Not sure sorry. You should grab the src.rpm file from CentOS, install it >> and >> take a look at the rsyslog.spec and it'll show you what it does on the >> post >> install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > >>> 2.5) If I keep using the old logrotate with rsyslog, will that create >>> any conflicts? >> >> I don't see how any conflicts will occur with logroate, since rsyslog >> basically logs to the same files that syslog logs to. It's meant to be a >> drop >> in replacement. >> >> Maybe specific questions about rsyslog with CentOS (or other derivatives) >> would actually be better in the CentOS or Scientific Linux mailing lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 > >>> Generally my aim is not to commit 100% to rsyslog yet, so I don't >>> want to get to a situation where it's a lot of work to get back to >>> the default syslog setup. > From oribani at gmail.com Fri Sep 4 07:21:08 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:21:08 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <20090821015920.M76525@npgx.com.au> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> Message-ID: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> one here created the RPM that's in the CentOS base repository. But I >> am guessing people here have installed RPMs like this before and can >> help anyway.... >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> version - too bad): >> >> Available Packages >> Name : rsyslog >> Arch : i386 >> Version: 2.0.6 >> Release: 1.el5 >> Size : 198 k >> Repo : base >> Summary: Enhanced system logging and kernel message trapping daemons >> Description: >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> filtering on any message part, and fine grain output format control. >> It is quite compatible to stock sysklogd and can be used as a drop- >> in replacement. Its advanced features make it suitable for >> enterprise-class, encryption protected syslog relay chains while at >> the same time being very easy to setup for the novice user. > > I use Scientific Linux 5.x and because they are RHEL derivatives I see the > same thing in the SL repo's. > > I have used the rsyslog from the repo's yet, all my rsyslog servers are > based > on EL4, but I'll try to help below. Thank you for your help. >> My questions are a little bit newbie... before I try installing >> this, I want to know what it's going to do to my system: >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself using >> the "alternatives" paradigm so I can switch between them that way? >> If neither, does it include startup scripts at all? If they are there >> but not used by default, is there a recommended way to make the >> switch and not really screw things up? > > You should try this on a test box. I haven't tried it but I think it should > remove syslog RPM's from your installation and then install rsyslog. It > should > also make a /etc/syslog.conf.rpmsave file which you can reference for use in > /etc/rsyslog.conf I wouldn't actually expect it to remove any other packages - I've never seen a yum installation remove something else - that seems like trouble. In fact, it turns out that it didn't do a thing to syslog/ksyslogd. It just installed itself in parallel (and it's up to you to turn it on). Everything is in place (startup scripts, config file that is a mirror of syslog.conf, etc.) and you just have to chkconfig syslog off chkconfig rsyslog on service syslog stop service rsyslog start I guess if you're going to be more permanent: chkconfig --del syslog chkconfig --add rsyslog And use yum to remove ksyslogd/syslog >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> (for now) leaving the log rotation alone (don't let rsyslog manage my >> rotations). If it adds itself to my cron jobs, does that mean it >> will remove the logrotate cron job? > > Not sure sorry. You should grab the src.rpm file from CentOS, install it and > take a look at the rsyslog.spec and it'll show you what it does on the post > install section. That's above my skill level. Instead I tried it out. It also adds itself to /etc/logrotate.d/syslog so you don't have to touch any of this. Here is the modified file: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true endscript } As you see, it left syslog there and added rsyslog. Because I have turned off syslog, this won't suddenly start it up, will it? >> 2.5) If I keep using the old logrotate with rsyslog, will that create >> any conflicts? > > I don't see how any conflicts will occur with logroate, since rsyslog > basically logs to the same files that syslog logs to. It's meant to be a > drop > in replacement. > > Maybe specific questions about rsyslog with CentOS (or other derivatives) > would actually be better in the CentOS or Scientific Linux mailing lists? I did, but it didn't help. That's disappointing. https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> want to get to a situation where it's a lot of work to get back to >> the default syslog setup. From rgerhards at hq.adiscon.com Fri Sep 4 12:51:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Sep 2009 12:51:21 +0200 Subject: [rsyslog] Need help with RPM(yum) version on CentOS References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com><20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> I have nothing technically to add to this discussion, but I would like remind you on the rsyslog wiki at http://wiki.rsyslog.com There already is one entry, but for an older version, not sure if that helps: http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story In any case, I would appreciate if you could share any knowledge you gain via the wiki. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ori Bani > Sent: Friday, September 04, 2009 7:21 AM > To: rsyslog-users > Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS > > >> I'm sorry if this isn't quite the right place to ask, since maybe no > >> one here created the RPM that's in the CentOS base repository. But > I > >> am guessing people here have installed RPMs like this before and can > >> help anyway.... > >> > >> When I ask yum on CentOS 5 about rsyslog, I get this (note older > >> version - too bad): > >> > >> Available Packages > >> Name : rsyslog > >> Arch : i386 > >> Version: 2.0.6 > >> Release: 1.el5 > >> Size : 198 k > >> Repo : base > >> Summary: Enhanced system logging and kernel message trapping daemons > >> Description: > >> Rsyslog is an enhanced multi-threaded syslogd supporting, among > >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, > >> filtering on any message part, and fine grain output format control. > >> It is quite compatible to stock sysklogd and can be used as a drop- > >> in replacement. Its advanced features make it suitable for > >> enterprise-class, encryption protected syslog relay chains while at > >> the same time being very easy to setup for the novice user. > > > > I use Scientific Linux 5.x and because they are RHEL derivatives I > see the > > same thing in the SL repo's. > > > > I have used the rsyslog from the repo's yet, all my rsyslog servers > are > > based > > on EL4, but I'll try to help below. > > Thank you for your help. > > >> My questions are a little bit newbie... before I try installing > >> this, I want to know what it's going to do to my system: > >> > >> 1) Will it disable syslogd and/or klogd? Or will it add itself > using > >> the "alternatives" paradigm so I can switch between them that way? > >> If neither, does it include startup scripts at all? If they are > there > >> but not used by default, is there a recommended way to make the > >> switch and not really screw things up? > > > > You should try this on a test box. I haven't tried it but I think it > should > > remove syslog RPM's from your installation and then install rsyslog. > It > > should > > also make a /etc/syslog.conf.rpmsave file which you can reference for > use in > > /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog > > And use yum to remove ksyslogd/syslog > > >> 2) Will it add itself to my cron jobs? Specifically, I don't mind > >> (for now) leaving the log rotation alone (don't let rsyslog manage > my > >> rotations). If it adds itself to my cron jobs, does that mean it > >> will remove the logrotate cron job? > > > > Not sure sorry. You should grab the src.rpm file from CentOS, install > it and > > take a look at the rsyslog.spec and it'll show you what it does on > the post > > install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > > >> 2.5) If I keep using the old logrotate with rsyslog, will that > create > >> any conflicts? > > > > I don't see how any conflicts will occur with logroate, since rsyslog > > basically logs to the same files that syslog logs to. It's meant to > be a > > drop > > in replacement. > > > > Maybe specific questions about rsyslog with CentOS (or other > derivatives) > > would actually be better in the CentOS or Scientific Linux mailing > lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start > =0#forumpost83694 > > >> Generally my aim is not to commit 100% to rsyslog yet, so I don't > >> want to get to a situation where it's a lot of work to get back to > >> the default syslog setup. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Fri Sep 4 21:25:30 2009 From: henry78 at gmx.at (Henry) Date: Fri, 04 Sep 2009 21:25:30 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message Message-ID: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Hi! This puzzels me: This is my tcprecieve config file for rsyslog v4 on ubuntu: -----8<----- $ModLoad imtcp $InputTCPServerRun 514 # some dynamic templates $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" # log remote local1 to dynamic diretory if $fromhost-ip != '127.0.0.1' and \ $syslogfacility-text == 'local1' \ then -?DYNlocal1 ----->8----- I created /var/log/remote with sufficient privileges. Unfortunately this doesn't work. rsyslog crates a folder named after the remote host (myhostname) and creates the file local1.log (again: sufficient permissions: syslog:syslog 640). But it doesn't write to that file, but logs the error: -----8<----- Could not open dynamic file '/var/log/remote/myhostname/local1.log' - discarding message ----->8----- As you might guess my question is: Why isn't rsyslog able to open a file it is able to create? Any help or hint is really appreciated. -- kind regards, Henry From joe at joetify.com Fri Sep 4 21:33:17 2009 From: joe at joetify.com (Joe Williams) Date: Fri, 4 Sep 2009 12:33:17 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: <20090904123317.172e0ca4@der-dieb> Thanks David, that ended up working after changing the "or" to an "and". Also I ended up finding a good example of this sort of configuration at http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory -Joe On Fri, 4 Sep 2009 13:51:38 +1000 David Gillies wrote: > > I think something like this should work: > > if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != > 'local2' ) then file3 > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > > Fairfax Digital > Level 2, 1 Darling Island Road > Pyrmont NSW 2009 > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams > Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] logging wildcards > > > Sorry I think we are misunderstanding each other. What I am wanting > to do is this: > > ### > > local0.* FILE1 > local2.* FILE2 > > *.* (but not local0.* or local2.*) FILE3 > > ### > > Is that possible? > > Thanks again. > -Joe > > > > On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) > david at lang.hm wrote: > > > On Thu, 3 Sep 2009, Joe Williams wrote: > > > > > I do not want everything to log to a single file, the local > > > facilities I would like to log to there own file and not be > > > caught by a wildcard. > > > > ahh, ok, you cannot say local*.* you would have to list > > local0.*,local1.*,.. to cover them all > > > > there are 16 facility numbers, and by filtering out local0-local7 > > you are wanting to eliminate exactly half of them > > > > as such it's probably just as easy to list all the ones you want to > > record as it is to say *.* and subtract half of them. > > > > David Lang > > > > > Thanks. > > > -Joe > > > > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > > >> > > >>> Hello again, > > >>> > > >>> I am trying to log everything (*.*) to /var/log/syslog but > > >>> local*.*. I tried a couple different ways to do this but didn't > > >>> find a solution. Is this possible? I have a couple services I > > >>> want to log to their own file rather than syslog, messages, etc. > > >> > > >> if you just do > > >> > > >> *.* /var/log/syslog > > >> > > >> that will write everything to that file > > >> > > >> I'm not sure what you are trying to say when you say 'but > > >> local*.*' above. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -- > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended > recipient, any use, dissemination, reliance, forwarding, printing or > copying of this e-mail or any attached files is unauthorised. This > e-mail is subject to copyright. No part of it should be reproduced, > adapted or communicated without the written consent of the copyright > owner. If you have received this e-mail in error please advise the > sender immediately by return e-mail or telephone and delete all > copies. Fairfax does not guarantee the accuracy or completeness of > any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept > legal responsibility for the contents of this message or attached > files. _______________________________________________ rsyslog > mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Sat Sep 5 04:04:59 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 19:04:59 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file Message-ID: I ahve a config file that fixes up broken syslog messages that has the following $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8;TraditionalForwardFormat the upstream box is seeing things as I would expect, but the local /var/log/messages file is not is it incorrect to have two entries that both write to /var/log/messages? David Lang From david at lang.hm Sat Sep 5 08:03:23 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 23:03:23 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file In-Reply-To: References: Message-ID: On Fri, 4 Sep 2009, david at lang.hm wrote: > I ahve a config file that fixes up broken syslog messages that has the > following > > $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" > :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat > & @192.168.210.8;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.210.8;TraditionalForwardFormat > > > the upstream box is seeing things as I would expect, but the local > /var/log/messages file is not > > is it incorrect to have two entries that both write to /var/log/messages? never mind, I just spotted the extra *.* in there (nothing was reported when starting up) David Lang From oribani at gmail.com Sun Sep 6 03:52:04 2009 From: oribani at gmail.com (Ori Bani) Date: Sat, 5 Sep 2009 18:52:04 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> Message-ID: <378058110909051852t1ae1f4dgd7f00d830b3b6284@mail.gmail.com> On 9/4/09, Rainer Gerhards wrote: > I have nothing technically to add to this discussion, but I would like > remind > you on the rsyslog wiki at > > http://wiki.rsyslog.com > > There already is one entry, but for an older version, not sure if that > helps: > > http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story > > In any case, I would appreciate if you could share any knowledge you gain > via > the wiki. I added my info, but that page was designed by someone who assumes you aren't using yum (or any similar system), so I hope I added it in an acceptable way >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Ori Bani >> Sent: Friday, September 04, 2009 7:21 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS >> >> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> >> one here created the RPM that's in the CentOS base repository. But >> I >> >> am guessing people here have installed RPMs like this before and can >> >> help anyway.... >> >> >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> >> version - too bad): >> >> >> >> Available Packages >> >> Name : rsyslog >> >> Arch : i386 >> >> Version: 2.0.6 >> >> Release: 1.el5 >> >> Size : 198 k >> >> Repo : base >> >> Summary: Enhanced system logging and kernel message trapping daemons >> >> Description: >> >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> >> filtering on any message part, and fine grain output format control. >> >> It is quite compatible to stock sysklogd and can be used as a drop- >> >> in replacement. Its advanced features make it suitable for >> >> enterprise-class, encryption protected syslog relay chains while at >> >> the same time being very easy to setup for the novice user. >> > >> > I use Scientific Linux 5.x and because they are RHEL derivatives I >> see the >> > same thing in the SL repo's. >> > >> > I have used the rsyslog from the repo's yet, all my rsyslog servers >> are >> > based >> > on EL4, but I'll try to help below. >> >> Thank you for your help. >> >> >> My questions are a little bit newbie... before I try installing >> >> this, I want to know what it's going to do to my system: >> >> >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself >> using >> >> the "alternatives" paradigm so I can switch between them that way? >> >> If neither, does it include startup scripts at all? If they are >> there >> >> but not used by default, is there a recommended way to make the >> >> switch and not really screw things up? >> > >> > You should try this on a test box. I haven't tried it but I think it >> should >> > remove syslog RPM's from your installation and then install rsyslog. >> It >> > should >> > also make a /etc/syslog.conf.rpmsave file which you can reference for >> use in >> > /etc/rsyslog.conf >> >> I wouldn't actually expect it to remove any other packages - I've >> never seen a yum installation remove something else - that seems like >> trouble. In fact, it turns out that it didn't do a thing to >> syslog/ksyslogd. It just installed itself in parallel (and it's up to >> you to turn it on). Everything is in place (startup scripts, config >> file that is a mirror of syslog.conf, etc.) and you just have to >> >> chkconfig syslog off >> chkconfig rsyslog on >> service syslog stop >> service rsyslog start >> >> I guess if you're going to be more permanent: >> >> chkconfig --del syslog >> chkconfig --add rsyslog >> >> And use yum to remove ksyslogd/syslog >> >> >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> >> (for now) leaving the log rotation alone (don't let rsyslog manage >> my >> >> rotations). If it adds itself to my cron jobs, does that mean it >> >> will remove the logrotate cron job? >> > >> > Not sure sorry. You should grab the src.rpm file from CentOS, install >> it and >> > take a look at the rsyslog.spec and it'll show you what it does on >> the post >> > install section. >> >> That's above my skill level. Instead I tried it out. It also adds >> itself to /etc/logrotate.d/syslog so you don't have to touch any of >> this. Here is the modified file: >> >> /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler >> /var/log/boot.log /var/log/cron { >> sharedscripts >> postrotate >> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> >> /dev/null || true >> /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> >> /dev/null || true >> endscript >> } >> >> As you see, it left syslog there and added rsyslog. Because I have >> turned off syslog, this won't suddenly start it up, will it? >> >> >> 2.5) If I keep using the old logrotate with rsyslog, will that >> create >> >> any conflicts? >> > >> > I don't see how any conflicts will occur with logroate, since rsyslog >> > basically logs to the same files that syslog logs to. It's meant to >> be a >> > drop >> > in replacement. >> > >> > Maybe specific questions about rsyslog with CentOS (or other >> derivatives) >> > would actually be better in the CentOS or Scientific Linux mailing >> lists? >> >> I did, but it didn't help. That's disappointing. >> >> https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start >> =0#forumpost83694 >> >> >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> >> want to get to a situation where it's a lot of work to get back to >> >> the default syslog setup. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 20:17:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 13:17:50 -0500 Subject: [rsyslog] syslog server and reports Message-ID: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> I have some debian lenny servers sending their logs (via TCP) to a central rsyslog server. Every remote servers has at /etc/rsyslog.conf: *.* @@IP_CENTRAL_SERVER So, I can see in the central syslog server all logs without problems. I'm looking for a single and simple report, like logwatch for example who process all logs and send me in ONE mail or on ONE html page all resume info of all logs. I tried with logwatch and I didn't get this report I'm looking for. My question is? Is there any tool, script, app, etc which I run on the syslog server and give me the information of all servers in a way as simple as possible? Maybe in a single resume mail separated by a line for example? Thanks for your time. -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Sun Sep 6 21:00:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 6 Sep 2009 21:00:46 +0200 Subject: [rsyslog] syslog server and reports References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Probably not exactly what you look for, but maybe worth a try: http://www.phplogcon.org More reporting featueres are being tackled in the next couple of weeks. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Sunday, September 06, 2009 8:18 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] syslog server and reports > > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? > > Thanks for your time. > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 21:20:34 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 14:20:34 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Message-ID: <194a2c240909061220k25cf03e4ycb7dcf379d45ab8f@mail.gmail.com> Hi Rainer, thanks for your soon answer.. On 9/6/09, Rainer Gerhards wrote: > Probably not exactly what you look for, but maybe worth a try: > > http://www.phplogcon.org I have installed phplogcon but, it's not whay I'm looking for. I need an email, a simple daily email with the reports of all my servers. I've tried to setup logwatch and logcheck but I could not get what I want. regards, Israel. > > More reporting featueres are being tackled in the next couple of weeks. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia >> Sent: Sunday, September 06, 2009 8:18 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] syslog server and reports >> >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? >> >> Thanks for your time. >> >> -- >> Regards; >> Israel Garcia >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Sun Sep 6 23:19:35 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 14:19:35 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? there are a lot of products and projects out there to analyse logs and generate reports. the problem is that what I am interested in seeing in a report may or may not match what you are interested in seeing. also, most of this effort is taking place within originizations that have large volumes of logs, so distilling it down to a single report or e-mail requires that a lot of detail gets left out (and that goes back to exactly what you are interested in seeing) when you say you want one page that shows you 'everything', what is it that you want to see? are there particular messages that you want to see if they show up even once? or are you interested in simplifying log messages into categories and seeing how many messages in each category you have. do you only care about the logs showing up sometime during the day? or are you interested in the trending of how many logs you get each second throughout the day (or anything in between) unfortunantly the result of all these questions probably means that you will need to customize whatever you use to exactly the report that you want. large companies can spend millions of dollars on systems and software to alert, report, and query their logs. I am currently getting ~300M log messages/day and I distill it down to a single e-mail report that I look at (and generate additional reports with subsets of the data for other people to look at). the best advice I ever got was to use the approach termed 'artificial ignorance' start off with all your logs for any log type that you can categorize create a summary of that log type (even if it's an unimportant log, count it because the number of times an unimportant thing happens can be important) look at what's left and repeat the process after several iterations of this you end up with the vast majority of your logs summarized and a report of "what's left", any new messages that you have never seen before (which usually mean they are important) show up in the "what's left" bucket and tend to stand out you do need to keep on top of this, upgrades to systems, new installs, etc cause new logs to show up, if you categorize and summarize them your final report stays small, if you let things slide for several months the final report can end up very large (and therefor useless) David Lang From igalvarez at gmail.com Mon Sep 7 01:40:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 18:40:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? > > there are a lot of products and projects out there to analyse logs and > generate reports. > > the problem is that what I am interested in seeing in a report may or may > not match what you are interested in seeing. > > also, most of this effort is taking place within originizations that have > large volumes of logs, so distilling it down to a single report or e-mail > requires that a lot of detail gets left out (and that goes back to exactly > what you are interested in seeing) > > when you say you want one page that shows you 'everything', what is it > that you want to see? Hi, David I mean, a report like logwatch use to send me everyday from each server. As I said before, I'm collecting all servers logs (syslog and auth.log) into my central syslog, so I need some tool like logwatch running on the collector which send in one mail or in one html page. . I tried to configure logwatch in the collector without sucess. That's what I need. :-) thanks. regards, Israel > > are there particular messages that you want to see if they show up even > once? or are you interested in simplifying log messages into categories > and seeing how many messages in each category you have. > > do you only care about the logs showing up sometime during the day? or are > you interested in the trending of how many logs you get each second > throughout the day (or anything in between) > > unfortunantly the result of all these questions probably means that you > will need to customize whatever you use to exactly the report that you > want. > > large companies can spend millions of dollars on systems and software to > alert, report, and query their logs. > > I am currently getting ~300M log messages/day and I distill it down to a > single e-mail report that I look at (and generate additional reports with > subsets of the data for other people to look at). > > > the best advice I ever got was to use the approach termed 'artificial > ignorance' > > start off with all your logs > > for any log type that you can categorize create a summary of that log type > (even if it's an unimportant log, count it because the number of times an > unimportant thing happens can be important) > > look at what's left and repeat the process > > after several iterations of this you end up with the vast majority of your > logs summarized and a report of "what's left", any new messages that you > have never seen before (which usually mean they are important) show up in > the "what's left" bucket and tend to stand out > > you do need to keep on top of this, upgrades to systems, new installs, > etc cause new logs to show up, if you categorize and summarize them your > final report stays small, if you let things slide for several months the > final report can end up very large (and therefor useless) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 02:15:40 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 17:15:40 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> I have some debian lenny servers sending their logs (via TCP) to a >>> central rsyslog server. >>> Every remote servers has at /etc/rsyslog.conf: >>> >>> *.* @@IP_CENTRAL_SERVER >>> >>> So, I can see in the central syslog server all logs without problems. >>> I'm looking for a single and simple report, like logwatch for example >>> who process all logs and send me in ONE mail or on ONE html page all >>> resume info of all logs. I tried with logwatch and I didn't get this >>> report I'm looking for. >>> >>> My question is? >>> Is there any tool, script, app, etc which I run on the syslog server >>> and give me the information of all servers in a way as simple as >>> possible? Maybe in a single resume mail separated by a line for >>> example? >> >> there are a lot of products and projects out there to analyse logs and >> generate reports. >> >> the problem is that what I am interested in seeing in a report may or may >> not match what you are interested in seeing. >> >> also, most of this effort is taking place within originizations that have >> large volumes of logs, so distilling it down to a single report or e-mail >> requires that a lot of detail gets left out (and that goes back to exactly >> what you are interested in seeing) >> >> when you say you want one page that shows you 'everything', what is it >> that you want to see? > Hi, David > I mean, a report like logwatch use to send me everyday from each > server. As I said before, I'm collecting all servers logs (syslog and > auth.log) into my central syslog, so I need some tool like logwatch > running on the collector which send in one mail or in one html page. > . > I tried to configure logwatch in the collector without sucess. > > That's what I need. :-) ok, so you want the report that you get from logwatch, that simplifies things. when you say you can't get it to work on the collector box, more info is needed. does logwatch give you the info that you want about the collector box? do you put the logs from all servers in one file? or do you split them by host? (or split them in other ways) how does logwatch fail? does it crash? give you incorrect information? other? David Lang From igalvarez at gmail.com Mon Sep 7 03:47:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 20:47:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> I have some debian lenny servers sending their logs (via TCP) to a >>>> central rsyslog server. >>>> Every remote servers has at /etc/rsyslog.conf: >>>> >>>> *.* @@IP_CENTRAL_SERVER >>>> >>>> So, I can see in the central syslog server all logs without problems. >>>> I'm looking for a single and simple report, like logwatch for example >>>> who process all logs and send me in ONE mail or on ONE html page all >>>> resume info of all logs. I tried with logwatch and I didn't get this >>>> report I'm looking for. >>>> >>>> My question is? >>>> Is there any tool, script, app, etc which I run on the syslog server >>>> and give me the information of all servers in a way as simple as >>>> possible? Maybe in a single resume mail separated by a line for >>>> example? >>> >>> there are a lot of products and projects out there to analyse logs and >>> generate reports. >>> >>> the problem is that what I am interested in seeing in a report may or may >>> not match what you are interested in seeing. >>> >>> also, most of this effort is taking place within originizations that have >>> large volumes of logs, so distilling it down to a single report or e-mail >>> requires that a lot of detail gets left out (and that goes back to >>> exactly >>> what you are interested in seeing) >>> >>> when you say you want one page that shows you 'everything', what is it >>> that you want to see? >> Hi, David >> I mean, a report like logwatch use to send me everyday from each >> server. As I said before, I'm collecting all servers logs (syslog and >> auth.log) into my central syslog, so I need some tool like logwatch >> running on the collector which send in one mail or in one html page. >> . >> I tried to configure logwatch in the collector without sucess. >> >> That's what I need. :-) > > ok, so you want the report that you get from logwatch, that simplifies > things. > > when you say you can't get it to work on the collector box, more info is > needed. > > does logwatch give you the info that you want about the collector box? My scenario: I added this two lines in /etc/rsyslog.conf of all exporting servers: auth,authpriv.* @@xx.xx.xx.xx *.*;auth,authpriv.none @@xx.xx.xx.xx In the collector syslog and auth.log files I see logs coming from those servers. logwatch.conf file is the default. I run logwatch (testing mode) in the collector and it merge logs from all servers, so you can not identify which log output is belongs to. It looks like all logs are from the collector server. here you can see a part of logwatch output: In my case deb2 is the hostname of the collector and debian is the hostname of one exporter. deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) #################### Processing Initiated: Sun Sep 6 21:35:29 2009 Date Range Processed: today ( 2009-Sep-06 ) Period is day. Detail Level of Output: 0 Type of Output/Format: stdout / text Logfiles for Host: deb2 ################################################################## ###This logs are from deb2 Installed: libdate-manip-perl 5.54-1 lockfile-progs 0.1.11-0.1 logtail 1.2.69 logwatch 7.3.6.cvs20080702-2 postfix 2.5.5-1.1 . . . . . --------------------- pam_unix Begin ------------------------ ### All this logs entries from user test123 are from one exporter server (debian). sshd: Authentication Failures: root (localhost): 1 Time(s) su: Authentication Failures: test123(1003) -> root: 2 Time(s) Sessions Opened: root -> logcheck: 17 Time(s) root -> root: 9 Time(s) sudo: Authentication Failures: test123(0) -> test123: 1 Time(s) **Unmatched Entries** useradd: failed adding user `test', data deleted: 1 Time(s) ---------------------- Connections (secure-log) End ------------------------- ============================================================================== ### This is from exporter debian server. test123 => root --------------- /bin/su - 1 Times. ---------------------- Sudo (secure-log) End ------------------------- ## This df output is from deb2 (collector) --------------------- Disk Space Begin ------------------------ Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.5G 2.0G 5.2G 28% / ---------------------- Disk Space End ------------------------- ###################### Logwatch End ################### As you can see, it seems like the report belongs to deb2 server and it's not. I'd be happy if at least logwatch put some tags at the beginning of each line to identify the source. thanks again. regards, israel. > > do you put the logs from all servers in one file? or do you split them by > host? (or split them in other ways) > > how does logwatch fail? does it crash? give you incorrect information? > other? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 04:23:51 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 19:23:51 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> On 9/6/09, david at lang.hm wrote: >>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>> >>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>> central rsyslog server. >>>>> Every remote servers has at /etc/rsyslog.conf: >>>>> >>>>> *.* @@IP_CENTRAL_SERVER >>>>> >>>>> So, I can see in the central syslog server all logs without problems. >>>>> I'm looking for a single and simple report, like logwatch for example >>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>> report I'm looking for. >>>>> >>>>> My question is? >>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>> and give me the information of all servers in a way as simple as >>>>> possible? Maybe in a single resume mail separated by a line for >>>>> example? >>>> >>>> there are a lot of products and projects out there to analyse logs and >>>> generate reports. >>>> >>>> the problem is that what I am interested in seeing in a report may or may >>>> not match what you are interested in seeing. >>>> >>>> also, most of this effort is taking place within originizations that have >>>> large volumes of logs, so distilling it down to a single report or e-mail >>>> requires that a lot of detail gets left out (and that goes back to >>>> exactly >>>> what you are interested in seeing) >>>> >>>> when you say you want one page that shows you 'everything', what is it >>>> that you want to see? >>> Hi, David >>> I mean, a report like logwatch use to send me everyday from each >>> server. As I said before, I'm collecting all servers logs (syslog and >>> auth.log) into my central syslog, so I need some tool like logwatch >>> running on the collector which send in one mail or in one html page. >>> . >>> I tried to configure logwatch in the collector without sucess. >>> >>> That's what I need. :-) >> >> ok, so you want the report that you get from logwatch, that simplifies >> things. >> >> when you say you can't get it to work on the collector box, more info is >> needed. >> >> does logwatch give you the info that you want about the collector box? > > My scenario: > I added this two lines in /etc/rsyslog.conf of all exporting servers: > > auth,authpriv.* @@xx.xx.xx.xx > *.*;auth,authpriv.none @@xx.xx.xx.xx > > In the collector syslog and auth.log files I see logs coming from > those servers. > > logwatch.conf file is the default. > > I run logwatch (testing mode) in the collector and it merge logs from > all servers, so you can not identify which log output is belongs to. > It looks like all logs are from the collector server. ahh, that's the problem. unforutnantly fixing this would take some significant surgury to logwatch. it assumes that all the logs it is dealing with are from the local box and therefor it ignores the server tag in the output. you could use the rsyslog dynafiles feature to create a different file for each server, run logwatch against each of those files, and then combine the reports (including adding text to tell you which server is up next) David Lang > here you can see a part of logwatch output: > > In my case deb2 is the hostname of the collector and debian is the > hostname of one exporter. > > deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today > > ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) > #################### > Processing Initiated: Sun Sep 6 21:35:29 2009 > Date Range Processed: today > ( 2009-Sep-06 ) > Period is day. > Detail Level of Output: 0 > Type of Output/Format: stdout / text > Logfiles for Host: deb2 > ################################################################## > > ###This logs are from deb2 > Installed: > libdate-manip-perl 5.54-1 > lockfile-progs 0.1.11-0.1 > logtail 1.2.69 > logwatch 7.3.6.cvs20080702-2 > postfix 2.5.5-1.1 > . > . > . > . > . > --------------------- pam_unix Begin ------------------------ > ### All this logs entries from user test123 are from one exporter > server (debian). > sshd: > Authentication Failures: > root (localhost): 1 Time(s) > > su: > Authentication Failures: > test123(1003) -> root: 2 Time(s) > Sessions Opened: > root -> logcheck: 17 Time(s) > root -> root: 9 Time(s) > > sudo: > Authentication Failures: > test123(0) -> test123: 1 Time(s) > > **Unmatched Entries** > useradd: failed adding user `test', data deleted: 1 Time(s) > > ---------------------- Connections (secure-log) End ------------------------- > > > ============================================================================== > ### This is from exporter debian server. > test123 => root > --------------- > /bin/su - 1 Times. > > ---------------------- Sudo (secure-log) End ------------------------- > > ## This df output is from deb2 (collector) > --------------------- Disk Space Begin ------------------------ > > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 7.5G 2.0G 5.2G 28% / > > ---------------------- Disk Space End ------------------------- > > ###################### Logwatch End ################### > > As you can see, it seems like the report belongs to deb2 server and it's not. > > I'd be happy if at least logwatch put some tags at the beginning of > each line to identify the source. > > thanks again. > regards, > israel. > > > > > >> >> do you put the logs from all servers in one file? or do you split them by >> host? (or split them in other ways) >> >> how does logwatch fail? does it crash? give you incorrect information? >> other? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > From igalvarez at gmail.com Mon Sep 7 04:46:41 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 21:46:41 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> On 9/6/09, david at lang.hm wrote: >>>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>>> >>>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>>> central rsyslog server. >>>>>> Every remote servers has at /etc/rsyslog.conf: >>>>>> >>>>>> *.* @@IP_CENTRAL_SERVER >>>>>> >>>>>> So, I can see in the central syslog server all logs without problems. >>>>>> I'm looking for a single and simple report, like logwatch for example >>>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>>> report I'm looking for. >>>>>> >>>>>> My question is? >>>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>>> and give me the information of all servers in a way as simple as >>>>>> possible? Maybe in a single resume mail separated by a line for >>>>>> example? >>>>> >>>>> there are a lot of products and projects out there to analyse logs and >>>>> generate reports. >>>>> >>>>> the problem is that what I am interested in seeing in a report may or >>>>> may >>>>> not match what you are interested in seeing. >>>>> >>>>> also, most of this effort is taking place within originizations that >>>>> have >>>>> large volumes of logs, so distilling it down to a single report or >>>>> e-mail >>>>> requires that a lot of detail gets left out (and that goes back to >>>>> exactly >>>>> what you are interested in seeing) >>>>> >>>>> when you say you want one page that shows you 'everything', what is it >>>>> that you want to see? >>>> Hi, David >>>> I mean, a report like logwatch use to send me everyday from each >>>> server. As I said before, I'm collecting all servers logs (syslog and >>>> auth.log) into my central syslog, so I need some tool like logwatch >>>> running on the collector which send in one mail or in one html page. >>>> . >>>> I tried to configure logwatch in the collector without sucess. >>>> >>>> That's what I need. :-) >>> >>> ok, so you want the report that you get from logwatch, that simplifies >>> things. >>> >>> when you say you can't get it to work on the collector box, more info is >>> needed. >>> >>> does logwatch give you the info that you want about the collector box? >> >> My scenario: >> I added this two lines in /etc/rsyslog.conf of all exporting servers: >> >> auth,authpriv.* @@xx.xx.xx.xx >> *.*;auth,authpriv.none @@xx.xx.xx.xx >> >> In the collector syslog and auth.log files I see logs coming from >> those servers. >> >> logwatch.conf file is the default. >> >> I run logwatch (testing mode) in the collector and it merge logs from >> all servers, so you can not identify which log output is belongs to. >> It looks like all logs are from the collector server. > > ahh, that's the problem. > > unforutnantly fixing this would take some significant surgury to logwatch. > it assumes that all the logs it is dealing with are from the local box and > therefor it ignores the server tag in the output. > > you could use the rsyslog dynafiles feature to create a different file for > each server, run logwatch against each of those files, and then combine > the reports (including adding text to tell you which server is up next) Hi David, I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. > > David Lang > >> here you can see a part of logwatch output: >> >> In my case deb2 is the hostname of the collector and debian is the >> hostname of one exporter. >> >> deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today >> >> ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) >> #################### >> Processing Initiated: Sun Sep 6 21:35:29 2009 >> Date Range Processed: today >> ( 2009-Sep-06 ) >> Period is day. >> Detail Level of Output: 0 >> Type of Output/Format: stdout / text >> Logfiles for Host: deb2 >> ################################################################## >> >> ###This logs are from deb2 >> Installed: >> libdate-manip-perl 5.54-1 >> lockfile-progs 0.1.11-0.1 >> logtail 1.2.69 >> logwatch 7.3.6.cvs20080702-2 >> postfix 2.5.5-1.1 >> . >> . >> . >> . >> . >> --------------------- pam_unix Begin ------------------------ >> ### All this logs entries from user test123 are from one exporter >> server (debian). >> sshd: >> Authentication Failures: >> root (localhost): 1 Time(s) >> >> su: >> Authentication Failures: >> test123(1003) -> root: 2 Time(s) >> Sessions Opened: >> root -> logcheck: 17 Time(s) >> root -> root: 9 Time(s) >> >> sudo: >> Authentication Failures: >> test123(0) -> test123: 1 Time(s) >> >> **Unmatched Entries** >> useradd: failed adding user `test', data deleted: 1 Time(s) >> >> ---------------------- Connections (secure-log) End >> ------------------------- >> >> >> ============================================================================== >> ### This is from exporter debian server. >> test123 => root >> --------------- >> /bin/su - 1 Times. >> >> ---------------------- Sudo (secure-log) End ------------------------- >> >> ## This df output is from deb2 (collector) >> --------------------- Disk Space Begin ------------------------ >> >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda1 7.5G 2.0G 5.2G 28% / >> >> ---------------------- Disk Space End ------------------------- >> >> ###################### Logwatch End ################### >> >> As you can see, it seems like the report belongs to deb2 server and it's >> not. >> >> I'd be happy if at least logwatch put some tags at the beginning of >> each line to identify the source. >> >> thanks again. >> regards, >> israel. >> >> >> >> >> >>> >>> do you put the logs from all servers in one file? or do you split them by >>> host? (or split them in other ways) >>> >>> how does logwatch fail? does it crash? give you incorrect information? >>> other? >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Mon Sep 7 15:18:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Sep 2009 15:18:13 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE1F@GRFEXC.intern.adiscon.com> Hi all, after some struggle, a new status: Thanks to David's data sets, I think I have finally been able to find a code spot that may be troublesome. It also is in an area that we already had under suspicion. While it is too early to say if I finally found the issue, it looks very promising. If I am right, the problem is actually environment-induced, what would also explain why other users did not yet report anything and I did not see anything in my lab so far. The ultimate root cause may even be a formatting error in another rsyslogd instance further up in the relay chain. If so, I'll try to work upward from where to problem currently occurs to the above-it root cause. I just thought I share this new information with you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, September 03, 2009 12:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > Hi David, > > > I haven't gone back to the 3.x series, but I did several more runs > with > > 4.2.0 doing the folloiwng > > > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd > -r > > -h ; mv /core /core-4.2.0-10 > > > > I have several complete steps, as well as several partial sets of > data. > > I > > will gzip them and attempt to send them to you directly. > > Thanks for the data set, I am right now working on it. Unfortunately, > as I > feared, the core files do not really help. There is a big mismatch > between > your system environment and mine, and so gdb is not able to extract any > useful information. All I see is that there are six threads in the > system, > and the rest is almost only question marks. > > So it would be great if you could issue the gdb commands in your > environment > and let me know the outcome. > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From DGillies at fairfaxdigital.com.au Tue Sep 8 02:15:29 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 8 Sep 2009 10:15:29 +1000 Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: Hi Israel, Its been a while since I last used it, but I'm pretty sure that epylog can handle reporting on log files with multiple hosts: https://fedorahosted.org/epylog/ David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia Sent: Monday, 7 September 2009 12:47 PM To: rsyslog-users Subject: Re: [rsyslog] syslog server and reports I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From igalvarez at gmail.com Tue Sep 8 05:19:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Mon, 7 Sep 2009 22:19:50 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: <194a2c240909072019q618594cay38b7ca7202d13b10@mail.gmail.com> On 9/7/09, David Gillies wrote: > Hi Israel, Hi David, > > Its been a while since I last used it, but I'm pretty sure that epylog can > handle reporting on log files with multiple hosts: > > https://fedorahosted.org/epylog/ umm... sounds good.. I see my rsyslog collector has the latest version of epylog..I'll try right now..:-) thanks regards, israel. > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Monday, 7 September 2009 12:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] syslog server and reports > > > > I'll try this way.. but do you know if there another tool more simple to get > jmy report? > thanks in advance. > > regards, > israel. > The information contained in this e-mail message and any accompanying files > is or may be confidential. If you are not the intended recipient, any use, > dissemination, reliance, forwarding, printing or copying of this e-mail or > any attached files is unauthorised. This e-mail is subject to copyright. No > part of it should be reproduced, adapted or communicated without the written > consent of the copyright owner. If you have received this e-mail in error > please advise the sender immediately by return e-mail or telephone and > delete all copies. Fairfax does not guarantee the accuracy or completeness > of any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept legal > responsibility for the contents of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From henry78 at gmx.at Tue Sep 8 09:03:49 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 09:03:49 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <1252092330.924.24.camel@eberhe.office.chipkarte.at> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Message-ID: <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Hello! Tried it with various log locations (e.g. /tmp/my.log), neither worked. Is this worth ab bug? -- kind regards, Henry On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > Hi! > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > ubuntu: > > -----8<----- > $ModLoad imtcp > $InputTCPServerRun 514 > > # some dynamic templates > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > # log remote local1 to dynamic diretory > if $fromhost-ip != '127.0.0.1' and \ > $syslogfacility-text == 'local1' \ > then -?DYNlocal1 > ----->8----- > > I created /var/log/remote with sufficient privileges. > > Unfortunately this doesn't work. rsyslog crates a folder named after the > remote host (myhostname) and creates the file local1.log (again: > sufficient permissions: syslog:syslog 640). But it doesn't write to that > file, but logs the error: > > -----8<----- > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > discarding message > ----->8----- > > As you might guess my question is: Why isn't rsyslog able to open a file > it is able to create? Any help or hint is really appreciated. > From rgerhards at hq.adiscon.com Tue Sep 8 09:55:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 09:55:21 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> can you provide a debug log? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 9:04 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > Hello! > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > Is this worth ab bug? > > -- > kind regards, Henry > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > Hi! > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > ubuntu: > > > > -----8<----- > > $ModLoad imtcp > > $InputTCPServerRun 514 > > > > # some dynamic templates > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > # log remote local1 to dynamic diretory > > if $fromhost-ip != '127.0.0.1' and \ > > $syslogfacility-text == 'local1' \ > > then -?DYNlocal1 > > ----->8----- > > > > I created /var/log/remote with sufficient privileges. > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > the > > remote host (myhostname) and creates the file local1.log (again: > > sufficient permissions: syslog:syslog 640). But it doesn't write to > that > > file, but logs the error: > > > > -----8<----- > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > discarding message > > ----->8----- > > > > As you might guess my question is: Why isn't rsyslog able to open a > file > > it is able to create? Any help or hint is really appreciated. > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:30:17 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Hi, I got the debug log, it was too big to be sent via the list (but I got it as list admin). I see that you drop privileges to the user "syslog". This probably explains what happens. I think the file is created before you drop privileges, but can then no longer be written when running in the new security context. Could you verify that the user "syslog" can access this file? Also, could you temporarily remove the Privilege drop? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 9:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Tue Sep 8 12:31:35 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:31:35 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <1252405895.17741.41.camel@eberhe.office.chipkarte.at> Hmm... a simple '-d' debug doesn't seem to give enough information, see attached rsyslogd.debug.full. Attached log starts with processing of the remote logging because the full log is too large for this list. Note, that this was started with an empty /var/log/remote and the file /var/log/remote/myhostname/local1.log got created during debug run. rsysloghost='loghost', remotehost='remotehost'. Thanks for having a look at this, -- regards, Henry On Di, 2009-09-08 at 09:55 +0200, Rainer Gerhards wrote: > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -------------- next part -------------- 5054.727606022:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 5054.727609261:main thread: main queue: EnqueueMsg advised worker start 5054.727630769:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 5054.727662249:imtcp.c: -------- calling select, active fds (max 4): 4 5054.727668059:main thread: initialization completed, transitioning to regular run mode 5056.400110663:imtcp.c: New connect on NSD 0x65a690. 5056.400240204:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400255357:imtcp.c: netstream 0x683350 with new data 5056.400272148:imtcp.c: logmsg: flags 20, from 'remotehost', msg Sep 8 12:17:36 remotehost root: test by henry 5056.400274671:imtcp.c: Message has legacy syslog format. 5056.400279417:imtcp.c: main queue: entry added, size now 1 entries 5056.400282172:imtcp.c: wtpAdviseMaxWorkers signals busy 5056.400290169:imtcp.c: main queue: EnqueueMsg advised worker start 5056.400301360:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400311400:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400328690:main queue:Reg/w0: result of expression evaluation: 0 5056.400337430:main queue:Reg/w0: result of expression evaluation: 0 5056.400346474:main queue:Reg/w0: result of expression evaluation: 0 5056.400351644:main queue:Reg/w0: result of expression evaluation: 0 5056.400358215:main queue:Reg/w0: result of expression evaluation: 0 5056.400366212:main queue:Reg/w0: result of expression evaluation: 0 5056.400371208:main queue:Reg/w0: result of expression evaluation: 0 5056.400381921:main queue:Reg/w0: result of expression evaluation: 0 5056.400388679:main queue:Reg/w0: result of expression evaluation: 0 5056.400395611:main queue:Reg/w0: result of expression evaluation: 0 5056.400403792:main queue:Reg/w0: result of expression evaluation: 0 5056.400420697:main queue:Reg/w0: result of expression evaluation: 0 5056.400430773:main queue:Reg/w0: result of expression evaluation: 1 5056.400434138:main queue:Reg/w0: Called action, logging to builtin-file 5056.400440938:main queue:Reg/w0: (DYNlocal1) 5056.400507577:main queue:Reg/w0: Called LogError, msg: Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400531156:main queue:Reg/w0: logmsg: flags 1, from 'loghost', msg Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400537445:main queue:Reg/w0: Message has legacy syslog format. 5056.400540197:main queue:Reg/w0: main queue: entry added, size now 1 entries 5056.400542937:main queue:Reg/w0: wtpAdviseMaxWorkers signals busy 5056.400544844:main queue:Reg/w0: main queue: EnqueueMsg advised worker start 5056.400548374:main queue:Reg/w0: Removed entry 0 for file '[OPEN FAILED]' from dynaCache. 5056.400550633:main queue:Reg/w0: Action requested to be suspended, done that. 5056.400554546:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400562730:main queue:Reg/w0: result of expression evaluation: 0 5056.400572312:main queue:Reg/w0: result of expression evaluation: 1 5056.400575746:main queue:Reg/w0: Called action, logging to builtin-file 5056.400580715:main queue:Reg/w0: (/var/log/syslog) 5056.400591557:main queue:Reg/w0: result of expression evaluation: 0 5056.400607572:main queue:Reg/w0: result of expression evaluation: 0 5056.400612225:main queue:Reg/w0: result of expression evaluation: 0 5056.400619969:main queue:Reg/w0: result of expression evaluation: 0 5056.400627299:main queue:Reg/w0: result of expression evaluation: 0 5056.400633371:main queue:Reg/w0: result of expression evaluation: 0 5056.400639122:main queue:Reg/w0: result of expression evaluation: 0 5056.400644767:main queue:Reg/w0: result of expression evaluation: 0 5056.400651994:main queue:Reg/w0: result of expression evaluation: 1 5056.400656433:main queue:Reg/w0: Called action, logging to builtin-file 5056.400668930:main queue:Reg/w0: (/var/log/debug) 5056.400687895:main queue:Reg/w0: result of expression evaluation: 0 5056.400702536:main queue:Reg/w0: result of expression evaluation: 0 5056.400705763:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. From henry78 at gmx.at Tue Sep 8 12:41:37 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:41:37 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Message-ID: <1252406497.17741.48.camel@eberhe.office.chipkarte.at> The file (and folder) are created by the syslog user and definitely accessible. But it works if i don't drop privileges. So I'll investigate this further and report back. Thanks for pushing me that far. -- regard, Henry On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > Hi, > > I got the debug log, it was too big to be sent via the list (but I got it as > list admin). I see that you drop privileges to the user "syslog". This > probably explains what happens. I think the file is created before you drop > privileges, but can then no longer be written when running in the new > security context. Could you verify that the user "syslog" can access this > file? Also, could you temporarily remove the Privilege drop? > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 08, 2009 9:55 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > can you provide a debug log? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > Hello! > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > worked. > > > Is this worth ab bug? > > > > > > -- > > > kind regards, Henry > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > Hi! > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > > on > > > > ubuntu: > > > > > > > > -----8<----- > > > > $ModLoad imtcp > > > > $InputTCPServerRun 514 > > > > > > > > # some dynamic templates > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > # log remote local1 to dynamic diretory > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > $syslogfacility-text == 'local1' \ > > > > then -?DYNlocal1 > > > > ----->8----- > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > after > > > the > > > > remote host (myhostname) and creates the file local1.log (again: > > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > > that > > > > file, but logs the error: > > > > > > > > -----8<----- > > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > > - > > > > discarding message > > > > ----->8----- > > > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > > file > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:47:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:47:05 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at><9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> <1252406497.17741.48.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2E@GRFEXC.intern.adiscon.com> It is important to know that the PrivDrop directive set was a quick and dirty "let's implement it as far as possible, some is better than nothing" approach. It is expected that a couple of things break if it is used. Of course, if the users has proper rights, what you intend to do should work. I just wanted to alert you on the state of this feature (a mailing list search probably brings up more, but I have no time right now to do this). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 12:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > The file (and folder) are created by the syslog user and definitely > accessible. > > But it works if i don't drop privileges. So I'll investigate this > further and report back. > > Thanks for pushing me that far. > > -- > regard, Henry > > > On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > > Hi, > > > > I got the debug log, it was too big to be sent via the list (but I > got it as > > list admin). I see that you drop privileges to the user "syslog". > This > > probably explains what happens. I think the file is created before > you drop > > privileges, but can then no longer be written when running in the new > > security context. Could you verify that the user "syslog" can access > this > > file? Also, could you temporarily remove the Privilege drop? > > > > Thanks, > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > > Sent: Tuesday, September 08, 2009 9:55 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > can you provide a debug log? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] Could not open dynamic file ... - > discarding > > > > message > > > > > > > > Hello! > > > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > > worked. > > > > Is this worth ab bug? > > > > > > > > -- > > > > kind regards, Henry > > > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > > Hi! > > > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog > v4 > > > on > > > > > ubuntu: > > > > > > > > > > -----8<----- > > > > > $ModLoad imtcp > > > > > $InputTCPServerRun 514 > > > > > > > > > > # some dynamic templates > > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > > > # log remote local1 to dynamic diretory > > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > > $syslogfacility-text == 'local1' \ > > > > > then -?DYNlocal1 > > > > > ----->8----- > > > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > > after > > > > the > > > > > remote host (myhostname) and creates the file local1.log > (again: > > > > > sufficient permissions: syslog:syslog 640). But it doesn't > write to > > > > that > > > > > file, but logs the error: > > > > > > > > > > -----8<----- > > > > > Could not open dynamic file > '/var/log/remote/myhostname/local1.log' > > > - > > > > > discarding message > > > > > ----->8----- > > > > > > > > > > As you might guess my question is: Why isn't rsyslog able to > open a > > > > file > > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 13:23:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 08 Sep 2009 13:23:10 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <1252408990.17679.10.camel@rgf11> Hi all, I am really banging my head on a problem which sounds too easy. I have seen that my systems (and some others as well), seem to not provide more than 1024 bytes on a recvfrom() call. With wireshark, I see that the system itself, at the IP layer, receives more data. I am a bit puzzled, to phrase it lightly. I did not find any information on such a limitation. I have created a strip-down version of a receiver, even built it on top of the Linux man pages samples. Out of desperation, I even set the receivebuf size, which I think has no effect on datagram sockets. Still... I only get 1024 bytes. Code is after my sig. Does anybody have an idea what is going on OR a good place where to ask this question? Thanks, Rainer #include #include #include #include #include #include #include #define BUF_SIZE 2048 int main(int argc, char *argv[]) { struct addrinfo hints; struct addrinfo *result, *rp; int sfd, s; struct sockaddr_storage peer_addr; socklen_t peer_addr_len; ssize_t nread; char buf[BUF_SIZE]; if (argc != 2) { fprintf(stderr, "Usage: %s port\n", argv[0]); exit(EXIT_FAILURE); } memset(&hints, 0, sizeof(struct addrinfo)); hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ hints.ai_protocol = 0; /* Any protocol */ hints.ai_canonname = NULL; hints.ai_addr = NULL; hints.ai_next = NULL; s = getaddrinfo(NULL, argv[1], &hints, &result); if (s != 0) { fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); exit(EXIT_FAILURE); } /* getaddrinfo() returns a list of address structures. Try each address until we successfully bind(2). If socket(2) (or bind(2)) fails, we (close the socket and) try the next address. */ for (rp = result; rp != NULL; rp = rp->ai_next) { sfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (sfd == -1) continue; int result2; int bufSize = 2048; result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, sizeof(bufSize)); printf("result of setsockopt: %d\n", result2); if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) break; /* Success */ close(sfd); } if (rp == NULL) { /* No address succeeded */ fprintf(stderr, "Could not bind\n"); exit(EXIT_FAILURE); } freeaddrinfo(result); /* No longer needed */ /* Read datagrams and echo them back to sender */ for (;;) { peer_addr_len = sizeof(struct sockaddr_storage); memset(buf, 0, BUF_SIZE); nread = recvfrom(sfd, buf, BUF_SIZE, 0, (struct sockaddr *) &peer_addr, &peer_addr_len); if(nread > 1024) printf("NREAD > 1024!"); if (nread == -1) continue; /* Ignore failed request */ char host[NI_MAXHOST], service[NI_MAXSERV]; s = getnameinfo((struct sockaddr *) &peer_addr, peer_addr_len, host, NI_MAXHOST, service, NI_MAXSERV, NI_NUMERICSERV); if (s == 0) printf("Received %ld bytes from %s:%s, msg:'%s'\n", (long) nread, host, service, buf); else fprintf(stderr, "getnameinfo: %s\n", gai_strerror(s)); } } From rgerhards at hq.adiscon.com Tue Sep 8 14:17:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 14:17:05 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <1252408990.17679.10.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> oh my... Please disregard this question. I was working on a tcpdump file, and the message length actually *is* 1024 bytes. I was confused by Wireshark's (correct!) indication that the frame is 1066 octets in length. Of course, this is correct, if you take the 42 octets of UDP header into account... I guess the dump file was created with a max of 1K... Sometimes it is sooo easy ... and yet so hard to see ;) Sorry for the interruption, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 1:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Help requested: UDP max message size? > > Hi all, > > I am really banging my head on a problem which sounds too easy. I have > seen that my systems (and some others as well), seem to not provide > more > than 1024 bytes on a recvfrom() call. With wireshark, I see that the > system itself, at the IP layer, receives more data. I am a bit puzzled, > to phrase it lightly. I did not find any information on such a > limitation. > > I have created a strip-down version of a receiver, even built it on top > of the Linux man pages samples. Out of desperation, I even set the > receivebuf size, which I think has no effect on datagram sockets. > Still... I only get 1024 bytes. Code is after my sig. > > Does anybody have an idea what is going on OR a good place where to ask > this question? > > Thanks, > Rainer > > #include > #include > #include > #include > #include > #include > #include > > #define BUF_SIZE 2048 > > int > main(int argc, char *argv[]) > { > struct addrinfo hints; > struct addrinfo *result, *rp; > int sfd, s; > struct sockaddr_storage peer_addr; > socklen_t peer_addr_len; > ssize_t nread; > char buf[BUF_SIZE]; > > if (argc != 2) { > fprintf(stderr, "Usage: %s port\n", argv[0]); > exit(EXIT_FAILURE); > } > > memset(&hints, 0, sizeof(struct addrinfo)); > hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ > hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > hints.ai_flags = AI_PASSIVE; /* For wildcard IP address > */ > hints.ai_protocol = 0; /* Any protocol */ > hints.ai_canonname = NULL; > hints.ai_addr = NULL; > hints.ai_next = NULL; > > s = getaddrinfo(NULL, argv[1], &hints, &result); > if (s != 0) { > fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); > exit(EXIT_FAILURE); > } > > /* getaddrinfo() returns a list of address structures. > Try each address until we successfully bind(2). > If socket(2) (or bind(2)) fails, we (close the socket > and) try the next address. */ > > for (rp = result; rp != NULL; rp = rp->ai_next) { > sfd = socket(rp->ai_family, rp->ai_socktype, > rp->ai_protocol); > if (sfd == -1) > continue; > > > int result2; > int bufSize = 2048; > result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > sizeof(bufSize)); > printf("result of setsockopt: %d\n", result2); > > if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > break; /* Success */ > > close(sfd); > } > > if (rp == NULL) { /* No address succeeded */ > fprintf(stderr, "Could not bind\n"); > exit(EXIT_FAILURE); > } > > freeaddrinfo(result); /* No longer needed */ > > /* Read datagrams and echo them back to sender */ > for (;;) { > peer_addr_len = sizeof(struct sockaddr_storage); > memset(buf, 0, BUF_SIZE); > nread = recvfrom(sfd, buf, BUF_SIZE, 0, > (struct sockaddr *) &peer_addr, &peer_addr_len); > if(nread > 1024) > printf("NREAD > 1024!"); > if (nread == -1) > continue; /* Ignore failed request */ > > char host[NI_MAXHOST], service[NI_MAXSERV]; > > s = getnameinfo((struct sockaddr *) &peer_addr, > peer_addr_len, host, NI_MAXHOST, > service, NI_MAXSERV, NI_NUMERICSERV); > if (s == 0) > printf("Received %ld bytes from %s:%s, msg:'%s'\n", > (long) nread, host, service, buf); > else > fprintf(stderr, "getnameinfo: %s\n", > gai_strerror(s)); > } > } > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Tue Sep 8 17:24:59 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Tue, 08 Sep 2009 12:24:59 -0300 Subject: [rsyslog] FailoverSyslogServer: Write buffer immediatly to disk instead to memory option available? Message-ID: <4AA6774B.6070604@swiss-wireless.com.ar> Hello Short: rsyslog V3-V4: Can I write to disk ONLY if the remote rsyslog server is not reachable? Can it be done with the following? $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $ActionQueueType Disk $ActionQueueFileName srvrfwd # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure If the server is reachable there will be nothing written do disk (my problem is using CF card in embedded system see below) or is it written first to disk and than processed by the dispatcher? Long: I use rsyslog on AP which I try now to log remotely to a syslog server because CF card dies if you log often. Now the problem is, that I don't want to loss my syslog messages in the case the syslog server is not available. Now this messages are helt in the memory but if there is a power loss all messages will be lost. We have many power losses here :-( Greetings and thanks Beat From david at lang.hm Tue Sep 8 19:55:21 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 10:55:21 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> References: <1252408990.17679.10.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 8 20:38:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 20:38:12 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 08.09.09 19:55 Betreff: Re: [rsyslog] Help requested: UDP max message size? On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Sep 8 20:41:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 11:41:39 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) yes, as far as I know the none of the senders are rsyslog yet. I am working from the central server out. the central server is rsyslog with no problems all but this one relay box are rsyslog things sending to these relay boxes are whatever syslog sender was on the OS/appliance (there may be some acting as relays as well as sending for themselves) David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 08.09.09 19:55 > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > >> oh my... Please disregard this question. I was working on a tcpdump file, and >> the message length actually *is* 1024 bytes. I was confused by Wireshark's >> (correct!) indication that the frame is 1066 octets in length. Of course, >> this is correct, if you take the 42 octets of UDP header into account... >> >> I guess the dump file was created with a max of 1K... > > the dump file was set -s 0 (up to 64k packet size), but many/most syslog > senders will limit their outbound data to 1k > > David Lang > >> Sometimes it is sooo easy ... and yet so hard to see ;) >> >> Sorry for the interruption, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>> Sent: Tuesday, September 08, 2009 1:23 PM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] Help requested: UDP max message size? >>> >>> Hi all, >>> >>> I am really banging my head on a problem which sounds too easy. I have >>> seen that my systems (and some others as well), seem to not provide >>> more >>> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >>> system itself, at the IP layer, receives more data. I am a bit puzzled, >>> to phrase it lightly. I did not find any information on such a >>> limitation. >>> >>> I have created a strip-down version of a receiver, even built it on top >>> of the Linux man pages samples. Out of desperation, I even set the >>> receivebuf size, which I think has no effect on datagram sockets. >>> Still... I only get 1024 bytes. Code is after my sig. >>> >>> Does anybody have an idea what is going on OR a good place where to ask >>> this question? >>> >>> Thanks, >>> Rainer >>> >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> #define BUF_SIZE 2048 >>> >>> int >>> main(int argc, char *argv[]) >>> { >>> struct addrinfo hints; >>> struct addrinfo *result, *rp; >>> int sfd, s; >>> struct sockaddr_storage peer_addr; >>> socklen_t peer_addr_len; >>> ssize_t nread; >>> char buf[BUF_SIZE]; >>> >>> if (argc != 2) { >>> fprintf(stderr, "Usage: %s port\n", argv[0]); >>> exit(EXIT_FAILURE); >>> } >>> >>> memset(&hints, 0, sizeof(struct addrinfo)); >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >>> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >>> */ >>> hints.ai_protocol = 0; /* Any protocol */ >>> hints.ai_canonname = NULL; >>> hints.ai_addr = NULL; >>> hints.ai_next = NULL; >>> >>> s = getaddrinfo(NULL, argv[1], &hints, &result); >>> if (s != 0) { >>> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >>> exit(EXIT_FAILURE); >>> } >>> >>> /* getaddrinfo() returns a list of address structures. >>> Try each address until we successfully bind(2). >>> If socket(2) (or bind(2)) fails, we (close the socket >>> and) try the next address. */ >>> >>> for (rp = result; rp != NULL; rp = rp->ai_next) { >>> sfd = socket(rp->ai_family, rp->ai_socktype, >>> rp->ai_protocol); >>> if (sfd == -1) >>> continue; >>> >>> >>> int result2; >>> int bufSize = 2048; >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >>> sizeof(bufSize)); >>> printf("result of setsockopt: %d\n", result2); >>> >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >>> break; /* Success */ >>> >>> close(sfd); >>> } >>> >>> if (rp == NULL) { /* No address succeeded */ >>> fprintf(stderr, "Could not bind\n"); >>> exit(EXIT_FAILURE); >>> } >>> >>> freeaddrinfo(result); /* No longer needed */ >>> >>> /* Read datagrams and echo them back to sender */ >>> for (;;) { >>> peer_addr_len = sizeof(struct sockaddr_storage); >>> memset(buf, 0, BUF_SIZE); >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >>> (struct sockaddr *) &peer_addr, &peer_addr_len); >>> if(nread > 1024) >>> printf("NREAD > 1024!"); >>> if (nread == -1) >>> continue; /* Ignore failed request */ >>> >>> char host[NI_MAXHOST], service[NI_MAXSERV]; >>> >>> s = getnameinfo((struct sockaddr *) &peer_addr, >>> peer_addr_len, host, NI_MAXHOST, >>> service, NI_MAXSERV, NI_NUMERICSERV); >>> if (s == 0) >>> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >>> (long) nread, host, service, buf); >>> else >>> fprintf(stderr, "getnameinfo: %s\n", >>> gai_strerror(s)); >>> } >>> } >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 21:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 21:24:04 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE36@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 08, 2009 8:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > Was there an non-rsyslog relay in the relay chain? If not, > it points to the rsyslog forwarding module doing the > truncation (what recent v3+ i think should not do...) > > yes, as far as I know the none of the senders are rsyslog yet. Well, from what I see in the tcpdump logs, the initial sender is rsyslog and the messages originated from imklog. I can point you to the entries in question, but I don't have logs with me now. Rainer > > I am working from the central server out. > > the central server is rsyslog with no problems > > all but this one relay box are rsyslog > > things sending to these relay boxes are whatever syslog > sender was on the > OS/appliance (there may be some acting as relays as well as > sending for > themselves) > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 08.09.09 19:55 > > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > >> oh my... Please disregard this question. I was working on > a tcpdump file, and > >> the message length actually *is* 1024 bytes. I was > confused by Wireshark's > >> (correct!) indication that the frame is 1066 octets in > length. Of course, > >> this is correct, if you take the 42 octets of UDP header > into account... > >> > >> I guess the dump file was created with a max of 1K... > > > > the dump file was set -s 0 (up to 64k packet size), but > many/most syslog > > senders will limit their outbound data to 1k > > > > David Lang > > > >> Sometimes it is sooo easy ... and yet so hard to see ;) > >> > >> Sorry for the interruption, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>> Sent: Tuesday, September 08, 2009 1:23 PM > >>> To: rsyslog at lists.adiscon.com > >>> Subject: [rsyslog] Help requested: UDP max message size? > >>> > >>> Hi all, > >>> > >>> I am really banging my head on a problem which sounds too > easy. I have > >>> seen that my systems (and some others as well), seem to > not provide > >>> more > >>> than 1024 bytes on a recvfrom() call. With wireshark, I > see that the > >>> system itself, at the IP layer, receives more data. I am > a bit puzzled, > >>> to phrase it lightly. I did not find any information on such a > >>> limitation. > >>> > >>> I have created a strip-down version of a receiver, even > built it on top > >>> of the Linux man pages samples. Out of desperation, I even set the > >>> receivebuf size, which I think has no effect on datagram sockets. > >>> Still... I only get 1024 bytes. Code is after my sig. > >>> > >>> Does anybody have an idea what is going on OR a good > place where to ask > >>> this question? > >>> > >>> Thanks, > >>> Rainer > >>> > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> > >>> #define BUF_SIZE 2048 > >>> > >>> int > >>> main(int argc, char *argv[]) > >>> { > >>> struct addrinfo hints; > >>> struct addrinfo *result, *rp; > >>> int sfd, s; > >>> struct sockaddr_storage peer_addr; > >>> socklen_t peer_addr_len; > >>> ssize_t nread; > >>> char buf[BUF_SIZE]; > >>> > >>> if (argc != 2) { > >>> fprintf(stderr, "Usage: %s port\n", argv[0]); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> memset(&hints, 0, sizeof(struct addrinfo)); > >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 > or IPv6 */ > >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > >>> hints.ai_flags = AI_PASSIVE; /* For > wildcard IP address > >>> */ > >>> hints.ai_protocol = 0; /* Any protocol */ > >>> hints.ai_canonname = NULL; > >>> hints.ai_addr = NULL; > >>> hints.ai_next = NULL; > >>> > >>> s = getaddrinfo(NULL, argv[1], &hints, &result); > >>> if (s != 0) { > >>> fprintf(stderr, "getaddrinfo: %s\n", > gai_strerror(s)); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> /* getaddrinfo() returns a list of address structures. > >>> Try each address until we successfully bind(2). > >>> If socket(2) (or bind(2)) fails, we (close > the socket > >>> and) try the next address. */ > >>> > >>> for (rp = result; rp != NULL; rp = rp->ai_next) { > >>> sfd = socket(rp->ai_family, rp->ai_socktype, > >>> rp->ai_protocol); > >>> if (sfd == -1) > >>> continue; > >>> > >>> > >>> int result2; > >>> int bufSize = 2048; > >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > >>> sizeof(bufSize)); > >>> printf("result of setsockopt: %d\n", result2); > >>> > >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > >>> break; /* Success */ > >>> > >>> close(sfd); > >>> } > >>> > >>> if (rp == NULL) { /* No address > succeeded */ > >>> fprintf(stderr, "Could not bind\n"); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> freeaddrinfo(result); /* No longer needed */ > >>> > >>> /* Read datagrams and echo them back to sender */ > >>> for (;;) { > >>> peer_addr_len = sizeof(struct sockaddr_storage); > >>> memset(buf, 0, BUF_SIZE); > >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, > >>> (struct sockaddr *) &peer_addr, > &peer_addr_len); > >>> if(nread > 1024) > >>> printf("NREAD > 1024!"); > >>> if (nread == -1) > >>> continue; /* Ignore > failed request */ > >>> > >>> char host[NI_MAXHOST], service[NI_MAXSERV]; > >>> > >>> s = getnameinfo((struct sockaddr *) &peer_addr, > >>> peer_addr_len, host, NI_MAXHOST, > >>> service, NI_MAXSERV, > NI_NUMERICSERV); > >>> if (s == 0) > >>> printf("Received %ld bytes from %s:%s, > msg:'%s'\n", > >>> (long) nread, host, service, buf); > >>> else > >>> fprintf(stderr, "getnameinfo: %s\n", > >>> gai_strerror(s)); > >>> } > >>> } > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From corsmith at gmail.com Tue Sep 8 21:46:25 2009 From: corsmith at gmail.com (Corey Smith) Date: Tue, 8 Sep 2009 15:46:25 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> References: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909081246q6535c9d0s997f814c18e69c83@mail.gmail.com> On Thu, Sep 3, 2009 at 2:58 PM, Rainer Gerhards wrote: > Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) I come from a FreeBSD background so the Solaris package management system leaves much to be desired. The limitations of the default toolset in Solaris are amazing. That is why I started using pkgsrc - a portable package management system originally developed for netbsd. The way I got gcc44 working on Solaris 10/Sparc64: Download, install pkgsrc and bootstrap using the gcc from sunfreeware (3.4) # Check out: http://www.netbsd.org/docs/pkgsrc/platforms.html#solaris Install pkgsrc-wip using a cvs checkout # Check out: http://pkgsrc-wip.sourceforge.net/ Replace the wip/rsyslog port with the one I attached earlier on the thread. Build rsyslog and dependencies using gcc3.4 Install gcc44 from wip/gcc44 and make the changes I described in the first message of the thread cd /usr/pkgsrc/wip/rsyslog && make update # rebuild rsyslog with gcc44 On a side note: I tried building rsyslog-5 from git which compiled but would core every time I started it. BTW: Which virtual machine are you using to emulate sparc64? -Corey Smith From rgerhards at hq.adiscon.com Wed Sep 9 15:00:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Sep 2009 15:00:55 +0200 Subject: [rsyslog] epoll-supporting imudp Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE43@GRFEXC.intern.adiscon.com> Hi all, I have finally begun to add some new features concurrently to my analysis of the reported segfault (which seems to be environment-induced and will very seldom show up in practice). I have now created an imudp-epoll branch, based on current master, which provides an imudp module that utilizes epoll() instead of select(). This is my first move towards supporting epoll() where useful. Please note that imudp will not tremendously benefit - on busy servers, select() is very infrequently called, as we read the socket as long there is data. On non-busy servers, there are few calls and I don't expect that epoll vs. select makes any real difference then. Please note that the most benefit from epoll we will gain on tcp based traffic. However, moving to epoll there is far more complicated, because I need to remodel the netstream driver layer. Thus I wanted to gain some experience with easy things first. Probably imuxsock is my next target after I have waited some time for feedback. I would appreciate if some folks could try out the new branch and tell me their experience. I plan to include the new functionality with the next v5-devel release in a couple of days. Rainer From joshsystem at gmail.com Thu Sep 10 07:45:28 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 13:45:28 +0800 Subject: [rsyslog] does rsyslog supports data analytic Message-ID: hi all, i want to receive each syslog msg then input it into my special processing module.after processing the data,output the new data into database.of course,the raw data we must keep it into files. can anyone give me some suggestions? PS: i browse the git source code, but i can't understand why the Experimental-lockfree is not adopted? thanks From david at lang.hm Thu Sep 10 08:26:09 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 9 Sep 2009 23:26:09 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: Message-ID: On Thu, 10 Sep 2009, Josh Zhao wrote: > hi all, i want to receive each syslog msg then input it into my special > processing module.after processing the data,output the new data into > database.of course,the raw data we must keep it into files. can anyone give > me some suggestions? would you not just list two destinations, one to the place you want the raw data archived and one to the processing module? I have a very high volume of logs (>300M/day), so I roll the logs every 5 min with this script #!/bin/sh # PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin umask 022 year=`date +%Y` month=`date +%m` day=`date +%d` fdate=`date +%Y%m%d.%H%M` logroot=/var/log logroll=$logroot/oldlogs cd $logroot mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null mv messages messages.$fdate mv messages.$fdate $logroll/$year/$month/$day/messages.$fdate mv /usr/local/bin/ita/system/itascan1a-p/winlogs /usr/local/bin/ita/system/itascan1a-p/winlogs.0 pkill -HUP syslogd pkill -HUP syslog-ng #pkill win-dump gzip -9 $logroll/$year/$month/$day/messages.$fdate > PS: i browse the git source code, but i can't understand why the > Experimental-lockfree > is > not adopted? I believe that it boils down to complications in being sure that there are no bugs, and the fact that even without that there has been a LOT of room for improvement from the early 3.x timeframe to the current 5.x version. I expect that after the current round of improvements are settled that aspect of things will get reexamined. David Lang From rgerhards at hq.adiscon.com Thu Sep 10 08:32:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 08:32:08 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, September 10, 2009 8:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > PS: i browse the git source code, but i can't understand why the > > > Experimental-lockfree > > is > > not adopted? > > I believe that it boils down to complications in being sure > that there are > no bugs, and the fact that even without that there has been a > LOT of room > for improvement from the early 3.x timeframe to the current > 5.x version. > > I expect that after the current round of improvements are > settled that > aspect of things will get reexamined. That branch is mostly there for historical reasons. I keep that branch as a think-tank, but it is is obsoleted. Also, in less polite words than David used, it simply doesn't work. Getting this code with multiple producers and consumers correct is far from being trivial and the literature I browsed indicates that it is probably not possible given the other predicates the code must obey to. Still, optimization is high up on the todo list. Rainer From joshsystem at gmail.com Thu Sep 10 15:25:23 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 21:25:23 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: Thanks for David and Rainer's reply.I m sorry that I did not explain my question clearly.I m new to rsyslog and want to add a processing module in rsyslog.The rsyslog has input plugins(front-end) and output plugins(back-end).My processing module receives data from input plugins and output the processed data and raw data both into output plugins.So how I add it? 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, September 10, 2009 8:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > PS: i browse the git source code, but i can't understand why the > > > > > Experimental-lockfree shortlog;h=refs/heads/Experimental-lockfree> > > > is > > > not adopted? > > > > I believe that it boils down to complications in being sure > > that there are > > no bugs, and the fact that even without that there has been a > > LOT of room > > for improvement from the early 3.x timeframe to the current > > 5.x version. > > > > I expect that after the current round of improvements are > > settled that > > aspect of things will get reexamined. > > That branch is mostly there for historical reasons. I keep that branch as a > think-tank, but it is is obsoleted. Also, in less polite words than David > used, it simply doesn't work. Getting this code with multiple producers and > consumers correct is far from being trivial and the literature I browsed > indicates that it is probably not possible given the other predicates the > code must obey to. Still, optimization is high up on the todo list. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Sep 10 17:06:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 17:06:33 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Thursday, September 10, 2009 3:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module > in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins > and > output the processed data and raw data both into output plugins.So how > I add > it? What you are looking for is a library plugin. Unfortunaley, library plugins will work together with the scripting engine. In other words: there currently is no in-proc method available. What you can do, however, is chain two rsyslog instances, pipe data to your plugin and send that data to the other instance. Far from perfect and easy to do, but maybe a workable work-around... Rainer > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that > branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > used, it simply doesn't work. Getting this code with multiple > producers and > > consumers correct is far from being trivial and the literature I > browsed > > indicates that it is probably not possible given the other predicates > the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Thu Sep 10 22:50:30 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:50:30 +0200 Subject: [rsyslog] use snmp as source Message-ID: <4AA96696.9080906@irontec.com> Hi! I have a SNMP capable VoIP gateway, and I want to be able to log in syslog, the messages received by SNMP. Is this possible? I have read that in the other direction, it is possible. http://www.rsyslog.com/doc-omsnmp.html Thanks From mikel at irontec.com Thu Sep 10 22:52:50 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:52:50 +0200 Subject: [rsyslog] use snmp as source In-Reply-To: <4AA96696.9080906@irontec.com> References: <4AA96696.9080906@irontec.com> Message-ID: <4AA96722.4090307@irontec.com> The solution is snmptrapd Thanks!! Mikel Jimenez wrote: > Hi! > > I have a SNMP capable VoIP gateway, and I want to be able to log in > syslog, the messages received by SNMP. > > > Is this possible? > > I have read that in the other direction, it is possible. > http://www.rsyslog.com/doc-omsnmp.html > > Thanks > > From joshsystem at gmail.com Fri Sep 11 02:13:35 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 08:13:35 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: You mean I have to rewrite the processing module in rainerscript.where can i find the detailed documents related to the scripting engine? Thank you! 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Thursday, September 10, 2009 3:25 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > > and > > output the processed data and raw data both into output plugins.So how > > I add > > it? > > What you are looking for is a library plugin. Unfortunaley, library plugins > will work together with the scripting engine. In other words: there > currently > is no in-proc method available. > > What you can do, however, is chain two rsyslog instances, pipe data to your > plugin and send that data to the other instance. Far from perfect and easy > to > do, but maybe a workable work-around... > > Rainer > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that > > branch as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > > David > > > used, it simply doesn't work. Getting this code with multiple > > producers and > > > consumers correct is far from being trivial and the literature I > > browsed > > > indicates that it is probably not possible given the other predicates > > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 02:26:46 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 17:26:46 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > You mean I have to rewrite the processing module in rainerscript.where can i > find the detailed documents related to the scripting engine? right now rainerscript is as much an idea as an implementation. it can be used for a few things, but mostly just for filter 'does this log match X' type of things. David Lang > Thank you! > 2009/9/10 Rainer Gerhards > >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>> Sent: Thursday, September 10, 2009 3:25 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>> >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>> question clearly.I m new to rsyslog and want to add a processing module >>> in >>> rsyslog.The rsyslog has input plugins(front-end) and output >>> plugins(back-end).My processing module receives data from input plugins >>> and >>> output the processed data and raw data both into output plugins.So how >>> I add >>> it? >> >> What you are looking for is a library plugin. Unfortunaley, library plugins >> will work together with the scripting engine. In other words: there >> currently >> is no in-proc method available. >> >> What you can do, however, is chain two rsyslog instances, pipe data to your >> plugin and send that data to the other instance. Far from perfect and easy >> to >> do, but maybe a workable work-around... >> >> Rainer >> >>> >>> >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>> >>>>>> PS: i browse the git source code, but i can't understand why the >>>>>> >>>>> Experimental-lockfree>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>> is >>>>>> not adopted? >>>>> >>>>> I believe that it boils down to complications in being sure >>>>> that there are >>>>> no bugs, and the fact that even without that there has been a >>>>> LOT of room >>>>> for improvement from the early 3.x timeframe to the current >>>>> 5.x version. >>>>> >>>>> I expect that after the current round of improvements are >>>>> settled that >>>>> aspect of things will get reexamined. >>>> >>>> That branch is mostly there for historical reasons. I keep that >>> branch as a >>>> think-tank, but it is is obsoleted. Also, in less polite words than >>> David >>>> used, it simply doesn't work. Getting this code with multiple >>> producers and >>>> consumers correct is far from being trivial and the literature I >>> browsed >>>> indicates that it is probably not possible given the other predicates >>> the >>>> code must obey to. Still, optimization is high up on the todo list. >>>> >>>> Rainer >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 03:39:01 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 09:39:01 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Is rsyslog no way to reslove problem, What about syslog-ng? What I think about,rsyslog's multi-thread archititure is better for my mulit-core hardware. The logs data is very high volume too. Could you give me any suggestion on this matter? Thank you! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > You mean I have to rewrite the processing module in rainerscript.where > can i > > find the detailed documents related to the scripting engine? > > right now rainerscript is as much an idea as an implementation. it can be > used for a few things, but mostly just for filter 'does this log match X' > type of things. > > David Lang > > > Thank you! > > 2009/9/10 Rainer Gerhards > > > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>> Sent: Thursday, September 10, 2009 3:25 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>> > >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my > >>> question clearly.I m new to rsyslog and want to add a processing module > >>> in > >>> rsyslog.The rsyslog has input plugins(front-end) and output > >>> plugins(back-end).My processing module receives data from input plugins > >>> and > >>> output the processed data and raw data both into output plugins.So how > >>> I add > >>> it? > >> > >> What you are looking for is a library plugin. Unfortunaley, library > plugins > >> will work together with the scripting engine. In other words: there > >> currently > >> is no in-proc method available. > >> > >> What you can do, however, is chain two rsyslog instances, pipe data to > your > >> plugin and send that data to the other instance. Far from perfect and > easy > >> to > >> do, but maybe a workable work-around... > >> > >> Rainer > >> > >>> > >>> > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>> david at lang.hm > >>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>> > >>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>> > >>>>> Experimental-lockfree >>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>> is > >>>>>> not adopted? > >>>>> > >>>>> I believe that it boils down to complications in being sure > >>>>> that there are > >>>>> no bugs, and the fact that even without that there has been a > >>>>> LOT of room > >>>>> for improvement from the early 3.x timeframe to the current > >>>>> 5.x version. > >>>>> > >>>>> I expect that after the current round of improvements are > >>>>> settled that > >>>>> aspect of things will get reexamined. > >>>> > >>>> That branch is mostly there for historical reasons. I keep that > >>> branch as a > >>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>> David > >>>> used, it simply doesn't work. Getting this code with multiple > >>> producers and > >>>> consumers correct is far from being trivial and the literature I > >>> browsed > >>>> indicates that it is probably not possible given the other predicates > >>> the > >>>> code must obey to. Still, optimization is high up on the todo list. > >>>> > >>>> Rainer > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 06:28:59 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 21:28:59 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > about,rsyslog's multi-thread archititure is better for my mulit-core > hardware. The logs data is very high volume too. Could you give me any > suggestion on this matter? my experiance with syslog-ng was not good, so I'm not the right person to talk about doing this sort of thing with it. but I am not aware of any syslog daemon that lets you insert your own logic in the middle of the processing. rsyslog has the concept, but it has not been implemented (fixing bugs and speeding it up has taken priority) what sort of volume do you consider 'high'? (it's amazing the range that this can span, so I've learned to ask rather than assume ;-) since you are needing to get your final data into a database, I think that you will find that rsyslog will (or will soon) suit your needs far better than alternate approaches. the ability to process multiple messages in one transaction that is being developed will be a huge improvement in terms of database interaction. I would look at what rainer suggested for now. have one copy of rsyslog that receives the messages, does whatever formatting/cleanup is needed on them, then passes the logs to one or more instances of your code to do additional processing, which can then feed the results into another instance of rsyslog to forward them on, insert them into a database, etc. when rainerscript gains the capability to alter the fields (instead of just testing them), then there will be a lot more that can be done inside rsyslog. David Lang > Thank you! > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> You mean I have to rewrite the processing module in rainerscript.where >> can i >>> find the detailed documents related to the scripting engine? >> >> right now rainerscript is as much an idea as an implementation. it can be >> used for a few things, but mostly just for filter 'does this log match X' >> type of things. >> >> David Lang >> >>> Thank you! >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>> >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>>>> question clearly.I m new to rsyslog and want to add a processing module >>>>> in >>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>> plugins(back-end).My processing module receives data from input plugins >>>>> and >>>>> output the processed data and raw data both into output plugins.So how >>>>> I add >>>>> it? >>>> >>>> What you are looking for is a library plugin. Unfortunaley, library >> plugins >>>> will work together with the scripting engine. In other words: there >>>> currently >>>> is no in-proc method available. >>>> >>>> What you can do, however, is chain two rsyslog instances, pipe data to >> your >>>> plugin and send that data to the other instance. Far from perfect and >> easy >>>> to >>>> do, but maybe a workable work-around... >>>> >>>> Rainer >>>> >>>>> >>>>> >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>> david at lang.hm >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>> >>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>> >>>>>>> Experimental-lockfree>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>> is >>>>>>>> not adopted? >>>>>>> >>>>>>> I believe that it boils down to complications in being sure >>>>>>> that there are >>>>>>> no bugs, and the fact that even without that there has been a >>>>>>> LOT of room >>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>> 5.x version. >>>>>>> >>>>>>> I expect that after the current round of improvements are >>>>>>> settled that >>>>>>> aspect of things will get reexamined. >>>>>> >>>>>> That branch is mostly there for historical reasons. I keep that >>>>> branch as a >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>> David >>>>>> used, it simply doesn't work. Getting this code with multiple >>>>> producers and >>>>>> consumers correct is far from being trivial and the literature I >>>>> browsed >>>>>> indicates that it is probably not possible given the other predicates >>>>> the >>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>> >>>>>> Rainer >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 08:16:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 08:16:43 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE6D@GRFEXC.intern.adiscon.com> Hi Davd, I think you hit it right on the nail. But I have also thought a bit more about the idea. Actually, I think, one can implement processing modules right now. Especially the configuration is a bit tricky, but it should really work. The rough outline is to use an output module for that. Output modules may do whatever they want as long as they use the provided interfaces. As such, they can also inject messages. So the idea is to define an output module, that accepts the message, does any processing necessary, indicated RS_RET_DISCARD to the rule engine (to prevent the message from being further processed) and inject the "newly generated" message back into the main message queue. That would also be much faster than whatever RainerScript will have to offer, because RainerScript relies on VM execution. I just don't have time to elaborately talk someone through this approach... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, September 11, 2009 6:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? > What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you > give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the > right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the > concept, but it has > not been implemented (fixing bugs and speeding it up has > taken priority) > > what sort of volume do you consider 'high'? (it's amazing the > range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, > I think that > you will find that rsyslog will (or will soon) suit your > needs far better > than alternate approaches. the ability to process multiple > messages in one > transaction that is being developed will be a huge > improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to > one or more > instances of your code to do additional processing, which can > then feed > the results into another instance of rsyslog to forward them > on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields > (instead of > just testing them), then there will be a lot more that can be > done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in > rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an > implementation. it can be > >> used for a few things, but mostly just for filter 'does > this log match X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I > did not explain my > >>>>> question clearly.I m new to rsyslog and want to add a > processing module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data > from input plugins > >>>>> and > >>>>> output the processed data and raw data both into output > plugins.So how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. > Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other > words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog > instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far > from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't > understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less > polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the > literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the > other predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on > the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 10:17:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 10:17:12 +0200 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <1252657032.17679.12.camel@rgf11> Now that I got an idea of how this could be implemented with current rsyslog technology, I would be interested in some more details of what you intend to do with the processing module. What exactly will it do with the message? I am asking because I would like to see a real use case. Thinking about the scenario I have proposed in my last mail, I think I see some pitfalls and I am not sure if they will cause any trouble in real projects. So I would appreciate if you could provide more in-depth info. Thanks, Rainer On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins and > output the processed data and raw data both into output plugins.So how I add > it? > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than David > > used, it simply doesn't work. Getting this code with multiple producers and > > consumers correct is far from being trivial and the literature I browsed > > indicates that it is probably not possible given the other predicates the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From thomas.mieslinger at 1und1.de Fri Sep 11 11:47:28 2009 From: thomas.mieslinger at 1und1.de (Thomas Mieslinger) Date: Fri, 11 Sep 2009 11:47:28 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelp transports In-Reply-To: <1236002254.28865.46.camel@rf10up.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com> <1236002254.28865.46.camel@rf10up.intern.adiscon.com> Message-ID: <4AAA1CB0.90106@1und1.de> Hi, I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. One machine (logsender) has: $ModLoad omrelp.so $WorkDirectory /var/spool/rsyslog $ActionQueueType LinkedList $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on user.* :omrelp:loghost:2514 or user.* @@loghost:1514 and the other machine (loghost) has $ModLoad imrelp.so $UDPServerRun 514 $InputTCPServerRun 1514 $InputRELPServerRun 2514 *.* -/some/logfile If I restart rsyslog on loghost without restarting rsyslog on logsender, the logs produced on logsender never appear on loghost. Is this working as designed? Is there a kind of syslog.debug facility where I can monitor the reconnect activity of rsyslog? Thanks in advance Thomas From rgerhards at hq.adiscon.com Fri Sep 11 12:13:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 12:13:12 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelptransports References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com><1236002254.28865.46.camel@rf10up.intern.adiscon.com> <4AAA1CB0.90106@1und1.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE79@GRFEXC.intern.adiscon.com> I suggest to turn on debug logging on both the client and sender: http://www.rsyslog.com/doc-troubleshoot.html Often, the debug log points to an obvious problem source. If it does not, feel free to mail me the logs to rgerhards at gmail.com BUT let me know you did so - I usually do not monitor this account. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger > Sent: Friday, September 11, 2009 11:47 AM > To: rsyslog-users > Subject: [rsyslog] rsyslogd not reconnecting when using tcp or > omrelptransports > > Hi, > > I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. > One > machine (logsender) has: > > $ModLoad omrelp.so > $WorkDirectory /var/spool/rsyslog > $ActionQueueType LinkedList > $ActionQueueFileName srvrfwd > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > > user.* :omrelp:loghost:2514 > or > user.* @@loghost:1514 > > and the other machine (loghost) > has > > $ModLoad imrelp.so > $UDPServerRun 514 > $InputTCPServerRun 1514 > $InputRELPServerRun 2514 > > *.* -/some/logfile > > If I restart rsyslog on loghost without restarting rsyslog on > logsender, > the logs produced on logsender never appear on loghost. Is this working > as designed? > > Is there a kind of syslog.debug facility where I can monitor the > reconnect activity of rsyslog? > > Thanks in advance Thomas > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Sep 11 15:21:34 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 11 Sep 2009 15:21:34 +0200 Subject: [rsyslog] rsyslog 5.1.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE82@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.5. This is the first public beta of the v5 branch. As such, it is an important milestone on the way to an even more powerful rsyslogd. As of our usual policies, this means that the first v5-stable will probably available within two to three month, so before the end of the year. Please note that this also means we are shifting our development efforts primarily to v5 for any new functionality (but we keep the option open to add some enhancements to v4-devel). Feedback and bug reports on the new v5-bea branch would be deeply appreciated. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-174.phtml Changelog: http://www.rsyslog.com/Article400.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From joshsystem at gmail.com Fri Sep 11 16:17:14 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 22:17:14 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Our raw data is "high" volume that means to prcocess data about 100M/min. Yes, I want to improve the system performance as soon as possibe. As you said,rsyslog has a concept that inserts my logic module into it ,but it was not been implemented. Could you point out in detail? The rainerscript seems not that strong,otherwise, it is a good idea for user interface. 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the concept, but it has > not been implemented (fixing bugs and speeding it up has taken priority) > > what sort of volume do you consider 'high'? (it's amazing the range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, I think that > you will find that rsyslog will (or will soon) suit your needs far better > than alternate approaches. the ability to process multiple messages in one > transaction that is being developed will be a huge improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to one or more > instances of your code to do additional processing, which can then feed > the results into another instance of rsyslog to forward them on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields (instead of > just testing them), then there will be a lot more that can be done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an implementation. it can > be > >> used for a few things, but mostly just for filter 'does this log match > X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain > my > >>>>> question clearly.I m new to rsyslog and want to add a processing > module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data from input > plugins > >>>>> and > >>>>> output the processed data and raw data both into output plugins.So > how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the other > predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 17:09:32 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 23:09:32 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <1252657032.17679.12.camel@rgf11> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> Message-ID: Thanks Rainer. The basic purpose is statistics,which can accumulate some fields of msgs,but I think the customers have more weird requirements. 2009/9/11 Rainer Gerhards > Now that I got an idea of how this could be implemented with current > rsyslog technology, I would be interested in some more details of what > you intend to do with the processing module. What exactly will it do > with the message? I am asking because I would like to see a real use > case. Thinking about the scenario I have proposed in my last mail, I > think I see some pitfalls and I am not sure if they will cause any > trouble in real projects. > > So I would appreciate if you could provide more in-depth info. > > Thanks, > Rainer > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > and > > output the processed data and raw data both into output plugins.So how I > add > > it? > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that branch > as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > > used, it simply doesn't work. Getting this code with multiple producers > and > > > consumers correct is far from being trivial and the literature I > browsed > > > indicates that it is probably not possible given the other predicates > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:18:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:18:27 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><1252657032.17679.12.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> OK, that's not very precise, but I have also thought a bit about this. What I have proposed this morning should be possible. But you should be warned, it requires a lot of reading and understanding the source code. A good place to start is the template input and output modules as well as some actual output modules. I think imdiag would be useful (because it is simple) and probably also either omstdout (simple) and omoracle (complex, but utilizes the vector interface which may be the best choice for what you intend to acomplish). As a side-note, if this is paid work you may want to think about purchasing some development help from Adiscon, which may dramatically reduce the time you need to get started (just a thought, omoracle was crafted very well without any such help - thanks again!). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Friday, September 11, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks Rainer. The basic purpose is statistics,which can accumulate > some > fields of msgs,but I think the customers have more weird requirements. > > 2009/9/11 Rainer Gerhards > > > Now that I got an idea of how this could be implemented with current > > rsyslog technology, I would be interested in some more details of > what > > you intend to do with the processing module. What exactly will it do > > with the message? I am asking because I would like to see a real use > > case. Thinking about the scenario I have proposed in my last mail, I > > think I see some pitfalls and I am not sure if they will cause any > > trouble in real projects. > > > > So I would appreciate if you could provide more in-depth info. > > > > Thanks, > > Rainer > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > Thanks for David and Rainer's reply.I m sorry that I did not > explain my > > > question clearly.I m new to rsyslog and want to add a processing > module > > in > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > plugins(back-end).My processing module receives data from input > plugins > > and > > > output the processed data and raw data both into output plugins.So > how I > > add > > > it? > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > PS: i browse the git source code, but i can't understand why > the > > > > > > > > > > > Experimental-lockfree > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > is > > > > > > not adopted? > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > that there are > > > > > no bugs, and the fact that even without that there has been a > > > > > LOT of room > > > > > for improvement from the early 3.x timeframe to the current > > > > > 5.x version. > > > > > > > > > > I expect that after the current round of improvements are > > > > > settled that > > > > > aspect of things will get reexamined. > > > > > > > > That branch is mostly there for historical reasons. I keep that > branch > > as a > > > > think-tank, but it is is obsoleted. Also, in less polite words > than > > David > > > > used, it simply doesn't work. Getting this code with multiple > producers > > and > > > > consumers correct is far from being trivial and the literature I > > browsed > > > > indicates that it is probably not possible given the other > predicates > > the > > > > code must obey to. Still, optimization is high up on the todo > list. > > > > > > > > Rainer > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Sep 11 17:24:04 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 08:24:04 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Our raw data is "high" volume that means to prcocess data about 100M/min. is this 100M log records, or 100MB of log data (if the latter, approximatly how large are the recors, of how many log records/min) I'm currently processing ~300K messages averaging ~256 bytes/message for a total of ~75MB of logs/min. in my testing v4 will support up to about 6x this volume before it runs into problems (it can receive them faster, up to gig-E wire speed, the limit is in the output, which is ~80K records a sec if doing trivial work like writing them to disk or ~30K records/sec if doing more complex things like forwarding them elsewhere) improvements in V5 include a batch mode that lets an output module process up to N records at a time. I expect this to provide close to a Nx speedup to the output capabilities (with single log per action much of the overhead is in the queue locking, so multiple output workers doesn't help much, with batches not only is much more getting done per pass, but you have the possibility of each output thread taking long enough to get it's work done that it's effective to run more of them without locking contention being the bottleneck) this batch mode will be especially useful for database work as it will let you insert multiple messages in the database in a single transaction. what transport are you using to deliver the logs to your server? > Yes, I want to improve the system performance as soon as possibe. what is the bottleneck you are running into today (what syslog system are you using, etc)? > As you > said,rsyslog has a concept that inserts my logic module into it ,but it was > not been implemented. Could you point out in detail? The rainerscript seems > not that strong,otherwise, it is a good idea for user interface. if you are looking at the source look for imtemplate and omtemplate, basicly he is suggesting creating a custom output module that rsyslog thinks is delivering the messages somewhere, have it be given the log, do it's processing, then acting like an input module and delivering the result to rsyslog as if it was a new message that just arrived. you will need to put some filters in rsyslog to keep your output module from seeing the logs that it creates, and either use discard or filters to keep the other output modules from seeing the raw input that your module is looking for. David Lang > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I think >>> about,rsyslog's multi-thread archititure is better for my mulit-core >>> hardware. The logs data is very high volume too. Could you give me any >>> suggestion on this matter? >> >> my experiance with syslog-ng was not good, so I'm not the right person to >> talk about doing this sort of thing with it. >> >> but I am not aware of any syslog daemon that lets you insert your own >> logic in the middle of the processing. rsyslog has the concept, but it has >> not been implemented (fixing bugs and speeding it up has taken priority) >> >> what sort of volume do you consider 'high'? (it's amazing the range that >> this can span, so I've learned to ask rather than assume ;-) >> >> since you are needing to get your final data into a database, I think that >> you will find that rsyslog will (or will soon) suit your needs far better >> than alternate approaches. the ability to process multiple messages in one >> transaction that is being developed will be a huge improvement in terms of >> database interaction. >> >> I would look at what rainer suggested for now. >> >> have one copy of rsyslog that receives the messages, does whatever >> formatting/cleanup is needed on them, then passes the logs to one or more >> instances of your code to do additional processing, which can then feed >> the results into another instance of rsyslog to forward them on, insert >> them into a database, etc. >> >> when rainerscript gains the capability to alter the fields (instead of >> just testing them), then there will be a lot more that can be done inside >> rsyslog. >> >> David Lang >> >>> Thank you! >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> You mean I have to rewrite the processing module in rainerscript.where >>>> can i >>>>> find the detailed documents related to the scripting engine? >>>> >>>> right now rainerscript is as much an idea as an implementation. it can >> be >>>> used for a few things, but mostly just for filter 'does this log match >> X' >>>> type of things. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>> >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain >> my >>>>>>> question clearly.I m new to rsyslog and want to add a processing >> module >>>>>>> in >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>> plugins(back-end).My processing module receives data from input >> plugins >>>>>>> and >>>>>>> output the processed data and raw data both into output plugins.So >> how >>>>>>> I add >>>>>>> it? >>>>>> >>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>> plugins >>>>>> will work together with the scripting engine. In other words: there >>>>>> currently >>>>>> is no in-proc method available. >>>>>> >>>>>> What you can do, however, is chain two rsyslog instances, pipe data to >>>> your >>>>>> plugin and send that data to the other instance. Far from perfect and >>>> easy >>>>>> to >>>>>> do, but maybe a workable work-around... >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>> david at lang.hm >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>> >>>>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>>>> >>>>>>>>> Experimental-lockfree>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>> is >>>>>>>>>> not adopted? >>>>>>>>> >>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>> that there are >>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>> LOT of room >>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>> 5.x version. >>>>>>>>> >>>>>>>>> I expect that after the current round of improvements are >>>>>>>>> settled that >>>>>>>>> aspect of things will get reexamined. >>>>>>>> >>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>> branch as a >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>>>> David >>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>> producers and >>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>> browsed >>>>>>>> indicates that it is probably not possible given the other >> predicates >>>>>>> the >>>>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>>>> >>>>>>>> Rainer >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:30:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:30:25 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE89@GRFEXC.intern.adiscon.com> > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, > do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. I think I did not state one important fact: this is not a dirty trick, but something that the engine was designed for. This mechanism was originally designed and is (somewhat) actually used to report back error conditions. It's used sparsely, because of the circular loop potential. But it is something the engine can handle and is designed to - so no abuse. Actually, I have begun to think if for some feature requests (string replacements before finally writing to an output) this may be good alternative approach. But it seems to involve more overhead than necessary for the job. > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters > to > keep the other output modules from seeing the raw input that your > module > is looking for. Returning RS_RET_DISCARD would solve this, as it stops processing. You just need to make sure that the newly injected messages don't go back into the same rule. With multiple rulesets we now have, this is trivial. But while all this is interesting, I unfortunately have more pressing things to do ;) Rainer > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit- > core > >>> hardware. The logs data is very high volume too. Could you give me > any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right > person to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your > own > >> logic in the middle of the processing. rsyslog has the concept, but > it has > >> not been implemented (fixing bugs and speeding it up has taken > priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range > that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I > think that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages > in one > >> transaction that is being developed will be a huge improvement in > terms of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then > feed > >> the results into another instance of rsyslog to forward them on, > insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead > of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it > can > >> be > >>>> used for a few things, but mostly just for filter 'does this log > match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a > processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output > plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, > library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: > there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe > data to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from > perfect and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand > why the > >>>>>>>>>> > >>>>>>>>> Experimental- > lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep > that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature > I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:39:30 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:39:30 -0600 Subject: [rsyslog] How to increase message size maximum Message-ID: Hello, I replied to an older thread on the forums but wanted to bring this up here. I have an application logging to rsyslog (version 3.22.0) which sends very large messages. We are trying to migrate logging from syslog-ng to rsyslog and I'm running into a problem where messages appear to be truncated or split across lines (I'm seeing both behaviors but I'm not sure if they are both the same problem). In syslog-ng we had to increase the maximum message size with the parameter "log_msg_size(65536);" within the options section. I'm trying to do the equivalent in rsyslog. I saw mention in the forums that this was possibly configurable via a #define but no mention of where I might find this. I realize this is probably outside the typical syslog spec but unfortunately it's a situation I have to deal with for rsyslog to be suitable in our environment. Unfortunately I cannot post the log messages publicly but I can probably provide sanitized samples if an individual was willing to help. Thank you, Aaron From rgerhards at hq.adiscon.com Fri Sep 11 18:48:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 18:48:39 +0200 Subject: [rsyslog] How to increase message size maximum References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, search for MAXLINE inside the code, change that, and recompile. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, September 11, 2009 6:40 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How to increase message size maximum > > Hello, > I replied to an older thread on the forums but wanted to bring this > up > here. I have an application logging to rsyslog (version 3.22.0) which > sends > very large messages. We are trying to migrate logging from syslog-ng to > rsyslog and I'm running into a problem where messages appear to be > truncated > or split across lines (I'm seeing both behaviors but I'm not sure if > they > are both the same problem). In syslog-ng we had to increase the maximum > message size with the parameter "log_msg_size(65536);" within the > options > section. I'm trying to do the equivalent in rsyslog. I saw mention in > the > forums that this was possibly configurable via a #define but no mention > of > where I might find this. > > I realize this is probably outside the typical syslog spec but > unfortunately > it's a situation I have to deal with for rsyslog to be suitable in our > environment. Unfortunately I cannot post the log messages publicly but > I can > probably provide sanitized samples if an individual was willing to > help. > > Thank you, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:59:38 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:59:38 -0600 Subject: [rsyslog] How to increase message size maximum In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> Message-ID: On Fri, Sep 11, 2009 at 10:48 AM, Rainer Gerhards wrote: > $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, > search for MAXLINE inside the code, change that, and recompile. > > Excellent - thank you, that is supported as of 3.21.4 per ChangeLog. From anichols at trumped.org Fri Sep 11 23:29:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 15:29:39 -0600 Subject: [rsyslog] Beginning of log messages being removed Message-ID: Hello, Log messages are now being delivered correctly after raising the messagesize value - now I seem to be having some issue with parsing. I am trying to log only the %msg% portion of the log message however the beginning of that message seems to be removed. Below are the two templates I used to log in both the rawmsg and then the value of %msg% so you can see what is being removed. I cannot post the entire %msg% value, but the two are the same with the exception of the beginning value. I just need to be able to log the message portion without the timestamp which is being delivered from the client. Thinking this may have been fixed with some of the parsing problems I have updated to the latest 4.x stable release - this problem has been observed on 3.22.1 & 4.4.1. I am currently running against 4.4.1. Two templates: $template ServerXML, "%timestamp% || %hostname% || %msg%\n" $template ServerXMLraw, "%rawmsg%\n" Using the first template the message looks like this: Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder of message removed for brevity, but it is intact in the logs) Using the second template the raw message looks like this: <142>Sep 11 21:15:01 localhost References: Message-ID: On Fri, 11 Sep 2009, Aaron Nichols wrote: > Hello, > Log messages are now being delivered correctly after raising the > messagesize value - now I seem to be having some issue with parsing. I am > trying to log only the %msg% portion of the log message however the > beginning of that message seems to be removed. Below are the two templates I > used to log in both the rawmsg and then the value of %msg% so you can see > what is being removed. I cannot post the entire %msg% value, but the two are > the same with the exception of the beginning value. I just need to be able > to log the message portion without the timestamp which is being delivered > from the client. > > Thinking this may have been fixed with some of the parsing problems I have > updated to the latest 4.x stable release - this problem has been observed on > 3.22.1 & 4.4.1. I am currently running against 4.4.1. > > Two templates: > $template ServerXML, "%timestamp% || %hostname% || %msg%\n" > $template ServerXMLraw, "%rawmsg%\n" > > Using the first template the message looks like this: > Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder > of message removed for brevity, but it is intact in the logs) > > Using the second template the raw message looks like this: > <142>Sep 11 21:15:01 localhost userId= > > I'm trying to understand why the value " from %msg%. because it's being put in %syslogtag% as the program name. David Lang From joshsystem at gmail.com Sat Sep 12 07:36:11 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:36:11 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/11 Rainer Gerhards > OK, that's not very precise, but I have also thought a bit about this. What > I > have proposed this morning should be possible. But you should be warned, it > requires a lot of reading and understanding the source code. A good place > to > start is the template input and output modules as well as some actual > output > modules. I think imdiag would be useful (because it is simple) and probably > also either omstdout (simple) and omoracle (complex, but utilizes the > vector > interface which may be the best choice for what you intend to acomplish). > > As a side-note, if this is paid work you may want to think about purchasing > some development help from Adiscon, which may dramatically reduce the time > you need to get started (just a thought, omoracle was crafted very well > without any such help - thanks again!). > Where can I find the purchased development help in details? Thanks > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Friday, September 11, 2009 5:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks Rainer. The basic purpose is statistics,which can accumulate > > some > > fields of msgs,but I think the customers have more weird requirements. > > > > 2009/9/11 Rainer Gerhards > > > > > Now that I got an idea of how this could be implemented with current > > > rsyslog technology, I would be interested in some more details of > > what > > > you intend to do with the processing module. What exactly will it do > > > with the message? I am asking because I would like to see a real use > > > case. Thinking about the scenario I have proposed in my last mail, I > > > think I see some pitfalls and I am not sure if they will cause any > > > trouble in real projects. > > > > > > So I would appreciate if you could provide more in-depth info. > > > > > > Thanks, > > > Rainer > > > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > > Thanks for David and Rainer's reply.I m sorry that I did not > > explain my > > > > question clearly.I m new to rsyslog and want to add a processing > > module > > > in > > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > > plugins(back-end).My processing module receives data from input > > plugins > > > and > > > > output the processed data and raw data both into output plugins.So > > how I > > > add > > > > it? > > > > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > > > -----Original Message----- > > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > david at lang.hm > > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > > To: rsyslog-users > > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > > > PS: i browse the git source code, but i can't understand why > > the > > > > > > > > > > > > > Experimental-lockfree > > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > > is > > > > > > > not adopted? > > > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > > that there are > > > > > > no bugs, and the fact that even without that there has been a > > > > > > LOT of room > > > > > > for improvement from the early 3.x timeframe to the current > > > > > > 5.x version. > > > > > > > > > > > > I expect that after the current round of improvements are > > > > > > settled that > > > > > > aspect of things will get reexamined. > > > > > > > > > > That branch is mostly there for historical reasons. I keep that > > branch > > > as a > > > > > think-tank, but it is is obsoleted. Also, in less polite words > > than > > > David > > > > > used, it simply doesn't work. Getting this code with multiple > > producers > > > and > > > > > consumers correct is far from being trivial and the literature I > > > browsed > > > > > indicates that it is probably not possible given the other > > predicates > > > the > > > > > code must obey to. Still, optimization is high up on the todo > > list. > > > > > > > > > > Rainer > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Sat Sep 12 07:39:36 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:39:36 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck is output module.it relies on storage disk. I currently test the splunk,but not strong enough.of course,the client delivers the message to server via ethernet. >if you are looking at the source look for imtemplate and omtemplate, >basicly he is suggesting creating a custom output module that rsyslog >thinks is delivering the messages somewhere, have it be given the log, do >it's processing, then acting like an input module and delivering the >result to rsyslog as if it was a new message that just arrived. . This approach as Rainer may be overhead :( ; >That would also be much faster than whatever RainerScript will have to offer, >because RainerScript relies on VM execution. As Rainer said that RainerScrpt is not easy to be extended,but I think it is the perfect approach. I can't find any documents about it:(; It's really hard to start it! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Our raw data is "high" volume that means to prcocess data about 100M/min. > > is this 100M log records, or 100MB of log data (if the latter, > approximatly how large are the recors, of how many log records/min) > > I'm currently processing ~300K messages averaging ~256 bytes/message for a > total of ~75MB of logs/min. > > in my testing v4 will support up to about 6x this volume before it runs > into problems (it can receive them faster, up to gig-E wire speed, the > limit is in the output, which is ~80K records a sec if doing trivial work > like writing them to disk or ~30K records/sec if doing more complex things > like forwarding them elsewhere) > > improvements in V5 include a batch mode that lets an output module process > up to N records at a time. I expect this to provide close to a Nx speedup > to the output capabilities (with single log per action much of the > overhead is in the queue locking, so multiple output workers doesn't help > much, with batches not only is much more getting done per pass, but you > have the possibility of each output thread taking long enough to get it's > work done that it's effective to run more of them without locking > contention being the bottleneck) > > this batch mode will be especially useful for database work as it will let > you insert multiple messages in the database in a single transaction. > > what transport are you using to deliver the logs to your server? > > > Yes, I want to improve the system performance as soon as possibe. > > what is the bottleneck you are running into today (what syslog system are > you using, etc)? > > > As you > > said,rsyslog has a concept that inserts my logic module into it ,but it > was > > not been implemented. Could you point out in detail? The rainerscript > seems > > not that strong,otherwise, it is a good idea for user interface. > > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. > > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters to > keep the other output modules from seeing the raw input that your module > is looking for. > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit-core > >>> hardware. The logs data is very high volume too. Could you give me any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right person > to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your own > >> logic in the middle of the processing. rsyslog has the concept, but it > has > >> not been implemented (fixing bugs and speeding it up has taken priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I think > that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages in > one > >> transaction that is being developed will be a huge improvement in terms > of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then feed > >> the results into another instance of rsyslog to forward them on, insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it can > >> be > >>>> used for a few things, but mostly just for filter 'does this log match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe data > to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from perfect > and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand why > the > >>>>>>>>>> > >>>>>>>>> Experimental-lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Sep 12 08:18:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 23:18:17 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Sat, 12 Sep 2009, Josh Zhao wrote: > It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck > is output module.it relies on storage disk. I currently test the splunk,but > not strong enough. I am also using splunk. at that volume I would expect to be able to handle everything with a single server. My old system has one box receiving logs from many sources, archiving them, and forwarding them on to several other systems for event correlation, reporting, etc (one of which is my old splunk box). it is comfortably handling about your volume (averaged over a few min, my peak seconds top 10K logs). things are spread across multiple systems less due to current load than in preperation for increasing the load (I am gearing up to handle ~10x my current load) I've done a fair bit of stress testing of the various components and applications. what sort of problems are you having? > of course,the client delivers the message to server via > ethernet. I was meaning are you using TCP syslog, UDP syslog, or something else? David Lang >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. > . > This approach as Rainer may be overhead :( ; > >> That would also be much faster than whatever RainerScript will have to > offer, >> because RainerScript relies on VM execution. > > As Rainer said that RainerScrpt is not easy to be extended,but I think it is > the perfect approach. I can't find any documents about it:(; It's really > hard to start it! > > > > > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Our raw data is "high" volume that means to prcocess data about 100M/min. >> >> is this 100M log records, or 100MB of log data (if the latter, >> approximatly how large are the recors, of how many log records/min) >> >> I'm currently processing ~300K messages averaging ~256 bytes/message for a >> total of ~75MB of logs/min. >> >> in my testing v4 will support up to about 6x this volume before it runs >> into problems (it can receive them faster, up to gig-E wire speed, the >> limit is in the output, which is ~80K records a sec if doing trivial work >> like writing them to disk or ~30K records/sec if doing more complex things >> like forwarding them elsewhere) >> >> improvements in V5 include a batch mode that lets an output module process >> up to N records at a time. I expect this to provide close to a Nx speedup >> to the output capabilities (with single log per action much of the >> overhead is in the queue locking, so multiple output workers doesn't help >> much, with batches not only is much more getting done per pass, but you >> have the possibility of each output thread taking long enough to get it's >> work done that it's effective to run more of them without locking >> contention being the bottleneck) >> >> this batch mode will be especially useful for database work as it will let >> you insert multiple messages in the database in a single transaction. >> >> what transport are you using to deliver the logs to your server? >> >>> Yes, I want to improve the system performance as soon as possibe. >> >> what is the bottleneck you are running into today (what syslog system are >> you using, etc)? >> >>> As you >>> said,rsyslog has a concept that inserts my logic module into it ,but it >> was >>> not been implemented. Could you point out in detail? The rainerscript >> seems >>> not that strong,otherwise, it is a good idea for user interface. >> >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. >> >> you will need to put some filters in rsyslog to keep your output module >> from seeing the logs that it creates, and either use discard or filters to >> keep the other output modules from seeing the raw input that your module >> is looking for. >> >> David Lang >> >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> Is rsyslog no way to reslove problem, What about syslog-ng? What I >> think >>>>> about,rsyslog's multi-thread archititure is better for my mulit-core >>>>> hardware. The logs data is very high volume too. Could you give me any >>>>> suggestion on this matter? >>>> >>>> my experiance with syslog-ng was not good, so I'm not the right person >> to >>>> talk about doing this sort of thing with it. >>>> >>>> but I am not aware of any syslog daemon that lets you insert your own >>>> logic in the middle of the processing. rsyslog has the concept, but it >> has >>>> not been implemented (fixing bugs and speeding it up has taken priority) >>>> >>>> what sort of volume do you consider 'high'? (it's amazing the range that >>>> this can span, so I've learned to ask rather than assume ;-) >>>> >>>> since you are needing to get your final data into a database, I think >> that >>>> you will find that rsyslog will (or will soon) suit your needs far >> better >>>> than alternate approaches. the ability to process multiple messages in >> one >>>> transaction that is being developed will be a huge improvement in terms >> of >>>> database interaction. >>>> >>>> I would look at what rainer suggested for now. >>>> >>>> have one copy of rsyslog that receives the messages, does whatever >>>> formatting/cleanup is needed on them, then passes the logs to one or >> more >>>> instances of your code to do additional processing, which can then feed >>>> the results into another instance of rsyslog to forward them on, insert >>>> them into a database, etc. >>>> >>>> when rainerscript gains the capability to alter the fields (instead of >>>> just testing them), then there will be a lot more that can be done >> inside >>>> rsyslog. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> >>>>> 2009/9/11 >>>>> >>>>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>>>> >>>>>>> You mean I have to rewrite the processing module in >> rainerscript.where >>>>>> can i >>>>>>> find the detailed documents related to the scripting engine? >>>>>> >>>>>> right now rainerscript is as much an idea as an implementation. it can >>>> be >>>>>> used for a few things, but mostly just for filter 'does this log match >>>> X' >>>>>> type of things. >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Thank you! >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>> >>>>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not >> explain >>>> my >>>>>>>>> question clearly.I m new to rsyslog and want to add a processing >>>> module >>>>>>>>> in >>>>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>>>> plugins(back-end).My processing module receives data from input >>>> plugins >>>>>>>>> and >>>>>>>>> output the processed data and raw data both into output plugins.So >>>> how >>>>>>>>> I add >>>>>>>>> it? >>>>>>>> >>>>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>>>> plugins >>>>>>>> will work together with the scripting engine. In other words: there >>>>>>>> currently >>>>>>>> is no in-proc method available. >>>>>>>> >>>>>>>> What you can do, however, is chain two rsyslog instances, pipe data >> to >>>>>> your >>>>>>>> plugin and send that data to the other instance. Far from perfect >> and >>>>>> easy >>>>>>>> to >>>>>>>> do, but maybe a workable work-around... >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2009/9/10 Rainer Gerhards >>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>>>> david at lang.hm >>>>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>>> >>>>>>>>>>>> PS: i browse the git source code, but i can't understand why >> the >>>>>>>>>>>> >>>>>>>>>>> Experimental-lockfree>>>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>>>> is >>>>>>>>>>>> not adopted? >>>>>>>>>>> >>>>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>>>> that there are >>>>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>>>> LOT of room >>>>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>>>> 5.x version. >>>>>>>>>>> >>>>>>>>>>> I expect that after the current round of improvements are >>>>>>>>>>> settled that >>>>>>>>>>> aspect of things will get reexamined. >>>>>>>>>> >>>>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>>>> branch as a >>>>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words >> than >>>>>>>>> David >>>>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>>>> producers and >>>>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>>>> browsed >>>>>>>>>> indicates that it is probably not possible given the other >>>> predicates >>>>>>>>> the >>>>>>>>>> code must obey to. Still, optimization is high up on the todo >> list. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Sep 14 10:14:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Sep 2009 10:14:39 +0200 Subject: [rsyslog] DNS Cache Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE9D@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that in parallel to my bughunt (which involves a lot of waiting for lab results), I will now begin to implement a real DNS cache. That will be a v5-exclusive feature (too much trouble to do it in v4 and v5, code base has changed too much). Together with the case, I will probably also implement a feature to override reverse DNS resolution via a file - simply by loading non-expiring entries from that file. I just thought I share this plan, if someone has feature requests in that regard. It would be good to know them, as now is a good time to integrate them into the design. Tech side-note: I'll be using AVL trees for the cache, as I don't outrule many entries and this hopefully speeds up cache searches for larger caches. Once the avl tree class is there, I can probably speed up a few other things that currently rely on simple linked lists). I will probably do two or even three releases until the full functionality is there. I also plan to do a "pre-cache" v5-devel today, so that all new features (including imudp epoll) are rolled out and can be tested. Rainer From tbergfeld at hq.adiscon.com Mon Sep 14 17:05:35 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 14 Sep 2009 17:05:35 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.0. This release starts a new v5-development branch. This release offers a lot of new features like the use of epoll, when possible, in imudp, which provides greater performance and is a pilot to more such enhancements. Further more there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml Changelog: http://www.rsyslog.com/Article402.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld From david at lang.hm Tue Sep 15 02:51:19 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 14 Sep 2009 17:51:19 -0700 (PDT) Subject: [rsyslog] rsyslog 5.3.0 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: I do not see this tagged in git. David Lang On Mon, 14 Sep 2009, Tom Bergfeld wrote: > Date: Mon, 14 Sep 2009 17:05:35 +0200 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.0 (devel) released > > Hi all, > > We have just released rsyslog 5.3.0. This release starts a new v5-development > branch. > This release offers a lot of new features like the use of epoll, when > possible, in imudp, which provides greater performance and is a pilot to more > such enhancements. Further more there are also some bug fixes. See Changelog > for more details. This is a recommended update for all users of the devel > branch. > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml > > Changelog: > > http://www.rsyslog.com/Article402.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mikel at irontec.com Tue Sep 15 10:52:37 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 10:52:37 +0200 Subject: [rsyslog] server frozen when remote logging Message-ID: <4AAF55D5.20807@irontec.com> Hi!! I have 80 servers logging to a centralized rsyslog, and I have experimented the kaos!! Accidentaly the central server shutdowns, and one hour later, all the 80 servers frezze. Can not access ssh, ping... I use Debian in central server, and suse in nodes. Thanks! -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 10:56:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:56:28 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> This sounds like you are overdoing "reliable delivery". But I need configs and version information to tell you what may be the case. If it is an older v3 version, this may also be a bug in rsyslog. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 10:53 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] server frozen when remote logging > > Hi!! > > I have 80 servers logging to a centralized rsyslog, and I have > experimented the kaos!! > > Accidentaly the central server shutdowns, and one hour later, all the > 80 > servers frezze. > > Can not access ssh, ping... > > I use Debian in central server, and suse in nodes. > > Thanks! > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 10:58:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:58:44 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECC@GRFEXC.intern.adiscon.com> Thanks - I had forgotten to push the tags (but thankfully this time not the tagging itself ;)) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 15, 2009 2:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.0 (devel) released > > I do not see this tagged in git. From mikel at irontec.com Tue Sep 15 11:56:26 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:56:26 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF64CA.8020002@irontec.com> Ok Rainer In the clients: OS= opensuse 10.0 rsyslog version: 3.19.7 In the server OS=Debian 4.0 rsyslog version: 3.18.2 I attach the configuration files of the clients and the servers. The remote server is 192.1.4.215. Thanks Rainer Gerhards wrote: > This sounds like you are overdoing "reliable delivery". But I need configs > and version information to tell you what may be the case. If it is an older > v3 version, this may also be a bug in rsyslog. > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 10:53 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] server frozen when remote logging >> >> Hi!! >> >> I have 80 servers logging to a centralized rsyslog, and I have >> experimented the kaos!! >> >> Accidentaly the central server shutdowns, and one hour later, all the >> 80 >> servers frezze. >> >> Can not access ssh, ping... >> >> I use Debian in central server, and suse in nodes. >> >> Thanks! >> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-client.conf URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-server.conf URL: From mikel at irontec.com Tue Sep 15 11:58:52 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:58:52 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9E6A72.8080202@irontec.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <4AAF655C.6050601@irontec.com> Hi!! We are very interested, how much do you estimate? Thanks Mikel Jimenez wrote: > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of > January 27 ... > > Thanks > > Mikel Jimenez wrote: >> hi >> >> Some news about this? >> >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >> Maybe with a bounty? >> >> thanks >> > > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:01:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:01:02 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:59 AM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Hi!! > > We are very interested, how much do you estimate? Thanks. I've just asked the right people, please expect a reply either today or (depending on discussion) tomorrow, must probably via private mail. Rainer > Thanks > > Mikel Jimenez wrote: > > Ok, I will comunicate you if we decide. > > > > Is the development of phplogcon frezzed? the last version is of > > January 27 ... > > > > Thanks > > > > Mikel Jimenez wrote: > >> hi > >> > >> Some news about this? > >> > >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > >> Maybe with a bounty? > >> > >> thanks > >> > > > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:05:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:05:40 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Ok, there are errors in the config files. I've stopped looking at them when I saw EST=... ... @@$EST This does not work in rsyslog (yet). Please make sure that your configs are OK. With the versions you have, you either need to start rsyslogd interactively in debug mode OR simply look at the syslogd logs (those with syslog facility). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:56 AM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Ok Rainer > > In the clients: > > OS= opensuse 10.0 > rsyslog version: 3.19.7 > > In the server > OS=Debian 4.0 > rsyslog version: 3.18.2 > > I attach the configuration files of the clients and the servers. > > The remote server is 192.1.4.215. > > Thanks > > Rainer Gerhards wrote: > > This sounds like you are overdoing "reliable delivery". But I need > configs > > and version information to tell you what may be the case. If it is an > older > > v3 version, this may also be a bug in rsyslog. > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 10:53 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] server frozen when remote logging > >> > >> Hi!! > >> > >> I have 80 servers logging to a centralized rsyslog, and I have > >> experimented the kaos!! > >> > >> Accidentaly the central server shutdowns, and one hour later, all > the > >> 80 > >> servers frezze. > >> > >> Can not access ssh, ping... > >> > >> I use Debian in central server, and suse in nodes. > >> > >> Thanks! > >> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > From mikel at irontec.com Tue Sep 15 12:10:16 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:10:16 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> Message-ID: <4AAF6808.2080402@irontec.com> Yeah!!! :) Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:59 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Hi!! >> >> We are very interested, how much do you estimate? >> > > Thanks. I've just asked the right people, please expect a reply either today > or (depending on discussion) tomorrow, must probably via private mail. > > Rainer > > > > >> Thanks >> >> Mikel Jimenez wrote: >> >>> Ok, I will comunicate you if we decide. >>> >>> Is the development of phplogcon frezzed? the last version is of >>> January 27 ... >>> >>> Thanks >>> >>> Mikel Jimenez wrote: >>> >>>> hi >>>> >>>> Some news about this? >>>> >>>> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >>>> Maybe with a bounty? >>>> >>>> thanks >>>> >>>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From mikel at irontec.com Tue Sep 15 12:14:20 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:14:20 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Message-ID: <4AAF68FC.1040205@irontec.com> But rsyslog starts... If I use UDP instead TCP? Rainer Gerhards wrote: > Ok, there are errors in the config files. I've stopped looking at them when I > saw > > EST=... > > ... @@$EST > > This does not work in rsyslog (yet). Please make sure that your configs are > OK. With the versions you have, you either need to start rsyslogd > interactively in debug mode OR simply look at the syslogd logs (those with > syslog facility). > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:56 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> Ok Rainer >> >> In the clients: >> >> OS= opensuse 10.0 >> rsyslog version: 3.19.7 >> >> In the server >> OS=Debian 4.0 >> rsyslog version: 3.18.2 >> >> I attach the configuration files of the clients and the servers. >> >> The remote server is 192.1.4.215. >> >> Thanks >> >> Rainer Gerhards wrote: >> >>> This sounds like you are overdoing "reliable delivery". But I need >>> >> configs >> >>> and version information to tell you what may be the case. If it is an >>> >> older >> >>> v3 version, this may also be a bug in rsyslog. >>> >>> HTH >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] server frozen when remote logging >>>> >>>> Hi!! >>>> >>>> I have 80 servers logging to a centralized rsyslog, and I have >>>> experimented the kaos!! >>>> >>>> Accidentaly the central server shutdowns, and one hour later, all >>>> >> the >> >>>> 80 >>>> servers frezze. >>>> >>>> Can not access ssh, ping... >>>> >>>> I use Debian in central server, and suse in nodes. >>>> >>>> Thanks! >>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:18:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:18:30 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > But rsyslog starts... > > If I use UDP instead TCP? sure, because the messages are thrown away, at least I think. The point is: it doesn't make sense to hunt for a problem as long as we know that the config is incorrect. Better get the config clean first, then see if the problem even persists and then look at it. Bluntly and not meant to be embarrassing: I've set aside some time to do this kind of support, but if you need more "full service" help, it would probably be a good idea to purchase one of the support packages. They exists so that we can look at issues in depth. This is often an excellent values, as it may safe you hours and hours of work. And, really, I can't develop all this and provide this kind of full-service support ;) Rainer > > Rainer Gerhards wrote: > > Ok, there are errors in the config files. I've stopped looking at > them when I > > saw > > > > EST=... > > > > ... @@$EST > > > > This does not work in rsyslog (yet). Please make sure that your > configs are > > OK. With the versions you have, you either need to start rsyslogd > > interactively in debug mode OR simply look at the syslogd logs (those > with > > syslog facility). > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 11:56 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> Ok Rainer > >> > >> In the clients: > >> > >> OS= opensuse 10.0 > >> rsyslog version: 3.19.7 > >> > >> In the server > >> OS=Debian 4.0 > >> rsyslog version: 3.18.2 > >> > >> I attach the configuration files of the clients and the servers. > >> > >> The remote server is 192.1.4.215. > >> > >> Thanks > >> > >> Rainer Gerhards wrote: > >> > >>> This sounds like you are overdoing "reliable delivery". But I need > >>> > >> configs > >> > >>> and version information to tell you what may be the case. If it is > an > >>> > >> older > >> > >>> v3 version, this may also be a bug in rsyslog. > >>> > >>> HTH > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: [rsyslog] server frozen when remote logging > >>>> > >>>> Hi!! > >>>> > >>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>> experimented the kaos!! > >>>> > >>>> Accidentaly the central server shutdowns, and one hour later, all > >>>> > >> the > >> > >>>> 80 > >>>> servers frezze. > >>>> > >>>> Can not access ssh, ping... > >>>> > >>>> I use Debian in central server, and suse in nodes. > >>>> > >>>> Thanks! > >>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 12:47:31 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:47:31 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> Message-ID: <4AAF70C3.1000800@irontec.com> Hi I will delete this line of the config. I will make probes. About the comercial support, I think that this issue is "basic" for the proper working of a production and seriour enviroment of rsyslog. In the future, if we want an especialezed support we call you for support, sure!! :) So any solution for this? UDP? Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 12:14 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> But rsyslog starts... >> >> If I use UDP instead TCP? >> > > sure, because the messages are thrown away, at least I think. The point is: > it doesn't make sense to hunt for a problem as long as we know that the > config is incorrect. Better get the config clean first, then see if the > problem even persists and then look at it. > > Bluntly and not meant to be embarrassing: I've set aside some time to do this > kind of support, but if you need more "full service" help, it would probably > be a good idea to purchase one of the support packages. They exists so that > we can look at issues in depth. This is often an excellent values, as it may > safe you hours and hours of work. And, really, I can't develop all this and > provide this kind of full-service support ;) > > Rainer > > >> Rainer Gerhards wrote: >> >>> Ok, there are errors in the config files. I've stopped looking at >>> >> them when I >> >>> saw >>> >>> EST=... >>> >>> ... @@$EST >>> >>> This does not work in rsyslog (yet). Please make sure that your >>> >> configs are >> >>> OK. With the versions you have, you either need to start rsyslogd >>> interactively in debug mode OR simply look at the syslogd logs (those >>> >> with >> >>> syslog facility). >>> >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] server frozen when remote logging >>>> >>>> Ok Rainer >>>> >>>> In the clients: >>>> >>>> OS= opensuse 10.0 >>>> rsyslog version: 3.19.7 >>>> >>>> In the server >>>> OS=Debian 4.0 >>>> rsyslog version: 3.18.2 >>>> >>>> I attach the configuration files of the clients and the servers. >>>> >>>> The remote server is 192.1.4.215. >>>> >>>> Thanks >>>> >>>> Rainer Gerhards wrote: >>>> >>>> >>>>> This sounds like you are overdoing "reliable delivery". But I need >>>>> >>>>> >>>> configs >>>> >>>> >>>>> and version information to tell you what may be the case. If it is >>>>> >> an >> >>>> older >>>> >>>> >>>>> v3 version, this may also be a bug in rsyslog. >>>>> >>>>> HTH >>>>> Rainer >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>> To: rsyslog at lists.adiscon.com >>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>> >>>>>> Hi!! >>>>>> >>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>> experimented the kaos!! >>>>>> >>>>>> Accidentaly the central server shutdowns, and one hour later, all >>>>>> >>>>>> >>>> the >>>> >>>> >>>>>> 80 >>>>>> servers frezze. >>>>>> >>>>>> Can not access ssh, ping... >>>>>> >>>>>> I use Debian in central server, and suse in nodes. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> Mikel Jimenez Fernandez >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>> >>>>>> >>>> http://www.irontec.com >>>> >>>> >>>>>> +34 94.404.81.82 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:52:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:52:27 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> <4AAF70C3.1000800@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Hi > > I will delete this line of the config. > I will make probes. > About the comercial support, I think that this issue is "basic" for > the > proper working of a production and seriour enviroment of rsyslog. Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in bug reports is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. > > In the future, if we want an especialezed support we call you for > support, sure!! :) > > > So any solution for this? > UDP? Anyhow, does that mean your config is now error-free and the problem still persists? Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 12:14 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> But rsyslog starts... > >> > >> If I use UDP instead TCP? > >> > > > > sure, because the messages are thrown away, at least I think. The > point is: > > it doesn't make sense to hunt for a problem as long as we know that > the > > config is incorrect. Better get the config clean first, then see if > the > > problem even persists and then look at it. > > > > Bluntly and not meant to be embarrassing: I've set aside some time to > do this > > kind of support, but if you need more "full service" help, it would > probably > > be a good idea to purchase one of the support packages. They exists > so that > > we can look at issues in depth. This is often an excellent values, as > it may > > safe you hours and hours of work. And, really, I can't develop all > this and > > provide this kind of full-service support ;) > > > > Rainer > > > > > >> Rainer Gerhards wrote: > >> > >>> Ok, there are errors in the config files. I've stopped looking at > >>> > >> them when I > >> > >>> saw > >>> > >>> EST=... > >>> > >>> ... @@$EST > >>> > >>> This does not work in rsyslog (yet). Please make sure that your > >>> > >> configs are > >> > >>> OK. With the versions you have, you either need to start rsyslogd > >>> interactively in debug mode OR simply look at the syslogd logs > (those > >>> > >> with > >> > >>> syslog facility). > >>> > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] server frozen when remote logging > >>>> > >>>> Ok Rainer > >>>> > >>>> In the clients: > >>>> > >>>> OS= opensuse 10.0 > >>>> rsyslog version: 3.19.7 > >>>> > >>>> In the server > >>>> OS=Debian 4.0 > >>>> rsyslog version: 3.18.2 > >>>> > >>>> I attach the configuration files of the clients and the servers. > >>>> > >>>> The remote server is 192.1.4.215. > >>>> > >>>> Thanks > >>>> > >>>> Rainer Gerhards wrote: > >>>> > >>>> > >>>>> This sounds like you are overdoing "reliable delivery". But I > need > >>>>> > >>>>> > >>>> configs > >>>> > >>>> > >>>>> and version information to tell you what may be the case. If it > is > >>>>> > >> an > >> > >>>> older > >>>> > >>>> > >>>>> v3 version, this may also be a bug in rsyslog. > >>>>> > >>>>> HTH > >>>>> Rainer > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>>>> To: rsyslog at lists.adiscon.com > >>>>>> Subject: [rsyslog] server frozen when remote logging > >>>>>> > >>>>>> Hi!! > >>>>>> > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>>>> experimented the kaos!! > >>>>>> > >>>>>> Accidentaly the central server shutdowns, and one hour later, > all > >>>>>> > >>>>>> > >>>> the > >>>> > >>>> > >>>>>> 80 > >>>>>> servers frezze. > >>>>>> > >>>>>> Can not access ssh, ping... > >>>>>> > >>>>>> I use Debian in central server, and suse in nodes. > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> -- > >>>>>> Mikel Jimenez Fernandez > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>>>> > >>>>>> > >>>> http://www.irontec.com > >>>> > >>>> > >>>>>> +34 94.404.81.82 > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>>> > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>>> > >>>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:54:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:54:24 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Used the wrong words ;) Of course, this should read: Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in support is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. And: why is my interest limited? Support to get someone else going contributes almost nothing back to the project... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > Sent: Tuesday, September 15, 2009 12:48 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > Hi > > > > I will delete this line of the config. > > I will make probes. > > About the comercial support, I think that this issue is "basic" for > > the > > proper working of a production and seriour enviroment of rsyslog. > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in bug reports is *extremely limited*. Still, there is the > rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > > > > In the future, if we want an especialezed support we call you for > > support, sure!! :) > > > > > > So any solution for this? > > UDP? > > Anyhow, does that mean your config is now error-free and the problem > still > persists? > > Rainer > > > > Rainer Gerhards wrote: > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] server frozen when remote logging > > >> > > >> But rsyslog starts... > > >> > > >> If I use UDP instead TCP? > > >> > > > > > > sure, because the messages are thrown away, at least I think. The > > point is: > > > it doesn't make sense to hunt for a problem as long as we know that > > the > > > config is incorrect. Better get the config clean first, then see if > > the > > > problem even persists and then look at it. > > > > > > Bluntly and not meant to be embarrassing: I've set aside some time > to > > do this > > > kind of support, but if you need more "full service" help, it would > > probably > > > be a good idea to purchase one of the support packages. They exists > > so that > > > we can look at issues in depth. This is often an excellent values, > as > > it may > > > safe you hours and hours of work. And, really, I can't develop all > > this and > > > provide this kind of full-service support ;) > > > > > > Rainer > > > > > > > > >> Rainer Gerhards wrote: > > >> > > >>> Ok, there are errors in the config files. I've stopped looking at > > >>> > > >> them when I > > >> > > >>> saw > > >>> > > >>> EST=... > > >>> > > >>> ... @@$EST > > >>> > > >>> This does not work in rsyslog (yet). Please make sure that your > > >>> > > >> configs are > > >> > > >>> OK. With the versions you have, you either need to start rsyslogd > > >>> interactively in debug mode OR simply look at the syslogd logs > > (those > > >>> > > >> with > > >> > > >>> syslog facility). > > >>> > > >>> Rainer > > >>> > > >>> > > >>> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > >>>> To: rsyslog-users > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > >>>> > > >>>> Ok Rainer > > >>>> > > >>>> In the clients: > > >>>> > > >>>> OS= opensuse 10.0 > > >>>> rsyslog version: 3.19.7 > > >>>> > > >>>> In the server > > >>>> OS=Debian 4.0 > > >>>> rsyslog version: 3.18.2 > > >>>> > > >>>> I attach the configuration files of the clients and the servers. > > >>>> > > >>>> The remote server is 192.1.4.215. > > >>>> > > >>>> Thanks > > >>>> > > >>>> Rainer Gerhards wrote: > > >>>> > > >>>> > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > need > > >>>>> > > >>>>> > > >>>> configs > > >>>> > > >>>> > > >>>>> and version information to tell you what may be the case. If it > > is > > >>>>> > > >> an > > >> > > >>>> older > > >>>> > > >>>> > > >>>>> v3 version, this may also be a bug in rsyslog. > > >>>>> > > >>>>> HTH > > >>>>> Rainer > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > >>>>>> To: rsyslog at lists.adiscon.com > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > >>>>>> > > >>>>>> Hi!! > > >>>>>> > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > > >>>>>> experimented the kaos!! > > >>>>>> > > >>>>>> Accidentaly the central server shutdowns, and one hour later, > > all > > >>>>>> > > >>>>>> > > >>>> the > > >>>> > > >>>> > > >>>>>> 80 > > >>>>>> servers frezze. > > >>>>>> > > >>>>>> Can not access ssh, ping... > > >>>>>> > > >>>>>> I use Debian in central server, and suse in nodes. > > >>>>>> > > >>>>>> Thanks! > > >>>>>> > > >>>>>> -- > > >>>>>> Mikel Jimenez Fernandez > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>>>> > > >>>>>> > > >>>> http://www.irontec.com > > >>>> > > >>>> > > >>>>>> +34 94.404.81.82 > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>> _______________________________________________ > > >>>>> rsyslog mailing list > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>> http://www.rsyslog.com > > >>>>> > > >>>>> > > >>>>> > > >>>> -- > > >>>> Mikel Jimenez Fernandez > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>> > > >> http://www.irontec.com > > >> > > >>>> +34 94.404.81.82 > > >>>> > > >>>> > > >>>> > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >>> > > >> -- > > >> Mikel Jimenez Fernandez > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > http://www.irontec.com > > >> +34 94.404.81.82 > > >> > > >> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > > > > > -- > > Mikel Jimenez Fernandez > > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > > +34 94.404.81.82 > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:55:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:55:29 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDB@GRFEXC.intern.adiscon.com> and a bit more on the philosophy, I knew I wrote it down ;) http://www.rsyslog.com/doc-free_support.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 15, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > Sent: Tuesday, September 15, 2009 12:48 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > > Hi > > > > > > I will delete this line of the config. > > > I will make probes. > > > About the comercial support, I think that this issue is "basic" > for > > > the > > > proper working of a production and seriour enviroment of rsyslog. > > > > Sure, but let me phrase it that way: My interest is finding bugs, > > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in bug reports is *extremely limited*. Still, there is the > > rest of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > > save > > time while doing so. > > > > > > > > In the future, if we want an especialezed support we call you for > > > support, sure!! :) > > > > > > > > > So any solution for this? > > > UDP? > > > > Anyhow, does that mean your config is now error-free and the problem > > still > > persists? > > > > Rainer > > > > > > Rainer Gerhards wrote: > > > >> -----Original Message----- > > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > > >> To: rsyslog-users > > > >> Subject: Re: [rsyslog] server frozen when remote logging > > > >> > > > >> But rsyslog starts... > > > >> > > > >> If I use UDP instead TCP? > > > >> > > > > > > > > sure, because the messages are thrown away, at least I think. The > > > point is: > > > > it doesn't make sense to hunt for a problem as long as we know > that > > > the > > > > config is incorrect. Better get the config clean first, then see > if > > > the > > > > problem even persists and then look at it. > > > > > > > > Bluntly and not meant to be embarrassing: I've set aside some > time > > to > > > do this > > > > kind of support, but if you need more "full service" help, it > would > > > probably > > > > be a good idea to purchase one of the support packages. They > exists > > > so that > > > > we can look at issues in depth. This is often an excellent > values, > > as > > > it may > > > > safe you hours and hours of work. And, really, I can't develop > all > > > this and > > > > provide this kind of full-service support ;) > > > > > > > > Rainer > > > > > > > > > > > >> Rainer Gerhards wrote: > > > >> > > > >>> Ok, there are errors in the config files. I've stopped looking > at > > > >>> > > > >> them when I > > > >> > > > >>> saw > > > >>> > > > >>> EST=... > > > >>> > > > >>> ... @@$EST > > > >>> > > > >>> This does not work in rsyslog (yet). Please make sure that your > > > >>> > > > >> configs are > > > >> > > > >>> OK. With the versions you have, you either need to start > rsyslogd > > > >>> interactively in debug mode OR simply look at the syslogd logs > > > (those > > > >>> > > > >> with > > > >> > > > >>> syslog facility). > > > >>> > > > >>> Rainer > > > >>> > > > >>> > > > >>> > > > >>>> -----Original Message----- > > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > > >>>> To: rsyslog-users > > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > > >>>> > > > >>>> Ok Rainer > > > >>>> > > > >>>> In the clients: > > > >>>> > > > >>>> OS= opensuse 10.0 > > > >>>> rsyslog version: 3.19.7 > > > >>>> > > > >>>> In the server > > > >>>> OS=Debian 4.0 > > > >>>> rsyslog version: 3.18.2 > > > >>>> > > > >>>> I attach the configuration files of the clients and the > servers. > > > >>>> > > > >>>> The remote server is 192.1.4.215. > > > >>>> > > > >>>> Thanks > > > >>>> > > > >>>> Rainer Gerhards wrote: > > > >>>> > > > >>>> > > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > > need > > > >>>>> > > > >>>>> > > > >>>> configs > > > >>>> > > > >>>> > > > >>>>> and version information to tell you what may be the case. If > it > > > is > > > >>>>> > > > >> an > > > >> > > > >>>> older > > > >>>> > > > >>>> > > > >>>>> v3 version, this may also be a bug in rsyslog. > > > >>>>> > > > >>>>> HTH > > > >>>>> Rainer > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > > >>>>>> To: rsyslog at lists.adiscon.com > > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > > >>>>>> > > > >>>>>> Hi!! > > > >>>>>> > > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I > have > > > >>>>>> experimented the kaos!! > > > >>>>>> > > > >>>>>> Accidentaly the central server shutdowns, and one hour > later, > > > all > > > >>>>>> > > > >>>>>> > > > >>>> the > > > >>>> > > > >>>> > > > >>>>>> 80 > > > >>>>>> servers frezze. > > > >>>>>> > > > >>>>>> Can not access ssh, ping... > > > >>>>>> > > > >>>>>> I use Debian in central server, and suse in nodes. > > > >>>>>> > > > >>>>>> Thanks! > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Mikel Jimenez Fernandez > > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>>>> > > > >>>>>> > > > >>>> http://www.irontec.com > > > >>>> > > > >>>> > > > >>>>>> +34 94.404.81.82 > > > >>>>>> > > > >>>>>> > > > >>>>>> _______________________________________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>> http://www.rsyslog.com > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> _______________________________________________ > > > >>>>> rsyslog mailing list > > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>> http://www.rsyslog.com > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>> -- > > > >>>> Mikel Jimenez Fernandez > > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>> > > > >> http://www.irontec.com > > > >> > > > >>>> +34 94.404.81.82 > > > >>>> > > > >>>> > > > >>>> > > > >>> _______________________________________________ > > > >>> rsyslog mailing list > > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>> http://www.rsyslog.com > > > >>> > > > >>> > > > >> -- > > > >> Mikel Jimenez Fernandez > > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > http://www.irontec.com > > > >> +34 94.404.81.82 > > > >> > > > >> > > > >> _______________________________________________ > > > >> rsyslog mailing list > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> http://www.rsyslog.com > > > >> > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > > > > > > > -- > > > Mikel Jimenez Fernandez > > > Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > > > +34 94.404.81.82 > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 13:09:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:09:02 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF75CE.7090605@irontec.com> Rainer Gerhards wrote: > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > Im going to make probes with deleting the config line EST=... When we have coclusion I will tell you, and I will back my concluison to this magic project. :) Thanks > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 15, 2009 12:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>> Sent: Tuesday, September 15, 2009 12:48 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] server frozen when remote logging >>> >>> Hi >>> >>> I will delete this line of the config. >>> I will make probes. >>> About the comercial support, I think that this issue is "basic" for >>> the >>> proper working of a production and seriour enviroment of rsyslog. >>> >> Sure, but let me phrase it that way: My interest is finding bugs, >> support >> questions often lead to that. If there is no bug involved, my personal >> interest in bug reports is *extremely limited*. Still, there is the >> rest of >> the community, and they often provide advice. So Adiscon created the >> commercial support for corporations that want to have a solution and >> save >> time while doing so. >> >> >>> In the future, if we want an especialezed support we call you for >>> support, sure!! :) >>> >>> >>> So any solution for this? >>> UDP? >>> >> Anyhow, does that mean your config is now error-free and the problem >> still >> persists? >> >> Rainer >> >>> Rainer Gerhards wrote: >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>> Sent: Tuesday, September 15, 2009 12:14 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>> >>>>> But rsyslog starts... >>>>> >>>>> If I use UDP instead TCP? >>>>> >>>>> >>>> sure, because the messages are thrown away, at least I think. The >>>> >>> point is: >>> >>>> it doesn't make sense to hunt for a problem as long as we know that >>>> >>> the >>> >>>> config is incorrect. Better get the config clean first, then see if >>>> >>> the >>> >>>> problem even persists and then look at it. >>>> >>>> Bluntly and not meant to be embarrassing: I've set aside some time >>>> >> to >> >>> do this >>> >>>> kind of support, but if you need more "full service" help, it would >>>> >>> probably >>> >>>> be a good idea to purchase one of the support packages. They exists >>>> >>> so that >>> >>>> we can look at issues in depth. This is often an excellent values, >>>> >> as >> >>> it may >>> >>>> safe you hours and hours of work. And, really, I can't develop all >>>> >>> this and >>> >>>> provide this kind of full-service support ;) >>>> >>>> Rainer >>>> >>>> >>>> >>>>> Rainer Gerhards wrote: >>>>> >>>>> >>>>>> Ok, there are errors in the config files. I've stopped looking at >>>>>> >>>>>> >>>>> them when I >>>>> >>>>> >>>>>> saw >>>>>> >>>>>> EST=... >>>>>> >>>>>> ... @@$EST >>>>>> >>>>>> This does not work in rsyslog (yet). Please make sure that your >>>>>> >>>>>> >>>>> configs are >>>>> >>>>> >>>>>> OK. With the versions you have, you either need to start rsyslogd >>>>>> interactively in debug mode OR simply look at the syslogd logs >>>>>> >>> (those >>> >>>>> with >>>>> >>>>> >>>>>> syslog facility). >>>>>> >>>>>> Rainer >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>>>> >>>>>>> Ok Rainer >>>>>>> >>>>>>> In the clients: >>>>>>> >>>>>>> OS= opensuse 10.0 >>>>>>> rsyslog version: 3.19.7 >>>>>>> >>>>>>> In the server >>>>>>> OS=Debian 4.0 >>>>>>> rsyslog version: 3.18.2 >>>>>>> >>>>>>> I attach the configuration files of the clients and the servers. >>>>>>> >>>>>>> The remote server is 192.1.4.215. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Rainer Gerhards wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> This sounds like you are overdoing "reliable delivery". But I >>>>>>>> >>> need >>> >>>>>>>> >>>>>>> configs >>>>>>> >>>>>>> >>>>>>> >>>>>>>> and version information to tell you what may be the case. If it >>>>>>>> >>> is >>> >>>>> an >>>>> >>>>> >>>>>>> older >>>>>>> >>>>>>> >>>>>>> >>>>>>>> v3 version, this may also be a bug in rsyslog. >>>>>>>> >>>>>>>> HTH >>>>>>>> Rainer >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>>>>> To: rsyslog at lists.adiscon.com >>>>>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>>>>> >>>>>>>>> Hi!! >>>>>>>>> >>>>>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>>>>> experimented the kaos!! >>>>>>>>> >>>>>>>>> Accidentaly the central server shutdowns, and one hour later, >>>>>>>>> >>> all >>> >>>>>>>>> >>>>>>> the >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> 80 >>>>>>>>> servers frezze. >>>>>>>>> >>>>>>>>> Can not access ssh, ping... >>>>>>>>> >>>>>>>>> I use Debian in central server, and suse in nodes. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Mikel Jimenez Fernandez >>>>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> http://www.irontec.com >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> +34 94.404.81.82 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Mikel Jimenez Fernandez >>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>> >>>>>>> >>>>> http://www.irontec.com >>>>> >>>>> >>>>>>> +34 94.404.81.82 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Mikel Jimenez Fernandez >>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>> >>> http://www.irontec.com >>> >>>>> +34 94.404.81.82 >>>>> >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> -- >>> Mikel Jimenez Fernandez >>> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >>> +34 94.404.81.82 >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 13:44:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 13:44:06 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> > Rainer Gerhards wrote: > > Used the wrong words ;) Of course, this should read: > > > > Sure, but let me phrase it that way: My interest is finding bugs, > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in support is *extremely limited*. Still, there is the rest > of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > save > > time while doing so. > > > > And: why is my interest limited? Support to get someone else going > > contributes almost nothing back to the project... > > > Im going to make probes with deleting the config line EST=... > > When we have coclusion I will tell you, and I will back my concluison > to > this magic project. :) I have taken another look at the log files in the meantime, assuming that $EST were not present ;) However, I do not see anything obviously wrong. But I think I remember there was a condition that caused messages to be processed to slowly. Probably the best idea is to see if the issue persists with the current v3-stable release. If it does, we should go to v4 and if it still persists we need to obtain debug logs. But I think chances are extremely high that the current v3-stable will solve it. however, those forwarding rules to $EST can never have worked, and may actually be overruning the retry mechanism after a while... Rainer From mikel at irontec.com Tue Sep 15 13:46:27 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:46:27 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> Message-ID: <4AAF7E93.1080309@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: >> Rainer Gerhards wrote: >> >>> Used the wrong words ;) Of course, this should read: >>> >>> Sure, but let me phrase it that way: My interest is finding bugs, >>> >> support >> >>> questions often lead to that. If there is no bug involved, my >>> >> personal >> >>> interest in support is *extremely limited*. Still, there is the rest >>> >> of >> >>> the community, and they often provide advice. So Adiscon created the >>> commercial support for corporations that want to have a solution and >>> >> save >> >>> time while doing so. >>> >>> And: why is my interest limited? Support to get someone else going >>> contributes almost nothing back to the project... >>> >>> >> Im going to make probes with deleting the config line EST=... >> >> When we have coclusion I will tell you, and I will back my concluison >> to >> this magic project. :) >> > > I have taken another look at the log files in the meantime, assuming that > $EST were not present ;) However, I do not see anything obviously wrong. But > I think I remember there was a condition that caused messages to be processed > to slowly. Probably the best idea is to see if the issue persists with the > current v3-stable release. If it does, we should go to v4 and if it still > persists we need to obtain debug logs. But I think chances are extremely high > that the current v3-stable will solve it. > > however, those forwarding rules to $EST can never have worked, and may > actually be overruning the retry mechanism after a while... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From tbergfeld at hq.adiscon.com Mon Sep 21 08:12:22 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 21 Sep 2009 08:12:22 +0200 Subject: [rsyslog] rsyslog 4.5.3 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FF73@GRFEXC.intern.adiscon.com> Rsyslog 4.5.3, a member of the v4- beta branch, has been released. It is a bug-fixing release. Most importantly, a bug that repeated messages were incorrectly processed by what it could lead to loss of the repeated message content, was fixed. As a side- effect, it could probably also be possible that some segfault occurs (quite unlikely). The root cause was that some counters introduced during the malloc optimizations were not properly duplicated in MsgDup(). Note that repeated message processing is not enabled by default. See Changelog for more details. This is a recommended update for all users of the beta branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-176.phtml Changelog: http://www.rsyslog.com/Article404.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From anichols at trumped.org Tue Sep 22 03:50:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 21 Sep 2009 19:50:39 -0600 Subject: [rsyslog] Improving filter performance & general performance Message-ID: Hi Everyone, I have rsyslog 4.4.1 chugging along reasonably well but am looking for ways to improve performance and optimize the filter ruleset. Unfortunately I have to create fairly extensive rulesets to filter on hostname, programname, facility, priority, etc. Some log sources generate a high volume of logs (a few Mbytes/sec) across multiple machines and others generate a fairly routine amount of log data - maybe 5 meg per day. Many filters have duplicate conditions for some values but there is always variance. I have tried to order the rules so that the highest volume logs match first and then are discarded. I've included a sample of the rules used for my highest volume logs (names changed to protect the innocent). If there are ways to chain or nest rules so that I can take advantage of matches already made against a log entry to filter it minimally that would be great. For example, most of the below rules filter on the same facility & list of hostnames but look for different values in the 'rawmsg'. If I could filter on the facility & hostname once and then rawmsg to sort to different destinations I'm guessing it would be lower overhead but I don't really know how the processing logic works. Also - if a condition is not met, are other parts of the filter evaluated? For example, if a message was received on local0, would any conditions beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more efficient to filter on the undecoded value syslogfacility vs. syslogfacility-text? I'm looking for suggestions or general techniques for optimizing rule performance under these circumstances. $template XMLFormat, "%syslogtag%%msg%\n" if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'protocolLogRecord' then -/log/syslog/collated/server/protocol.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'messageLogRecord' then -/log/syslog/collated/server/message.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'clientLogRecord' then -/log/syslog/collated/server/client.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local2' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) then -/log/syslog/collated/server/usage.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'WAP Page Service ID' then -/log/syslog/collated/server/customer-service;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'locationlogrecord' then -/log/syslog/collated/server/lbs.log;XMLFormat & ~ # discard after match From rgerhards at hq.adiscon.com Tue Sep 22 07:23:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Sep 2009 07:23:43 +0200 Subject: [rsyslog] Improving filter performance & general performance References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FFA1@GRFEXC.intern.adiscon.com> Sorry, I am swamped with fixing an important segfault issue we see in one environment, so I do not have time for an more in-depth answer (other list members may have). But I suggest to look into multiple ruleset support, which is in its infancy, but may help. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, September 22, 2009 3:51 AM > To: rsyslog-users > Subject: [rsyslog] Improving filter performance & general performance > > Hi Everyone, > > I have rsyslog 4.4.1 chugging along reasonably well but am > looking for ways > to improve performance and optimize the filter ruleset. > Unfortunately I have > to create fairly extensive rulesets to filter on hostname, > programname, > facility, priority, etc. Some log sources generate a high > volume of logs (a > few Mbytes/sec) across multiple machines and others generate a fairly > routine amount of log data - maybe 5 meg per day. Many filters have > duplicate conditions for some values but there is always > variance. I have > tried to order the rules so that the highest volume logs > match first and > then are discarded. I've included a sample of the rules used > for my highest > volume logs (names changed to protect the innocent). > > If there are ways to chain or nest rules so that I can take > advantage of > matches already made against a log entry to filter it > minimally that would > be great. For example, most of the below rules filter on the > same facility & > list of hostnames but look for different values in the > 'rawmsg'. If I could > filter on the facility & hostname once and then rawmsg to > sort to different > destinations I'm guessing it would be lower overhead but I > don't really know > how the processing logic works. > > Also - if a condition is not met, are other parts of the > filter evaluated? > For example, if a message was received on local0, would any conditions > beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more > efficient to filter on the undecoded value syslogfacility vs. > syslogfacility-text? > > I'm looking for suggestions or general techniques for optimizing rule > performance under these circumstances. > > $template XMLFormat, "%syslogtag%%msg%\n" > > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'protocolLogRecord' then > -/log/syslog/collated/server/protocol.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'messageLogRecord' then > -/log/syslog/collated/server/message.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'clientLogRecord' then > -/log/syslog/collated/server/client.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local2' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) then -/log/syslog/collated/server/usage.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'WAP Page Service ID' then > -/log/syslog/collated/server/customer-service;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'locationlogrecord' then > -/log/syslog/collated/server/lbs.log;XMLFormat > & ~ # discard after match > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 14:54:38 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 07:54:38 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error Message-ID: <20090925125437.GA28679@it.is.rice.edu> I just looked at our PostgreSQL DB for our rsyslog system and the following error was logged: ERROR: value too long for type character varying(60) STATEMENT: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, '////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////') The problem is not so much the error but that it stopped logging to the database. I had to restart rsyslog to get it to start logging once more. Should rsyslog check that its values match the schema or should I need to setup a trigger in the DB to handle off-the-wall input. Regards, Ken From rgerhards at hq.adiscon.com Fri Sep 25 15:30:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:30:58 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Actually, it should have dropped this message, but that depends on the configuration. In general, rsyslog does not know about the schema. And to be more precise, we are not really talking about rsyslogd itself but rather the output plugin. Every output plugin can perform its own checks. But the best answer probably is to use a trigger ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 2:55 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > I just looked at our PostgreSQL DB for our rsyslog system and > the following error was logged: > > ERROR: value too long for type character varying(60) > STATEMENT: insert into SystemEvents (Message, Facility, > FromHost, Priority, DeviceReportedTime, ReceivedAt, > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > '///////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > /////////////////////////////////////////////////////////////////') > > The problem is not so much the error but that it stopped logging > to the database. I had to restart rsyslog to get it to start logging > once more. Should rsyslog check that its values match the schema or > should I need to setup a trigger in the DB to handle off-the-wall > input. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 15:41:28 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 08:41:28 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> References: <20090925125437.GA28679@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Message-ID: <20090925134128.GB28679@it.is.rice.edu> Okay, I will take a look at the output plugin to see where it makes the most sense to fix this. A trigger will always work, but would require every DB to setup and maybe having the plugin perform the truncation would be better. Thank you for the recommendation. Regards, Ken On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > Actually, it should have dropped this message, but that depends on the > configuration. In general, rsyslog does not know about the schema. And to be > more precise, we are not really talking about rsyslogd itself but rather the > output plugin. Every output plugin can perform its own checks. > > But the best answer probably is to use a trigger ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > Kenneth Marshall > > Sent: Friday, September 25, 2009 2:55 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > the following error was logged: > > > > ERROR: value too long for type character varying(60) > > STATEMENT: insert into SystemEvents (Message, Facility, > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > '///////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > /////////////////////////////////////////////////////////////////') > > > > The problem is not so much the error but that it stopped logging > > to the database. I had to restart rsyslog to get it to start logging > > once more. Should rsyslog check that its values match the schema or > > should I need to setup a trigger in the DB to handle off-the-wall > > input. > > > > Regards, > > Ken > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 25 15:43:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:43:36 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> <20090925134128.GB28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BE@GRFEXC.intern.adiscon.com> Ken, The postgres output is quite simple. You may also want to have a look at omoracle, just to see how flexible an output plugin is (postgres was contributed, as was oracle, btw). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 3:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog bug - logging stops after a DB error > > Okay, I will take a look at the output plugin to see where it > makes the most sense to fix this. A trigger will always work, > but would require every DB to setup and maybe having the plugin > perform the truncation would be better. Thank you for the > recommendation. > > Regards, > Ken > > On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > > Actually, it should have dropped this message, but that > depends on the > > configuration. In general, rsyslog does not know about the > schema. And to be > > more precise, we are not really talking about rsyslogd > itself but rather the > > output plugin. Every output plugin can perform its own checks. > > > > But the best answer probably is to use a trigger ;) > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > Kenneth Marshall > > > Sent: Friday, September 25, 2009 2:55 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > > the following error was logged: > > > > > > ERROR: value too long for type character varying(60) > > > STATEMENT: insert into SystemEvents (Message, Facility, > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > > '///////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > > /////////////////////////////////////////////////////////////////') > > > > > > The problem is not so much the error but that it stopped logging > > > to the database. I had to restart rsyslog to get it to > start logging > > > once more. Should rsyslog check that its values match the > schema or > > > should I need to setup a trigger in the DB to handle off-the-wall > > > input. > > > > > > Regards, > > > Ken > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 29 09:59:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Sep 2009 09:59:14 +0200 Subject: [rsyslog] rsyslog 4.5.4 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030E3@GRFEXC.intern.adiscon.com> Hi all, I have just released 4.5.4, a member of the v4-beta branch. This beta contains an important fix that can lead to a segfault when the gzip output writer is used. It also contains some other fixes. Users of v4-beta are strongly advised to upgrade to that version. ChangeLog: http://www.rsyslog.com/Article406.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-177.phtml I hope this release is useful, Rainer From rgerhards at hq.adiscon.com Wed Sep 30 16:31:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 16:31:27 +0200 Subject: [rsyslog] DNS cache and expiration Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Hi all, I think we had discussed this some time in the past, but I cannot find a record of it. So I thought I ask (again?): After my bughunt looks almost completed, I have come back to implementing the name lookup cache. However, I just found out that obtaining the expiration period of the name lookup seems not to be covered by the "usual" socket calls. Or did I just miss them? Any advise, comments and hints regarding name caching and expiration would deeply be appreciated. Rainer From aland at freeradius.org Wed Sep 30 17:44:26 2009 From: aland at freeradius.org (Alan T DeKok) Date: Wed, 30 Sep 2009 17:44:26 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4AC37CDA.9040707@freeradius.org> Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? You didn't miss anything. They're not available. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. You'll have to use a more powerful DNS library, like adns. Alan DeKok. From aoz.syn at gmail.com Wed Sep 30 17:55:09 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 09:55:09 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? Unfortunately not - most resolver libraries provide only what the programmer usually wants - the symbolic (name) or numeric (IP) result of a query. I've not looked carefully at APIs like res_query, though, and that might bring what you need. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. This was my greatest concern with doing *good* internal caching in rsyslog - you're almost guaranteed to use and/or implement a large chunk of proper resolver functionality. Depending on how readable you find Perl, the Net::DNS infrastructure may provide some good pointers on implementing custom resolution toolkits. The djbdns 'dnscache' program (and perhaps the djbdns client resolver library itself) could also be good pointers. From rgerhards at hq.adiscon.com Wed Sep 30 18:52:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 18:52:44 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Thanks for the quick responses, this was what I feared. In the mean time, I have thought a bit about the design. I think I will start not with the cache, but rather by checking to see if I can move the reverse name resolution further down in the processing flow AND move it to one central location. That makes it easier and more efficient to do caching. One drawback when doing so is that the name resolution potentially happens much later than the message reception. Just think about a busy system, or even one waiting for an upstream server to come online again, that lacks behind some minutes or even some hours. When I do the name resolution in the backend thread, the reverse entries may have changed since the message was received :( For many cases, this may be acceptable, for some not. I will probably need to at least define a config value which enables direct queries vs. deferred ones. Any comments on that issue would also be most welcome. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, September 30, 2009 5:55 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards > wrote: > > After my bughunt looks almost completed, I have come back to > implementing the > > name lookup cache. However, I just found out that obtaining the > expiration > > period of the name lookup seems not to be covered by the "usual" > socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration > would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 30 19:00:58 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:00:58 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Hi all, > > I think we had discussed this some time in the past, but I cannot find a > record of it. So I thought I ask (again?): > > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? no they don't, they are name resolution calls, not DNS calls. there are many sources of name resolution (/etc/hosts, LDAP, wins, NIS, etc) and most of them do not have a concept of expiration, and those that do have specific rules for how expiration works (for DNS you have a time after which you are supposed to try and re-resolve it, but can continue to use the name, and a different time after which you are not supposed to use the name for example) > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. going back to basics here. Why is this feature desired? why not just use a caching nameserver listening on localhost? 1. doing a name lookup can cause lost logs as you saw when you were doing testing recently, doing name lookups on each received UDP log can cause you to loose log messages when the OS can no longer queue them up. 2. throughput in a high volume site, the cost of doing a name lookup for each log message can be high enough to be a problem. even a local nameserver can be expensive if you are dealing with 10's of thousands of messages/sec what if you were to move the name resolution from the input module to the output module? that would solve problem #1 immediatly by just eliminating any lookups as the messages are received. note: this may not be possible due to name based rules for what hosts to accept logs from, although the answer here may be to lookup the names when you startup and do the filtering by IP while running. if you delay the name resolution until the output module, you may be able to only do it if the output module needs it (if it uses a name property in the template or ruleset), and if it doesn't you skip the work entirely. in anything short of a very high volume site a local caching nameserver will satisfy the throughput issue nicely (especially if the name resolution is delayed to the output as I mentioned above). in a high volume site I really think that it can be good enough to just throw away the name cache when you do a HUP. a high volume site is going to be doing a HUP on a frequent basis anyway to rotate the logs. This avoids a LOT of overhead and complications in managing expirations. In my site I send rsyslog a HUP every 5 min currently (and have some cases where I plan to change this to every 1 min in the near future) David Lang From david at lang.hm Wed Sep 30 19:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:05:04 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the quick responses, this was what I feared. > > In the mean time, I have thought a bit about the design. I think I will start > not with the cache, but rather by checking to see if I can move the reverse > name resolution further down in the processing flow AND move it to one > central location. That makes it easier and more efficient to do caching. > > One drawback when doing so is that the name resolution potentially happens > much later than the message reception. Just think about a busy system, or > even one waiting for an upstream server to come online again, that lacks > behind some minutes or even some hours. When I do the name resolution in the > backend thread, the reverse entries may have changed since the message was > received :( > > For many cases, this may be acceptable, for some not. I will probably need to > at least define a config value which enables direct queries vs. deferred > ones. > > Any comments on that issue would also be most welcome. it is actually pretty unusual for the source of logs to change it's name. remember that DNS takes time to propogate changes, so even if you do queries immediatly the data may be out of date if you are in an environemnt where it changes. David Lang > Thanks, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, September 30, 2009 5:55 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards >> wrote: >>> After my bughunt looks almost completed, I have come back to >> implementing the >>> name lookup cache. However, I just found out that obtaining the >> expiration >>> period of the name lookup seems not to be covered by the "usual" >> socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration >> would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbiebl at gmail.com Wed Sep 30 19:36:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 30 Sep 2009 19:36:20 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/30 : > > going back to basics here. > > Why is this feature desired? why not just use a caching nameserver > listening on localhost? > Was wondering about this myself. There are small caching nameservers like dnsmasq which will do all the hard work for you. Rainer, have you evaluated such an option? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Sep 30 19:52:07 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:52:07 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Michael Biebl wrote: > 2009/9/30 : >> >> going back to basics here. >> >> Why is this feature desired? why not just use a caching nameserver >> listening on localhost? >> > > Was wondering about this myself. There are small caching nameservers > like dnsmasq which will do all the hard work for you. > Rainer, have you evaluated such an option? by itself, this would not solve the problems 1. with the current situation where the lookups are done as the message is being received, the time taken to do the lookup (especially in the case where the lookup is not yet in the cache) can take long enough that log messages get lost 2. when you are talking message rates of 100K logs/sec and up the overhead of doing a DNS query, even to a server running on localhost that has the info cached in it, becomes a signficant amount of the total time you have to process that message before the next message arrives. David Lang From ktm at rice.edu Wed Sep 30 19:54:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 12:54:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <20090930175456.GD6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > > After my bughunt looks almost completed, I have come back to implementing the > > name lookup cache. However, I just found out that obtaining the expiration > > period of the name lookup seems not to be covered by the "usual" socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. I do not think that the goal of this feature in rsyslog is to re-implement resolver functionality but to provide a fast-path mechanism to map IP addresses to names for the purposes of logging error messages. As such, pretty much the only piece that needs to be tracked within rsyslog is the TTL for the entry and the ip -> name mapping. A thread would be responsible for expiring entries from the cache (or refreshing the timeout) after validating the correctness of the mapping. I think the DNS lookups should be handled by a good resolver like pdns-recursor, djbdns,... The goal here is to allow names in the log entries and not just IP addresses and in a very high performance logging environment. Regards, Ken From david at lang.hm Wed Sep 30 20:15:42 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:15:42 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: >>> After my bughunt looks almost completed, I have come back to implementing the >>> name lookup cache. However, I just found out that obtaining the expiration >>> period of the name lookup seems not to be covered by the "usual" socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. > > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. As such, pretty much the only piece that needs to > be tracked within rsyslog is the TTL for the entry and the ip -> > name mapping. A thread would be responsible for expiring entries > from the cache (or refreshing the timeout) after validating the > correctness of the mapping. I think the DNS lookups should be > handled by a good resolver like pdns-recursor, djbdns,... The > goal here is to allow names in the log entries and not just IP > addresses and in a very high performance logging environment. the trouble is that doing _proper_ TTL expiration isn't as simple as it sounds. and if you are willing to back away from 'proper' expiration to something that will work in practice, why not go much further (as I have detailed in the other messages) David Lang From aoz.syn at gmail.com Wed Sep 30 20:18:37 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 12:18:37 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. Although I agree with your assessment of the goal, the only difference I see between the two is wording semantics. An RFC-compliant DNS cache will, for all intents and purposes, look an awful lot like any other caching, recursive-only DNS resolver (like dnscache). The only major difference would be that it would accept requests via an API as opposed to through a socket interface. Regardless, I have to sit on the same side as David and Michael - in very high-performance environments, I doubt the difference between an internal cache and an external one is going to be significant. From ktm at rice.edu Wed Sep 30 20:25:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 13:25:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <20090930182556.GE6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 11:15:42AM -0700, david at lang.hm wrote: > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > >>> After my bughunt looks almost completed, I have come back to implementing the > >>> name lookup cache. However, I just found out that obtaining the expiration > >>> period of the name lookup seems not to be covered by the "usual" socket > >>> calls. Or did I just miss them? > >> > >> Unfortunately not - most resolver libraries provide only what the > >> programmer usually wants - the symbolic (name) or numeric (IP) result > >> of a query. I've not looked carefully at APIs like res_query, though, > >> and that might bring what you need. > >> > >>> Any advise, comments and hints regarding name caching and expiration would > >>> deeply be appreciated. > >> > >> This was my greatest concern with doing *good* internal caching in > >> rsyslog - you're almost guaranteed to use and/or implement a large > >> chunk of proper resolver functionality. Depending on how readable you > >> find Perl, the Net::DNS infrastructure may provide some good pointers > >> on implementing custom resolution toolkits. The djbdns 'dnscache' > >> program (and perhaps the djbdns client resolver library itself) could > >> also be good pointers. > > > > I do not think that the goal of this feature in rsyslog is to > > re-implement resolver functionality but to provide a fast-path > > mechanism to map IP addresses to names for the purposes of logging > > error messages. As such, pretty much the only piece that needs to > > be tracked within rsyslog is the TTL for the entry and the ip -> > > name mapping. A thread would be responsible for expiring entries > > from the cache (or refreshing the timeout) after validating the > > correctness of the mapping. I think the DNS lookups should be > > handled by a good resolver like pdns-recursor, djbdns,... The > > goal here is to allow names in the log entries and not just IP > > addresses and in a very high performance logging environment. > > the trouble is that doing _proper_ TTL expiration isn't as simple as it > sounds. > > and if you are willing to back away from 'proper' expiration to something > that will work in practice, why not go much further (as I have detailed in > the other messages) > > David Lang I agree. I only mention TTL values as a reasonable upperbound of the refresh check. The advantage is to remove the large pause in logging due to a DNS refresh after a HUP to rsyslog, if that were the only method to flush/refresh. Like you mention, the IPs/names of systems being logged change rarely so we should tune this for speed and not worry about expiration correctness. I do not agree so other statements that an external cache will perform as well as an internal cache. Too many software products that I work with have needed exactly that functionality to support very high levels of performance. Regards, Ken From david at lang.hm Wed Sep 30 20:51:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:51:28 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, RB wrote: > On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: >> I do not think that the goal of this feature in rsyslog is to >> re-implement resolver functionality but to provide a fast-path >> mechanism to map IP addresses to names for the purposes of logging >> error messages. > > Although I agree with your assessment of the goal, the only difference > I see between the two is wording semantics. An RFC-compliant DNS > cache will, for all intents and purposes, look an awful lot like any > other caching, recursive-only DNS resolver (like dnscache). The only > major difference would be that it would accept requests via an API as > opposed to through a socket interface. > > Regardless, I have to sit on the same side as David and Michael - in > very high-performance environments, I doubt the difference between an > internal cache and an external one is going to be significant. actually, I am thinking that in a high-performance environment, the difference between an internal name cache and an external one _is_ significant I just don't think the internal one should be a DNS RFC complient one. if you put everything in /etc/hosts it is faster than doing a query against a local caching server, but it's still significantly slower than looking it up in memory. remember that when you make the gethostbyname() call it has to do a lot of checking to see which name resolver libraries you have configured (which includes checking for the existance of multiple files), then call them in order until it finds the name. if you do a strace of this sometime you will see how much stuff goes on under the covers. skipping all of this is significant at high log rates. David Lang From david at lang.hm Wed Sep 30 20:54:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:54:12 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930182556.GE6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <20090930182556.GE6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>> mechanism to map IP addresses to names for the purposes of logging >>> error messages. As such, pretty much the only piece that needs to >>> be tracked within rsyslog is the TTL for the entry and the ip -> >>> name mapping. A thread would be responsible for expiring entries >>> from the cache (or refreshing the timeout) after validating the >>> correctness of the mapping. I think the DNS lookups should be >>> handled by a good resolver like pdns-recursor, djbdns,... The >>> goal here is to allow names in the log entries and not just IP >>> addresses and in a very high performance logging environment. >> >> the trouble is that doing _proper_ TTL expiration isn't as simple as it >> sounds. >> >> and if you are willing to back away from 'proper' expiration to something >> that will work in practice, why not go much further (as I have detailed in >> the other messages) >> >> David Lang > > I agree. I only mention TTL values as a reasonable upperbound of the > refresh check. The advantage is to remove the large pause in logging > due to a DNS refresh after a HUP to rsyslog, if that were the only > method to flush/refresh. Like you mention, the IPs/names of systems > being logged change rarely so we should tune this for speed and > not worry about expiration correctness. I do not agree so other > statements that an external cache will perform as well as an internal > cache. Too many software products that I work with have needed exactly > that functionality to support very high levels of performance. actually, you could have the cache be configurable in three modes 1. no caching 2. blank the cache on HUP 3. never blank the cache (i.e. require a full restart to clear it) David Lang From rgerhards at hq.adiscon.com Wed Sep 30 21:45:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:45:00 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Thanks for the good discussion. I am lacking somewhat behind, but will review it in depth tomorrow morning. I just wanted to stress the point that an external cache does not really help, much for the reason David mentioned: if you process messages at very high data rates, the context switch overhead involved with any external solution is extremely costly. Also, in the usual cases, I may do several million queries within a few seconds for just a handful of hosts. With an internal cache, the overhead in doing so is very minimal. With an external solution, the overhead in calling the external cache causes a lot of performance degredation, what in the case of UDP also implies (heavy!) message loss. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 8:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >>> mechanism to map IP addresses to names for the purposes of logging > >>> error messages. As such, pretty much the only piece that needs to > >>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>> name mapping. A thread would be responsible for expiring entries > >>> from the cache (or refreshing the timeout) after validating the > >>> correctness of the mapping. I think the DNS lookups should be > >>> handled by a good resolver like pdns-recursor, djbdns,... The > >>> goal here is to allow names in the log entries and not just IP > >>> addresses and in a very high performance logging environment. > >> > >> the trouble is that doing _proper_ TTL expiration isn't as > simple as it > >> sounds. > >> > >> and if you are willing to back away from 'proper' > expiration to something > >> that will work in practice, why not go much further (as I > have detailed in > >> the other messages) > >> > >> David Lang > > > > I agree. I only mention TTL values as a reasonable upperbound of the > > refresh check. The advantage is to remove the large pause in logging > > due to a DNS refresh after a HUP to rsyslog, if that were the only > > method to flush/refresh. Like you mention, the IPs/names of systems > > being logged change rarely so we should tune this for speed and > > not worry about expiration correctness. I do not agree so other > > statements that an external cache will perform as well as > an internal > > cache. Too many software products that I work with have > needed exactly > > that functionality to support very high levels of performance. > > actually, you could have the cache be configurable in three modes > > 1. no caching > > 2. blank the cache on HUP > > 3. never blank the cache (i.e. require a full restart to clear it) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Sep 30 21:53:00 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 12:53:00 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the good discussion. I am lacking somewhat behind, but will review > it in depth tomorrow morning. > > I just wanted to stress the point that an external cache does not really > help, much for the reason David mentioned: if you process messages at very > high data rates, the context switch overhead involved with any external > solution is extremely costly. Also, in the usual cases, I may do several > million queries within a few seconds for just a handful of hosts. With an > internal cache, the overhead in doing so is very minimal. With an external > solution, the overhead in calling the external cache causes a lot of > performance degredation, what in the case of UDP also implies (heavy!) > message loss. the message loss problem with UDP will not be solved completely by an internal cache. when the source is not in the cache and you have to go out to find it the lookup can take several seconds. moving the lookup out of the input module and into the output module would address this, anything else would leave you with losses as the cache gets populated. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, September 30, 2009 8:54 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >> >>>>> mechanism to map IP addresses to names for the purposes of logging >>>>> error messages. As such, pretty much the only piece that needs to >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> >>>>> name mapping. A thread would be responsible for expiring entries >>>>> from the cache (or refreshing the timeout) after validating the >>>>> correctness of the mapping. I think the DNS lookups should be >>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>> goal here is to allow names in the log entries and not just IP >>>>> addresses and in a very high performance logging environment. >>>> >>>> the trouble is that doing _proper_ TTL expiration isn't as >> simple as it >>>> sounds. >>>> >>>> and if you are willing to back away from 'proper' >> expiration to something >>>> that will work in practice, why not go much further (as I >> have detailed in >>>> the other messages) >>>> >>>> David Lang >>> >>> I agree. I only mention TTL values as a reasonable upperbound of the >>> refresh check. The advantage is to remove the large pause in logging >>> due to a DNS refresh after a HUP to rsyslog, if that were the only >>> method to flush/refresh. Like you mention, the IPs/names of systems >>> being logged change rarely so we should tune this for speed and >>> not worry about expiration correctness. I do not agree so other >>> statements that an external cache will perform as well as >> an internal >>> cache. Too many software products that I work with have >> needed exactly >>> that functionality to support very high levels of performance. >> >> actually, you could have the cache be configurable in three modes >> >> 1. no caching >> >> 2. blank the cache on HUP >> >> 3. never blank the cache (i.e. require a full restart to clear it) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 30 21:56:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:56:33 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 9:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > Thanks for the good discussion. I am lacking somewhat > behind, but will review > > it in depth tomorrow morning. > > > > I just wanted to stress the point that an external cache > does not really > > help, much for the reason David mentioned: if you process > messages at very > > high data rates, the context switch overhead involved with > any external > > solution is extremely costly. Also, in the usual cases, I > may do several > > million queries within a few seconds for just a handful of > hosts. With an > > internal cache, the overhead in doing so is very minimal. > With an external > > solution, the overhead in calling the external cache causes a lot of > > performance degredation, what in the case of UDP also > implies (heavy!) > > message loss. > > the message loss problem with UDP will not be solved completely by an > internal cache. when the source is not in the cache and you > have to go out > to find it the lookup can take several seconds. > > moving the lookup out of the input module and into the output > module would > address this, anything else would leave you with losses as > the cache gets > populated. That's right and that's one reason why I intend to move this (optionally) over to the "backend" processing. However, even that does not completely solve the message loss problem, as we, in extreme cases, may loose messages when the queue is full - and for a myriad of other reasons, like routers discarding frames and such. Of course, you know that, but I'd like to mention if for those folks that at some time find our conversation via Google ;) Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Wednesday, September 30, 2009 8:54 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] DNS cache and expiration > >> > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > >> > >>>>> mechanism to map IP addresses to names for the purposes > of logging > >>>>> error messages. As such, pretty much the only piece > that needs to > >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>>>> name mapping. A thread would be responsible for expiring entries > >>>>> from the cache (or refreshing the timeout) after validating the > >>>>> correctness of the mapping. I think the DNS lookups should be > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > >>>>> goal here is to allow names in the log entries and not just IP > >>>>> addresses and in a very high performance logging environment. > >>>> > >>>> the trouble is that doing _proper_ TTL expiration isn't as > >> simple as it > >>>> sounds. > >>>> > >>>> and if you are willing to back away from 'proper' > >> expiration to something > >>>> that will work in practice, why not go much further (as I > >> have detailed in > >>>> the other messages) > >>>> > >>>> David Lang > >>> > >>> I agree. I only mention TTL values as a reasonable > upperbound of the > >>> refresh check. The advantage is to remove the large pause > in logging > >>> due to a DNS refresh after a HUP to rsyslog, if that were the only > >>> method to flush/refresh. Like you mention, the IPs/names > of systems > >>> being logged change rarely so we should tune this for speed and > >>> not worry about expiration correctness. I do not agree so other > >>> statements that an external cache will perform as well as > >> an internal > >>> cache. Too many software products that I work with have > >> needed exactly > >>> that functionality to support very high levels of performance. > >> > >> actually, you could have the cache be configurable in three modes > >> > >> 1. no caching > >> > >> 2. blank the cache on HUP > >> > >> 3. never blank the cache (i.e. require a full restart to clear it) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 1 10:51:35 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 10:51:35 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> > As a side-note: I think that my UDP message loss may partly be related > to DNS > resolution. I will try this in a lab tomorrow. But I still think a lot > of > packets never leave the source system. This may be related to the > virtual > environment I am currently using for the lab. I hope to be able to > generate > the traffic by a program, because that offers me the flexibility (now > and in > the future) to test complex messages scenarios (what, granted, does not > help > if it does not expose the problem...). Very interesting - I just did a couple of tests with UDP and various DNS resolution settings. The message loss I see is definitely related to DNS resolution. This is especially interesting as in my lab setup there should be no need to do more than the initial query. This points into some area that either is buggy or needs to be optimized. When I turn off DNS resolution, I have far fewer lost message. Still, there is between 1% and 10% loss for reasonable high traffic, but that is OK from my expectations given the lab environment I use. With DNS resolution, I have > 90% loss, and this difference is clearly not acceptable. I will look into this issue, but will try to find the segfault first (better not change the environment so that the bug moves to some other region). In the light of this, I'll probably rerun some of my tests today without reverse DNS resolution - the higher rate will hopefully trigger the bug in my lab. Rainer From rgerhards at hq.adiscon.com Tue Sep 1 12:26:31 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 12:26:31 +0200 Subject: [rsyslog] abort in 4.2.1 / UDP message loss References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><9B6E2A8877C38245BFB15CC491A11DA706FDC9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDD1@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 01, 2009 10:52 AM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 / UDP message loss > > > As a side-note: I think that my UDP message loss may partly be > related > > to DNS > > resolution. I will try this in a lab tomorrow. But I still think a > lot > > of > > packets never leave the source system. This may be related to the > > virtual > > environment I am currently using for the lab. I hope to be able to > > generate > > the traffic by a program, because that offers me the flexibility (now > > and in > > the future) to test complex messages scenarios (what, granted, does > not > > help > > if it does not expose the problem...). > > Very interesting - I just did a couple of tests with UDP and various > DNS > resolution settings. The message loss I see is definitely related to > DNS > resolution. This is especially interesting as in my lab setup there > should be > no need to do more than the initial query. This points into some area > that > either is buggy or needs to be optimized. ... my simplistic requery-avoidance logic does not take the source port into account. So a requery is also done if the host is the same as before, but the port changes. Thus the difference. Needs to be optimized ;) While this does not point to an obvious bug, I'll still try to get a segfault without DNS resolution. Rainer From mikel at irontec.com Tue Sep 1 14:20:35 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 01 Sep 2009 14:20:35 +0200 Subject: [rsyslog] milliseconds timestamp Message-ID: <4A9D1193.4090806@irontec.com> hi Some news about this? http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html Maybe with a bounty? thanks From rgerhards at hq.adiscon.com Tue Sep 1 14:25:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 14:25:12 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDD4@GRFEXC.intern.adiscon.com> Hi, Andre has just gone on vacation, expect a real answer in two weeks ;) But I don't think he had time to look at this (too many paid projects in the way...). So a bounty may be useful ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 01, 2009 2:21 PM > To: rsyslog-users > Subject: [rsyslog] milliseconds timestamp > > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From theinric at redhat.com Tue Sep 1 18:59:25 2009 From: theinric at redhat.com (Tomas Heinrich) Date: Tue, 01 Sep 2009 18:59:25 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FDCA@GRFEXC.intern.adiscon.com> Message-ID: <4A9D52ED.4090103@redhat.com> On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas From rgerhards at hq.adiscon.com Tue Sep 1 19:55:57 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:55:57 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001501ca2b2d$7e16309d$100013ac@intern.adiscon.com> Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 1 19:58:56 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 1 Sep 2009 19:58:56 +0200 Subject: [rsyslog] Three bugs to stable v2 reported to Red Hat Message-ID: <001601ca2b2d$e8e76725$100013ac@intern.adiscon.com> I was too quick. I should have said "backported with ease **to v4**". v2 obviously is so outdated, that this will require a totally different effort. ----- Urspr?ngliche Nachricht ----- Von: "Rainer Gerhards" An: "rsyslog-users" Gesendet: 01.09.09 19:56 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat Interesting - as i said, everything works fine under fedora with 2000 connections... Anyhow: going away from select is not trivial, but on my schedule for v5. This functionality can probably be backported with relative ease once it is available. Depending on the bug hunt effort, i'd say within the autumn. rainer ----- Urspr?ngliche Nachricht ----- Von: "Tomas Heinrich" An: "rsyslog-users" Gesendet: 01.09.09 19:01 Betreff: Re: [rsyslog] Three bugs to stable v2 reported to Red Hat On 08/31/2009 10:00 PM, Rainer Gerhards wrote: >> The limitation of 1000 open file descriptors however (limitation of >> select()) is still there in newer rsyslog releases and >> therefor we are >> probably forced to work around it. Although I find it >> personally strange >> that this limitation is not a more widespread problem. Is >> everybody using >> a database backend ? Or are people segregating syslog messages by >> location/importance ? > > I am not sure tha it is a select() limit. I routinely run tests with 2000 tcp > connections under Fedora and it works well. An issue, of course, is the > per-process file handle limit, which (on many systems) is 1,024. In current > releases, you can simple increase that limit via the $MaxOpenFiles directive: Part of the problem, based on the log excerpt from one of the bug reports, is that the file descriptor limit for the process is too low; Aug 1 19:10:30 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files Aug 1 19:10:34 lg2log01 rsyslogd:last message repeated 49 times Aug 1 19:10:33 lg2log01 rsyslogd:tcp accept, ignoring error and connection request: Too many open files This isn't caused by select(). It should be possible to change the limit with 'ulimit -n '. The issue with select() remains, though. It has a hardcoded limit on the number of file descriptors of FD_SETSIZE. I've run a test with rsyslog 2.0.6 and I haven't been able to receive messages from tcp clients with fds > 1024. Tomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Wed Sep 2 01:06:17 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 16:06:17 -0700 (PDT) Subject: [rsyslog] -cX command line option Message-ID: with version 3+ do we really need to change the X in this option? if you run v5 with -c4 is it really going to do something different with the config file than if you use -c5? yes, there are new config options in the newer versions, and once in a while some depriciated config options stop working, but does changing from -c3 to -c4 to -c5 actually fix any of these? in my testing I keep switching between the v4 series and the v5 series and having to change the startup to give the correct -c flag has tripped me up more than once. it would also be helpful if rsyslog would spit out errors about unknown config files (either to the console or as syslog messages) without needing to be in debug mode. it may that it tries to do this, but I don't see them (either with the debian startup scripts or when starting it directly on the command line) David Lang From david at lang.hm Wed Sep 2 03:11:56 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 1 Sep 2009 18:11:56 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251715849.4897.13.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: I got a core file with 4.2.0 I did git checkout -f v4.2.0 configure --enable-imfile and installed the result. I will go through the core file either later tonight or in the morning. in this case it did take a while for it to die. (over an hour) David Lang On Mon, 31 Aug 2009, Rainer Gerhards wrote: > Date: Mon, 31 Aug 2009 12:50:49 +0200 > From: Rainer Gerhards > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > On Fri, 2009-08-28 at 14:55 -0700, david at lang.hm wrote: >> On Fri, 28 Aug 2009, Rainer Gerhards wrote: >>> Also, it would be good if you could --enable-rtinst --enable-debug and try >>> out that version on your machine. I am a bit concerned about the speed of the >>> resulting executable, it may be too slow. You do not need to run it in debug >>> mode itself. These option (especially--enable-debug) will activate in-depth >>> runtime checks (assert, will abort when something wrong happens) and my hope >>> is that they will catch the bug closer to the root cause. If so, I would need >>> the gdb abort info (actually enabling debug output would be an option some >>> time later). >>> >>> Please let me know what would be OK with you. >> >> I will give this a try. >> >> I was going to suggest that since we have the message getting corrupted it >> may make sense to make a temporary branch that has multiple message >> buffers and at various times through the message processing it makes a >> copy of the emssage to the buffer. when the system crashes I will be able >> to look at the core and see where the message is getting corrupted. > > David, I fear it is even more complicated than that. It looks like not > only the message got corrupted but the message object itself. There are > already two copies of some of the message elements, and they also look > inconsistent - except, if we really had a null message, that is one with > no content at all (and generating a message object from a null message, > I think, would be a bug in itself - but I am sure there are no such > messages in your actual traffic). If you think there could be a real > null message, I'd follow that path (will probably do so in any case...). > > I think that what really happens is that some part of the code runs > wild, thus invalidating some random part of the main memory. At some > times, it hits queue structures (or the message object that is held by > them) and if so, we will see the abort you experience. With that > scenario, duplicating the message buffer does not really help, because > looking at the corrupted message object would not provide any additional > information. > > However, if that's easy enough to reproduce, it would probably be good > if you could send me the core analysis (the backtrace and the print > statements) from a few (five maybe?) independent aborts. Maybe they show > a pattern. It would probably best to send them via private mail, as I am > not sure if they disclose more than they should. > >> >> I will see about doing a tcpdump at the time that I do this and send it to >> you (I'll need to check with management, but since we have a contract in >> place for other reasons I think we can do this) >> > > That would probably be a good thing. I've made some progress with my > testing tool, and I have created a basic version right now. Probably not > good enough to mimic your traffic pattern, but closer. I am doing a test > run for quite some time now, unfortunately so far without abort. > > Note that I run into the trouble with UDP - even though I've put some > one-ms sleeps into the code, I lose a lot of messages, as it looks even > before they hit the wire. It's always real trobulesome to test with > UDP... > > Rainer >> I can't do this late on a friday, but I should be able to do this monday >> afternoon. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 12:14:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:14:21 +0200 Subject: [rsyslog] abort in 4.2.1 In-Reply-To: References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> Message-ID: <1251886461.5821.8.camel@rgf11> David, thanks for this test. The outcome is obviously other than I hoped/expected, but that makes it very useful. Obviously I have been looking for the wrong root cause. Any abort information you can provide would be useful. Even more useful would be if you could try out some earlier releases. Not sure if that is possible from a feature point of view. If it is, I would appreciate if you could give v3-stable a try and, if and only if that fails, too, checkout v3.18.6 and try that one. The 3.18.6 is the version that Debian ships and so I know it has a lot of testers and received a lot of bug-finding attention (I thankfully receive lots of very qualified bug reports from the Debian community :)). Please let me know what is possible. In any case, the 4.2.0 failure even more points to environment-specific problems. Rainer On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: > I got a core file with 4.2.0 > > I did git checkout -f v4.2.0 configure --enable-imfile and installed the > result. > > I will go through the core file either later tonight or in the morning. > > in this case it did take a while for it to die. (over an hour) > > David Lang From rgerhards at hq.adiscon.com Wed Sep 2 12:23:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 02 Sep 2009 12:23:02 +0200 Subject: [rsyslog] -cX command line option In-Reply-To: References: Message-ID: <1251886982.5821.17.camel@rgf11> On Tue, 2009-09-01 at 16:06 -0700, david at lang.hm wrote: > with version 3+ do we really need to change the X in this option? > if you run v5 with -c4 is it really going to do something different with > the config file than if you use -c5? > > yes, there are new config options in the newer versions, and once in a > while some depriciated config options stop working, but does changing from > -c3 to -c4 to -c5 actually fix any of these? The -cX is more a vehicle to change things like *defaults*, that is something that breaks existing configurations. So far, there is no difference between v4 and v5 in this regard. However, I would not like to give up this vehicle. That would actually force me to never change any defaults. > > in my testing I keep switching between the v4 series and the v5 series and > having to change the startup to give the correct -c flag has tripped me up > more than once. > > it would also be helpful if rsyslog would spit out errors about unknown > config files (either to the console or as syslog messages) without needing > to be in debug mode. The current versions already does this. I think they go to stderr (maybe stdout). > > it may that it tries to do this, but I don't see them (either with the > debian startup scripts or when starting it directly on the command line) > I could offer the follwing solution for what you describe: I could permit (in newer v3/v4 builds) to specify a higher version (-c5) and only sending an alert. Doing so, of course, means "I know what I do and I can live with any consequences from it" what should be fine for your use case. Please let me know if that would be helpful for you. Rainer From tbergfeld at hq.adiscon.com Wed Sep 2 14:41:49 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Wed, 2 Sep 2009 14:41:49 +0200 Subject: [rsyslog] rsyslog 4.4.1 (v4-stable) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDDF@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 4.4.1., a member of the v4-development branch. This is a bug-fixing release, providing some important fixes for issues that have only been detected after the beta phase. Some of them are serious (like a segfault when UDP messageforwarding is activated), so users of 4.4.0 are urged to upgrade to this release. Have a look at the change log to see all new features included in this release. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-173.phtml Changelog: http://www.rsyslog.com/Article398.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From mikel at irontec.com Wed Sep 2 14:52:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 14:52:02 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9D1193.4090806@irontec.com> References: <4A9D1193.4090806@irontec.com> Message-ID: <4A9E6A72.8080202@irontec.com> Ok, I will comunicate you if we decide. Is the development of phplogcon frezzed? the last version is of January 27 ... Thanks Mikel Jimenez wrote: > hi > > Some news about this? > > http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > Maybe with a bounty? > > thanks > From rgerhards at hq.adiscon.com Wed Sep 2 14:56:34 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 14:56:34 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 2:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of January > 27 ... Definitely not, it is active. But it looks like the web site did not receive proper attention. I'll check what's going on... See the git log for what's going on: http://git.adiscon.com/?p=phplogcon.git;a=summary The pace of changes is somewhat lower than in the initial phase, because there have been more pressing projects. But I have talked with Andre over big reporting features, which he will (hopefully) be able to tackle once he is back from his vacation. Rainer From mikel at irontec.com Wed Sep 2 15:00:57 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Wed, 02 Sep 2009 15:00:57 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> Message-ID: <4A9E6C89.4060309@irontec.com> Ahhh!! Ok Ok I see that it is active... so in near future the web page would be syncronized with the real state of the development? I usually use the web for news about phplogcon. (www.phplogcon.org) Thanks Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Wednesday, September 02, 2009 2:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Ok, I will comunicate you if we decide. >> >> Is the development of phplogcon frezzed? the last version is of January >> 27 ... >> > > Definitely not, it is active. But it looks like the web site did not receive > proper attention. I'll check what's going on... > > See the git log for what's going on: > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > The pace of changes is somewhat lower than in the initial phase, because > there have been more pressing projects. But I have talked with Andre over big > reporting features, which he will (hopefully) be able to tackle once he is > back from his vacation. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 2 15:03:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 15:03:46 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FDE1@GRFEXC.intern.adiscon.com> <4A9E6C89.4060309@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE3@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Wednesday, September 02, 2009 3:01 PM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Ahhh!! Ok Ok > > I see that it is active... so in near future the web page would be > syncronized with the real state of the development? > > I usually use the web for news about phplogcon. (www.phplogcon.org) > Thanks I've already pinged the web folks. I agree, I also go to the sites. I think there also have been no release annoucements (actually my primary source of new release info). Interestingly, I just saw that freshmeat has announcements: http://freshmeat.net/projects/phplogcon/ ... strange ;) Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Wednesday, September 02, 2009 2:52 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] milliseconds timestamp > >> > >> Ok, I will comunicate you if we decide. > >> > >> Is the development of phplogcon frezzed? the last version is of > January > >> 27 ... > >> > > > > Definitely not, it is active. But it looks like the web site did not > receive > > proper attention. I'll check what's going on... > > > > See the git log for what's going on: > > > > http://git.adiscon.com/?p=phplogcon.git;a=summary > > > > The pace of changes is somewhat lower than in the initial phase, > because > > there have been more pressing projects. But I have talked with Andre > over big > > reporting features, which he will (hopefully) be able to tackle once > he is > > back from his vacation. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Wed Sep 2 16:45:39 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 11:45:39 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" Message-ID: <4A9E8513.8020107@swiss-wireless.com.ar> Hello I'm pretty new to rsyslog. I know that you can specifiy 2 server for remote logging which will be handled "independent" i.e. rsyslog will log to the 2 server in parallel. What I want is a primary rsyslog server and a secondary rsyslog server and only if the primary is not avaiable the secondary should be used. Is this possible with rsyslog? Thanks for any hints Beat From rgerhards at hq.adiscon.com Wed Sep 2 19:06:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 2 Sep 2009 19:06:14 +0200 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" References: <4A9E8513.8020107@swiss-wireless.com.ar> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Beat Meier > Sent: Wednesday, September 02, 2009 4:46 PM > To: rsyslog-users > Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with > "priority" > > Hello > > I'm pretty new to rsyslog. > I know that you can specifiy 2 server for remote logging which will be > handled "independent" > i.e. rsyslog will log to the 2 server in parallel. > What I want is a primary rsyslog server and a secondary rsyslog server > and only if the primary > is not avaiable the secondary should be used. Is this possible with > rsyslog? > > Thanks for any hints > > Beat > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 2 19:55:32 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 2 Sep 2009 10:55:32 -0700 (PDT) Subject: [rsyslog] abort in 4.2.1 In-Reply-To: <1251886461.5821.8.camel@rgf11> References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com> <1251715849.4897.13.camel@rgf11> <1251886461.5821.8.camel@rgf11> Message-ID: On Wed, 2 Sep 2009, Rainer Gerhards wrote: > David, > > thanks for this test. The outcome is obviously other than I > hoped/expected, but that makes it very useful. Obviously I have been > looking for the wrong root cause. > > Any abort information you can provide would be useful. Even more useful > would be if you could try out some earlier releases. Not sure if that is > possible from a feature point of view. > > If it is, I would appreciate if you could give v3-stable a try and, if > and only if that fails, too, checkout v3.18.6 and try that one. The > 3.18.6 is the version that Debian ships and so I know it has a lot of > testers and received a lot of bug-finding attention (I thankfully > receive lots of very qualified bug reports from the Debian > community :)). > > Please let me know what is possible. In any case, the 4.2.0 failure even > more points to environment-specific problems. I haven't gone back to the 3.x series, but I did several more runs with 4.2.0 doing the folloiwng killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r -h ; mv /core /core-4.2.0-10 I have several complete steps, as well as several partial sets of data. I will gzip them and attempt to send them to you directly. David Lang > Rainer > > On Tue, 2009-09-01 at 18:11 -0700, david at lang.hm wrote: >> I got a core file with 4.2.0 >> >> I did git checkout -f v4.2.0 configure --enable-imfile and installed the >> result. >> >> I will go through the core file either later tonight or in the morning. >> >> in this case it did take a while for it to die. (over an hour) >> >> David Lang > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbe_ml at swiss-wireless.com.ar Wed Sep 2 22:18:09 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 17:18:09 -0300 Subject: [rsyslog] Remote loggin to 2 rsyslog servers but with "priority" In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> References: <4A9E8513.8020107@swiss-wireless.com.ar> <9B6E2A8877C38245BFB15CC491A11DA706FDE9@GRFEXC.intern.adiscon.com> Message-ID: <4A9ED301.2020500@swiss-wireless.com.ar> Thanks Rainer That's exactly what I looked for. One more question I use templates of the form: $template DynFileAuth,"/var/log/%HOSTNAME%/auth.log" $template DynFileSyslog,"/var/log/%HOSTNAME%/syslog" $template DynFileCron,"/var/log/%HOSTNAME%/cron.log" How can I use variables to replace the path /var/log So I can use something like: path="/var/log" $template DynFileAuth,"$path/%HOSTNAME%/auth.log" $template DynFileSyslog,"$path/%HOSTNAME%/syslog" $template DynFileCron,"$path/%HOSTNAME%/cron.log" Is the template way the only one? I think that template is expanded at runtime, isn't it? Is there a variable method that is expanded when daemon starts, for efficiency? I have nothing found in the wiki with search, no in the docu index, nor the man page of rsyslog.conf. Greetings and thanks Beat Rainer Gerhards wrote: >please see: http://wiki.rsyslog.com/index.php/FailoverSyslogServer > > > From mbe_ml at swiss-wireless.com.ar Thu Sep 3 00:19:19 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Wed, 02 Sep 2009 19:19:19 -0300 Subject: [rsyslog] Which librelp does work with rsyslog-3.18.6 ? Message-ID: <4A9EEF67.20502@swiss-wireless.com.ar> Hello I'm using rsyslog 3.18.6 on debian 4.0 (backport of rsyslog) There is no backport of a newer version with librelp support :-( so I downloaded librelp-0.1.3 and compiled it on debian 4.0. This will install librelp.so I have moved this to /usr/lib/rsyslog but rsyslog is complaining that he cannot find imrelp.so. I have configured the module imrelp as noted some ware else. I have now seen that debian version of relp (for debian 5.0) has imrelp.so and omrelp.so defined. Has the name of the shared object changed from an old release to 0.1.3? Nor renaming librelp.so nor changing module name did work. (renaming library results in an undedfined symbol: modInit) Which release of lbrelp can I use with rsyslog V3.18.6 or can I use librelp only with rsyslog-4.4? Greetings and thanks Beat From joe at joetify.com Thu Sep 3 05:11:28 2009 From: joe at joetify.com (Joe Williams) Date: Wed, 2 Sep 2009 20:11:28 -0700 Subject: [rsyslog] case sensitivity in templates Message-ID: <20090902201128.507f2449@der-dieb> Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 07:47:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 07:47:14 +0200 Subject: [rsyslog] case sensitivity in templates Message-ID: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> You need to check the property replacer documentation. There are options for case conversion. I don't know the exact syntax out of my head, but it is along the lines of %field:::ucase%. Hth rainer ----- Urspr?ngliche Nachricht ----- Von: "Joe Williams" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 05:19 Betreff: [rsyslog] case sensitivity in templates Hello, I am new to the list sorry if this has been covered already. I am logging using a per-host template like: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" The services that log directly to the rsyslog server (haproxy, etc) are using all lower case hostname directories where as the logs that use the rsyslog client daemon to log to the server are using the case specified in the hostname which in my case have capital letters in them. Is there any way to specify which to use? I would like to have a single directory for each host regardless of the case used in the hostname. It doesn't matter to me which case is used as long as it's the same for all logs. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From rgerhards at hq.adiscon.com Thu Sep 3 12:23:45 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 12:23:45 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Hi David, > I haven't gone back to the 3.x series, but I did several more runs with > 4.2.0 doing the folloiwng > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd -r > -h ; mv /core /core-4.2.0-10 > > I have several complete steps, as well as several partial sets of data. > I > will gzip them and attempt to send them to you directly. Thanks for the data set, I am right now working on it. Unfortunately, as I feared, the core files do not really help. There is a big mismatch between your system environment and mine, and so gdb is not able to extract any useful information. All I see is that there are six threads in the system, and the rest is almost only question marks. So it would be great if you could issue the gdb commands in your environment and let me know the outcome. Thanks, Rainer From joe at joetify.com Thu Sep 3 17:51:23 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 08:51:23 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> Message-ID: <20090903085123.0bcea4b0@der-dieb> Thanks, that worked perfectly. -Joe On Thu, 3 Sep 2009 07:47:14 +0200 "Rainer Gerhards" wrote: > You need to check the property replacer documentation. There are > options for case conversion. I don't know the exact syntax out of my > head, but it is along the lines of %field:::ucase%. > > Hth > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "Joe Williams" > An: "rsyslog at lists.adiscon.com" > Gesendet: 03.09.09 05:19 > Betreff: [rsyslog] case sensitivity in templates > > > Hello, I am new to the list sorry if this has been covered already. I > am logging using a per-host template like: > > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > The services that log directly to the rsyslog server (haproxy, etc) > are using all lower case hostname directories where as the logs that > use the rsyslog client daemon to log to the server are using the case > specified in the hostname which in my case have capital letters in > them. > > Is there any way to specify which to use? I would like to have a > single directory for each host regardless of the case used in the > hostname. It doesn't matter to me which case is used as long as it's > the same for all logs. > > Thanks. > > -Joe > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From joe at joetify.com Thu Sep 3 18:02:58 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 09:02:58 -0700 Subject: [rsyslog] case sensitivity in templates In-Reply-To: <20090903085123.0bcea4b0@der-dieb> References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com> <20090903085123.0bcea4b0@der-dieb> Message-ID: <20090903090258.384a9214@der-dieb> BTW, if anyone else has this problem or something similar the doc is at: http://www.rsyslog.com/module-Static_Docs-view-f-property_replacer.html.phtml The fix was changing the template to be: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase%/debug" Thanks again. -Joe On Thu, 3 Sep 2009 08:51:23 -0700 Joe Williams wrote: > Thanks, that worked perfectly. > > -Joe > > > On Thu, 3 Sep 2009 07:47:14 +0200 > "Rainer Gerhards" wrote: > > > You need to check the property replacer documentation. There are > > options for case conversion. I don't know the exact syntax out of my > > head, but it is along the lines of %field:::ucase%. > > > > Hth > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "Joe Williams" > > An: "rsyslog at lists.adiscon.com" > > Gesendet: 03.09.09 05:19 > > Betreff: [rsyslog] case sensitivity in templates > > > > > > Hello, I am new to the list sorry if this has been covered already. > > I am logging using a per-host template like: > > > > $template > > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/debug" > > > > The services that log directly to the rsyslog server (haproxy, etc) > > are using all lower case hostname directories where as the logs that > > use the rsyslog client daemon to log to the server are using the > > case specified in the hostname which in my case have capital > > letters in them. > > > > Is there any way to specify which to use? I would like to have a > > single directory for each host regardless of the case used in the > > hostname. It doesn't matter to me which case is used as long as it's > > the same for all logs. > > > > Thanks. > > > > -Joe > > > > > > -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From rgerhards at hq.adiscon.com Thu Sep 3 18:05:42 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 18:05:42 +0200 Subject: [rsyslog] case sensitivity in templates References: <001701ca2c59$fedc919f$100013ac@intern.adiscon.com><20090903085123.0bcea4b0@der-dieb> <20090903090258.384a9214@der-dieb> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDF5@GRFEXC.intern.adiscon.com> > $template > PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercas > e%/debug" Now that you say it: it would probably make sense to use one of the "secpath" (or so) options to make this file writer more secure - see the doc you quoted for details. It will then look something along these lines: $template PerHostDebug,"/srv/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase,secp ath-...%/debug" > > Thanks again. my pleasure :) Rainer From corsmith at gmail.com Thu Sep 3 19:39:15 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 13:39:15 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <8061fbee0909031039y637bdf52j72b1322cf2538f55@mail.gmail.com> I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:01:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:01:12 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Thats an very interesting effort. Do you needed to patch the source? Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog at lists.adiscon.com" Gesendet: 03.09.09 19:45 Betreff: [rsyslog] rsyslog 4.4.1 and solaris I'm new to the list so be kind. Here are my notes for building, installing and running rsyslog 4.4.1 on Solaris 10/Sparc64 on a V210. Using gcc4.4 it is possible to get atomic operations working. root at csmith-rsyslog# uname -a SunOS csmith-rsyslog 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V210 root at csmith-rsyslog# gcc --version gcc (GCC) 4.4.0 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. root at csmith-rsyslog# rsyslogd -c3 -v rsyslogd 4.4.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): No See http://www.rsyslog.com for more information. # build gcc44 from pkgsrc-wip # - installation of pkgsrc and pkgsrc-wip is left as an exercise for the reader cd /usr/pkgsrc/wip/gcc44 && make install clean # add these to /usr/pkg/etc/mk.conf to use gcc44 and make 64-bit binaries PKGSRC_COMPILER= gcc USE_NATIVE_GCC= yes CC= /usr/pkg/gcc44/bin/gcc CPP= /usr/pkg/gcc44/bin/cpp CXX= /usr/pkg/gcc44/bin/g++ CFLAGS+= -m64 -O -pipe PKG_OPTIONS.rsyslog= relp # extract librelp and rsyslog pkgsrc tarball (attached) cd /usr/pkgsrc/wip && gunzip -c wip-rsyslog.tgz | tar xvf - # compile rsyslog cd /usr/pkgsrc/wip/rsyslog && make install clean # fix runtime linking problem with solaris + gcc44 # - generate a test config crle -64 -c /test.conf -u -l /usr/pkg/gcc44/lib/sparcv9/ # - test that rsyslog runs with this config LD_CONFIG=/test.conf rsyslogd -c5 -f /usr/pkg/etc/rsyslog.conf -d -n # - if it works then copy /test.conf to the system location mv test.conf /var/ld/sparcv9/ld.conf # END OF NOTES If other people provide positive feedback I will look at getting librelp and the rsyslog updates into pkgsrc-wip. Initial testing looks good forwarding messages via TCP from a FreeBSD rsyslog to the Solaris rsyslog server. I will let you know if I run into problems during testing. -Corey Smith From corsmith at gmail.com Thu Sep 3 20:22:09 2009 From: corsmith at gmail.com (Corey Smith) Date: Thu, 3 Sep 2009 14:22:09 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> References: <001801ca2cc0$8438307f$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909031122x4e6469fo426368e6b7363a84@mail.gmail.com> On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith From rgerhards at hq.adiscon.com Thu Sep 3 20:58:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 3 Sep 2009 20:58:27 +0200 Subject: [rsyslog] rsyslog 4.4.1 and solaris Message-ID: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) Rainer (from the phone, thus brief) ----- Urspr?ngliche Nachricht ----- Von: "Corey Smith" An: "rsyslog-users" Gesendet: 03.09.09 20:22 Betreff: Re: [rsyslog] rsyslog 4.4.1 and solaris On Thu, Sep 3, 2009 at 2:01 PM, Rainer Gerhards wrote: > Thats an very interesting effort. Do you needed to patch the source? No patching necessary although there are several warning messages during the compile. I could send the build output to the list if it would be beneficial... -Corey Smith _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From srinivasan.sreenivasan at gmail.com Fri Sep 4 01:19:05 2009 From: srinivasan.sreenivasan at gmail.com (Srinivasan Sreenivasan) Date: Thu, 3 Sep 2009 18:19:05 -0500 Subject: [rsyslog] rsyslog on Solaris Message-ID: Hi, We are trying to run rsyslog version 4.4.1 on Solaris 2.8. We cannot get it do any logging. Rainer has a blog entry (its a bit dated) that says that rsyslogd does not do local logging on Solaris. Is that still valid? -Srini From joe at joetify.com Fri Sep 4 01:48:18 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 16:48:18 -0700 Subject: [rsyslog] logging wildcards Message-ID: <20090903164818.00c7bc9d@der-dieb> Hello again, I am trying to log everything (*.*) to /var/log/syslog but local*.*. I tried a couple different ways to do this but didn't find a solution. Is this possible? I have a couple services I want to log to their own file rather than syslog, messages, etc. Any help is appreciated. Thanks. -Joe -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:05:04 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903164818.00c7bc9d@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > Hello again, > > I am trying to log everything (*.*) to /var/log/syslog but local*.*. I > tried a couple different ways to do this but didn't find a solution. Is > this possible? I have a couple services I want to log to their own file > rather than syslog, messages, etc. if you just do *.* /var/log/syslog that will write everything to that file I'm not sure what you are trying to say when you say 'but local*.*' above. David Lang From joe at joetify.com Fri Sep 4 02:10:22 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 17:10:22 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> Message-ID: <20090903171022.5f37eade@der-dieb> I do not want everything to log to a single file, the local facilities I would like to log to there own file and not be caught by a wildcard. Thanks. -Joe On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > Hello again, > > > > I am trying to log everything (*.*) to /var/log/syslog but > > local*.*. I tried a couple different ways to do this but didn't > > find a solution. Is this possible? I have a couple services I want > > to log to their own file rather than syslog, messages, etc. > > if you just do > > *.* /var/log/syslog > > that will write everything to that file > > I'm not sure what you are trying to say when you say 'but local*.*' > above. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Fri Sep 4 02:24:06 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 3 Sep 2009 17:24:06 -0700 (PDT) Subject: [rsyslog] logging wildcards In-Reply-To: <20090903171022.5f37eade@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: On Thu, 3 Sep 2009, Joe Williams wrote: > I do not want everything to log to a single file, the local facilities > I would like to log to there own file and not be caught by a wildcard. ahh, ok, you cannot say local*.* you would have to list local0.*,local1.*,.. to cover them all there are 16 facility numbers, and by filtering out local0-local7 you are wanting to eliminate exactly half of them as such it's probably just as easy to list all the ones you want to record as it is to say *.* and subtract half of them. David Lang > Thanks. > -Joe > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > david at lang.hm wrote: > >> On Thu, 3 Sep 2009, Joe Williams wrote: >> >>> Hello again, >>> >>> I am trying to log everything (*.*) to /var/log/syslog but >>> local*.*. I tried a couple different ways to do this but didn't >>> find a solution. Is this possible? I have a couple services I want >>> to log to their own file rather than syslog, messages, etc. >> >> if you just do >> >> *.* /var/log/syslog >> >> that will write everything to that file >> >> I'm not sure what you are trying to say when you say 'but local*.*' >> above. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > > > From joe at joetify.com Fri Sep 4 05:44:31 2009 From: joe at joetify.com (Joe Williams) Date: Thu, 3 Sep 2009 20:44:31 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> Message-ID: <20090903204431.5a4a2008@der-dieb> Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) > > david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From DGillies at fairfaxdigital.com.au Fri Sep 4 05:51:38 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Fri, 4 Sep 2009 13:51:38 +1000 Subject: [rsyslog] logging wildcards In-Reply-To: <20090903204431.5a4a2008@der-dieb> References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: I think something like this should work: if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != 'local2' ) then file3 David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital Level 2, 1 Darling Island Road Pyrmont NSW 2009 -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com Subject: Re: [rsyslog] logging wildcards Sorry I think we are misunderstanding each other. What I am wanting to do is this: ### local0.* FILE1 local2.* FILE2 *.* (but not local0.* or local2.*) FILE3 ### Is that possible? Thanks again. -Joe On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) david at lang.hm wrote: > On Thu, 3 Sep 2009, Joe Williams wrote: > > > I do not want everything to log to a single file, the local > > facilities I would like to log to there own file and not be caught > > by a wildcard. > > ahh, ok, you cannot say local*.* you would have to list > local0.*,local1.*,.. to cover them all > > there are 16 facility numbers, and by filtering out local0-local7 you > are wanting to eliminate exactly half of them > > as such it's probably just as easy to list all the ones you want to > record as it is to say *.* and subtract half of them. > > David Lang > > > Thanks. > > -Joe > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > >> > >>> Hello again, > >>> > >>> I am trying to log everything (*.*) to /var/log/syslog but > >>> local*.*. I tried a couple different ways to do this but didn't > >>> find a solution. Is this possible? I have a couple services I want > >>> to log to their own file rather than syslog, messages, etc. > >> > >> if you just do > >> > >> *.* /var/log/syslog > >> > >> that will write everything to that file > >> > >> I'm not sure what you are trying to say when you say 'but local*.*' > >> above. > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From oribani at gmail.com Fri Sep 4 07:25:47 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:25:47 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <378058110909032225i10d9eb60g3039134f21eb2d64@mail.gmail.com> On 9/3/09, Ori Bani wrote: >>> I'm sorry if this isn't quite the right place to ask, since maybe no >>> one here created the RPM that's in the CentOS base repository. But I >>> am guessing people here have installed RPMs like this before and can >>> help anyway.... >>> >>> When I ask yum on CentOS 5 about rsyslog, I get this (note older >>> version - too bad): >>> >>> Available Packages >>> Name : rsyslog >>> Arch : i386 >>> Version: 2.0.6 >>> Release: 1.el5 >>> Size : 198 k >>> Repo : base >>> Summary: Enhanced system logging and kernel message trapping daemons >>> Description: >>> Rsyslog is an enhanced multi-threaded syslogd supporting, among >>> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >>> filtering on any message part, and fine grain output format control. >>> It is quite compatible to stock sysklogd and can be used as a drop- >>> in replacement. Its advanced features make it suitable for >>> enterprise-class, encryption protected syslog relay chains while at >>> the same time being very easy to setup for the novice user. >> >> I use Scientific Linux 5.x and because they are RHEL derivatives I see >> the >> same thing in the SL repo's. >> >> I have used the rsyslog from the repo's yet, all my rsyslog servers are >> based >> on EL4, but I'll try to help below. > > Thank you for your help. > >>> My questions are a little bit newbie... before I try installing >>> this, I want to know what it's going to do to my system: >>> >>> 1) Will it disable syslogd and/or klogd? Or will it add itself using >>> the "alternatives" paradigm so I can switch between them that way? >>> If neither, does it include startup scripts at all? If they are there >>> but not used by default, is there a recommended way to make the >>> switch and not really screw things up? >> >> You should try this on a test box. I haven't tried it but I think it >> should >> remove syslog RPM's from your installation and then install rsyslog. It >> should >> also make a /etc/syslog.conf.rpmsave file which you can reference for use >> in >> /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog I don't think that last line is needed; rsyslog is already added for you during the install process by yum. > And use yum to remove ksyslogd/syslog > >>> 2) Will it add itself to my cron jobs? Specifically, I don't mind >>> (for now) leaving the log rotation alone (don't let rsyslog manage my >>> rotations). If it adds itself to my cron jobs, does that mean it >>> will remove the logrotate cron job? >> >> Not sure sorry. You should grab the src.rpm file from CentOS, install it >> and >> take a look at the rsyslog.spec and it'll show you what it does on the >> post >> install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > >>> 2.5) If I keep using the old logrotate with rsyslog, will that create >>> any conflicts? >> >> I don't see how any conflicts will occur with logroate, since rsyslog >> basically logs to the same files that syslog logs to. It's meant to be a >> drop >> in replacement. >> >> Maybe specific questions about rsyslog with CentOS (or other derivatives) >> would actually be better in the CentOS or Scientific Linux mailing lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 > >>> Generally my aim is not to commit 100% to rsyslog yet, so I don't >>> want to get to a situation where it's a lot of work to get back to >>> the default syslog setup. > From oribani at gmail.com Fri Sep 4 07:21:08 2009 From: oribani at gmail.com (Ori Bani) Date: Thu, 3 Sep 2009 22:21:08 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <20090821015920.M76525@npgx.com.au> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> Message-ID: <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> one here created the RPM that's in the CentOS base repository. But I >> am guessing people here have installed RPMs like this before and can >> help anyway.... >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> version - too bad): >> >> Available Packages >> Name : rsyslog >> Arch : i386 >> Version: 2.0.6 >> Release: 1.el5 >> Size : 198 k >> Repo : base >> Summary: Enhanced system logging and kernel message trapping daemons >> Description: >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> filtering on any message part, and fine grain output format control. >> It is quite compatible to stock sysklogd and can be used as a drop- >> in replacement. Its advanced features make it suitable for >> enterprise-class, encryption protected syslog relay chains while at >> the same time being very easy to setup for the novice user. > > I use Scientific Linux 5.x and because they are RHEL derivatives I see the > same thing in the SL repo's. > > I have used the rsyslog from the repo's yet, all my rsyslog servers are > based > on EL4, but I'll try to help below. Thank you for your help. >> My questions are a little bit newbie... before I try installing >> this, I want to know what it's going to do to my system: >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself using >> the "alternatives" paradigm so I can switch between them that way? >> If neither, does it include startup scripts at all? If they are there >> but not used by default, is there a recommended way to make the >> switch and not really screw things up? > > You should try this on a test box. I haven't tried it but I think it should > remove syslog RPM's from your installation and then install rsyslog. It > should > also make a /etc/syslog.conf.rpmsave file which you can reference for use in > /etc/rsyslog.conf I wouldn't actually expect it to remove any other packages - I've never seen a yum installation remove something else - that seems like trouble. In fact, it turns out that it didn't do a thing to syslog/ksyslogd. It just installed itself in parallel (and it's up to you to turn it on). Everything is in place (startup scripts, config file that is a mirror of syslog.conf, etc.) and you just have to chkconfig syslog off chkconfig rsyslog on service syslog stop service rsyslog start I guess if you're going to be more permanent: chkconfig --del syslog chkconfig --add rsyslog And use yum to remove ksyslogd/syslog >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> (for now) leaving the log rotation alone (don't let rsyslog manage my >> rotations). If it adds itself to my cron jobs, does that mean it >> will remove the logrotate cron job? > > Not sure sorry. You should grab the src.rpm file from CentOS, install it and > take a look at the rsyslog.spec and it'll show you what it does on the post > install section. That's above my skill level. Instead I tried it out. It also adds itself to /etc/logrotate.d/syslog so you don't have to touch any of this. Here is the modified file: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true endscript } As you see, it left syslog there and added rsyslog. Because I have turned off syslog, this won't suddenly start it up, will it? >> 2.5) If I keep using the old logrotate with rsyslog, will that create >> any conflicts? > > I don't see how any conflicts will occur with logroate, since rsyslog > basically logs to the same files that syslog logs to. It's meant to be a > drop > in replacement. > > Maybe specific questions about rsyslog with CentOS (or other derivatives) > would actually be better in the CentOS or Scientific Linux mailing lists? I did, but it didn't help. That's disappointing. https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start=0#forumpost83694 >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> want to get to a situation where it's a lot of work to get back to >> the default syslog setup. From rgerhards at hq.adiscon.com Fri Sep 4 12:51:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 4 Sep 2009 12:51:21 +0200 Subject: [rsyslog] Need help with RPM(yum) version on CentOS References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com><20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> I have nothing technically to add to this discussion, but I would like remind you on the rsyslog wiki at http://wiki.rsyslog.com There already is one entry, but for an older version, not sure if that helps: http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story In any case, I would appreciate if you could share any knowledge you gain via the wiki. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Ori Bani > Sent: Friday, September 04, 2009 7:21 AM > To: rsyslog-users > Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS > > >> I'm sorry if this isn't quite the right place to ask, since maybe no > >> one here created the RPM that's in the CentOS base repository. But > I > >> am guessing people here have installed RPMs like this before and can > >> help anyway.... > >> > >> When I ask yum on CentOS 5 about rsyslog, I get this (note older > >> version - too bad): > >> > >> Available Packages > >> Name : rsyslog > >> Arch : i386 > >> Version: 2.0.6 > >> Release: 1.el5 > >> Size : 198 k > >> Repo : base > >> Summary: Enhanced system logging and kernel message trapping daemons > >> Description: > >> Rsyslog is an enhanced multi-threaded syslogd supporting, among > >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, > >> filtering on any message part, and fine grain output format control. > >> It is quite compatible to stock sysklogd and can be used as a drop- > >> in replacement. Its advanced features make it suitable for > >> enterprise-class, encryption protected syslog relay chains while at > >> the same time being very easy to setup for the novice user. > > > > I use Scientific Linux 5.x and because they are RHEL derivatives I > see the > > same thing in the SL repo's. > > > > I have used the rsyslog from the repo's yet, all my rsyslog servers > are > > based > > on EL4, but I'll try to help below. > > Thank you for your help. > > >> My questions are a little bit newbie... before I try installing > >> this, I want to know what it's going to do to my system: > >> > >> 1) Will it disable syslogd and/or klogd? Or will it add itself > using > >> the "alternatives" paradigm so I can switch between them that way? > >> If neither, does it include startup scripts at all? If they are > there > >> but not used by default, is there a recommended way to make the > >> switch and not really screw things up? > > > > You should try this on a test box. I haven't tried it but I think it > should > > remove syslog RPM's from your installation and then install rsyslog. > It > > should > > also make a /etc/syslog.conf.rpmsave file which you can reference for > use in > > /etc/rsyslog.conf > > I wouldn't actually expect it to remove any other packages - I've > never seen a yum installation remove something else - that seems like > trouble. In fact, it turns out that it didn't do a thing to > syslog/ksyslogd. It just installed itself in parallel (and it's up to > you to turn it on). Everything is in place (startup scripts, config > file that is a mirror of syslog.conf, etc.) and you just have to > > chkconfig syslog off > chkconfig rsyslog on > service syslog stop > service rsyslog start > > I guess if you're going to be more permanent: > > chkconfig --del syslog > chkconfig --add rsyslog > > And use yum to remove ksyslogd/syslog > > >> 2) Will it add itself to my cron jobs? Specifically, I don't mind > >> (for now) leaving the log rotation alone (don't let rsyslog manage > my > >> rotations). If it adds itself to my cron jobs, does that mean it > >> will remove the logrotate cron job? > > > > Not sure sorry. You should grab the src.rpm file from CentOS, install > it and > > take a look at the rsyslog.spec and it'll show you what it does on > the post > > install section. > > That's above my skill level. Instead I tried it out. It also adds > itself to /etc/logrotate.d/syslog so you don't have to touch any of > this. Here is the modified file: > > /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler > /var/log/boot.log /var/log/cron { > sharedscripts > postrotate > /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> > /dev/null || true > /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> > /dev/null || true > endscript > } > > As you see, it left syslog there and added rsyslog. Because I have > turned off syslog, this won't suddenly start it up, will it? > > >> 2.5) If I keep using the old logrotate with rsyslog, will that > create > >> any conflicts? > > > > I don't see how any conflicts will occur with logroate, since rsyslog > > basically logs to the same files that syslog logs to. It's meant to > be a > > drop > > in replacement. > > > > Maybe specific questions about rsyslog with CentOS (or other > derivatives) > > would actually be better in the CentOS or Scientific Linux mailing > lists? > > I did, but it didn't help. That's disappointing. > > https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start > =0#forumpost83694 > > >> Generally my aim is not to commit 100% to rsyslog yet, so I don't > >> want to get to a situation where it's a lot of work to get back to > >> the default syslog setup. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Fri Sep 4 21:25:30 2009 From: henry78 at gmx.at (Henry) Date: Fri, 04 Sep 2009 21:25:30 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message Message-ID: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Hi! This puzzels me: This is my tcprecieve config file for rsyslog v4 on ubuntu: -----8<----- $ModLoad imtcp $InputTCPServerRun 514 # some dynamic templates $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" # log remote local1 to dynamic diretory if $fromhost-ip != '127.0.0.1' and \ $syslogfacility-text == 'local1' \ then -?DYNlocal1 ----->8----- I created /var/log/remote with sufficient privileges. Unfortunately this doesn't work. rsyslog crates a folder named after the remote host (myhostname) and creates the file local1.log (again: sufficient permissions: syslog:syslog 640). But it doesn't write to that file, but logs the error: -----8<----- Could not open dynamic file '/var/log/remote/myhostname/local1.log' - discarding message ----->8----- As you might guess my question is: Why isn't rsyslog able to open a file it is able to create? Any help or hint is really appreciated. -- kind regards, Henry From joe at joetify.com Fri Sep 4 21:33:17 2009 From: joe at joetify.com (Joe Williams) Date: Fri, 4 Sep 2009 12:33:17 -0700 Subject: [rsyslog] logging wildcards In-Reply-To: References: <20090903164818.00c7bc9d@der-dieb> <20090903171022.5f37eade@der-dieb> <20090903204431.5a4a2008@der-dieb> Message-ID: <20090904123317.172e0ca4@der-dieb> Thanks David, that ended up working after changing the "or" to an "and". Also I ended up finding a good example of this sort of configuration at http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory -Joe On Fri, 4 Sep 2009 13:51:38 +1000 David Gillies wrote: > > I think something like this should work: > > if ( $syslogfacility-text != 'local0' ) or ( $syslogfacility-text != > 'local2' ) then file3 > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > > Fairfax Digital > Level 2, 1 Darling Island Road > Pyrmont NSW 2009 > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Joe Williams > Sent: Friday, 4 September 2009 1:45 PM To: rsyslog at lists.adiscon.com > Subject: Re: [rsyslog] logging wildcards > > > Sorry I think we are misunderstanding each other. What I am wanting > to do is this: > > ### > > local0.* FILE1 > local2.* FILE2 > > *.* (but not local0.* or local2.*) FILE3 > > ### > > Is that possible? > > Thanks again. > -Joe > > > > On Thu, 3 Sep 2009 17:24:06 -0700 (PDT) > david at lang.hm wrote: > > > On Thu, 3 Sep 2009, Joe Williams wrote: > > > > > I do not want everything to log to a single file, the local > > > facilities I would like to log to there own file and not be > > > caught by a wildcard. > > > > ahh, ok, you cannot say local*.* you would have to list > > local0.*,local1.*,.. to cover them all > > > > there are 16 facility numbers, and by filtering out local0-local7 > > you are wanting to eliminate exactly half of them > > > > as such it's probably just as easy to list all the ones you want to > > record as it is to say *.* and subtract half of them. > > > > David Lang > > > > > Thanks. > > > -Joe > > > > > > > > > On Thu, 3 Sep 2009 17:05:04 -0700 (PDT) david at lang.hm wrote: > > > > > >> On Thu, 3 Sep 2009, Joe Williams wrote: > > >> > > >>> Hello again, > > >>> > > >>> I am trying to log everything (*.*) to /var/log/syslog but > > >>> local*.*. I tried a couple different ways to do this but didn't > > >>> find a solution. Is this possible? I have a couple services I > > >>> want to log to their own file rather than syslog, messages, etc. > > >> > > >> if you just do > > >> > > >> *.* /var/log/syslog > > >> > > >> that will write everything to that file > > >> > > >> I'm not sure what you are trying to say when you say 'but > > >> local*.*' above. > > >> > > >> David Lang > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -- > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > The information contained in this e-mail message and any accompanying > files is or may be confidential. If you are not the intended > recipient, any use, dissemination, reliance, forwarding, printing or > copying of this e-mail or any attached files is unauthorised. This > e-mail is subject to copyright. No part of it should be reproduced, > adapted or communicated without the written consent of the copyright > owner. If you have received this e-mail in error please advise the > sender immediately by return e-mail or telephone and delete all > copies. Fairfax does not guarantee the accuracy or completeness of > any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept > legal responsibility for the contents of this message or attached > files. _______________________________________________ rsyslog > mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -- Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ From david at lang.hm Sat Sep 5 04:04:59 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 19:04:59 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file Message-ID: I ahve a config file that fixes up broken syslog messages that has the following $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat & @192.168.210.8;fixsnareForwardFormat & ~ *.* /var/log/messages;TraditionalFormat *.* @192.168.210.8;TraditionalForwardFormat the upstream box is seeing things as I would expect, but the local /var/log/messages file is not is it incorrect to have two entries that both write to /var/log/messages? David Lang From david at lang.hm Sat Sep 5 08:03:23 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 4 Sep 2009 23:03:23 -0700 (PDT) Subject: [rsyslog] what happens if you have multiple selectors pointing at one file In-Reply-To: References: Message-ID: On Fri, 4 Sep 2009, david at lang.hm wrote: > I ahve a config file that fixes up broken syslog messages that has the > following > > $template fixsnareFormat,"%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template fixsnareForwardFormat,"<%pri%>%timereported% %HOSTNAME% MSWinEventLog %syslogtag%%msg:18:$:drop-last-lf%\n" > $template TraditionalFormat,"%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > $template TraditionalForwardFormat,"<%pri%>%timereported% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" > #$template TraditionalFormat,"%timegenerated% %syslogtag%%msg:::drop-last-lf%\n" > :syslogtag, startswith, "MSWinEventLog#011" *.* /var/log/messages;fixsnareFormat > & @192.168.210.8;fixsnareForwardFormat > & ~ > *.* /var/log/messages;TraditionalFormat > *.* @192.168.210.8;TraditionalForwardFormat > > > the upstream box is seeing things as I would expect, but the local > /var/log/messages file is not > > is it incorrect to have two entries that both write to /var/log/messages? never mind, I just spotted the extra *.* in there (nothing was reported when starting up) David Lang From oribani at gmail.com Sun Sep 6 03:52:04 2009 From: oribani at gmail.com (Ori Bani) Date: Sat, 5 Sep 2009 18:52:04 -0700 Subject: [rsyslog] Need help with RPM(yum) version on CentOS In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> References: <378058110908201753v41c58b4fx401efda639d058e4@mail.gmail.com> <20090821015920.M76525@npgx.com.au> <378058110909032221o54d44613v62ff526485ea8ea3@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FDFB@GRFEXC.intern.adiscon.com> Message-ID: <378058110909051852t1ae1f4dgd7f00d830b3b6284@mail.gmail.com> On 9/4/09, Rainer Gerhards wrote: > I have nothing technically to add to this discussion, but I would like > remind > you on the rsyslog wiki at > > http://wiki.rsyslog.com > > There already is one entry, but for an older version, not sure if that > helps: > > http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story > > In any case, I would appreciate if you could share any knowledge you gain > via > the wiki. I added my info, but that page was designed by someone who assumes you aren't using yum (or any similar system), so I hope I added it in an acceptable way >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Ori Bani >> Sent: Friday, September 04, 2009 7:21 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Need help with RPM(yum) version on CentOS >> >> >> I'm sorry if this isn't quite the right place to ask, since maybe no >> >> one here created the RPM that's in the CentOS base repository. But >> I >> >> am guessing people here have installed RPMs like this before and can >> >> help anyway.... >> >> >> >> When I ask yum on CentOS 5 about rsyslog, I get this (note older >> >> version - too bad): >> >> >> >> Available Packages >> >> Name : rsyslog >> >> Arch : i386 >> >> Version: 2.0.6 >> >> Release: 1.el5 >> >> Size : 198 k >> >> Repo : base >> >> Summary: Enhanced system logging and kernel message trapping daemons >> >> Description: >> >> Rsyslog is an enhanced multi-threaded syslogd supporting, among >> >> others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, >> >> filtering on any message part, and fine grain output format control. >> >> It is quite compatible to stock sysklogd and can be used as a drop- >> >> in replacement. Its advanced features make it suitable for >> >> enterprise-class, encryption protected syslog relay chains while at >> >> the same time being very easy to setup for the novice user. >> > >> > I use Scientific Linux 5.x and because they are RHEL derivatives I >> see the >> > same thing in the SL repo's. >> > >> > I have used the rsyslog from the repo's yet, all my rsyslog servers >> are >> > based >> > on EL4, but I'll try to help below. >> >> Thank you for your help. >> >> >> My questions are a little bit newbie... before I try installing >> >> this, I want to know what it's going to do to my system: >> >> >> >> 1) Will it disable syslogd and/or klogd? Or will it add itself >> using >> >> the "alternatives" paradigm so I can switch between them that way? >> >> If neither, does it include startup scripts at all? If they are >> there >> >> but not used by default, is there a recommended way to make the >> >> switch and not really screw things up? >> > >> > You should try this on a test box. I haven't tried it but I think it >> should >> > remove syslog RPM's from your installation and then install rsyslog. >> It >> > should >> > also make a /etc/syslog.conf.rpmsave file which you can reference for >> use in >> > /etc/rsyslog.conf >> >> I wouldn't actually expect it to remove any other packages - I've >> never seen a yum installation remove something else - that seems like >> trouble. In fact, it turns out that it didn't do a thing to >> syslog/ksyslogd. It just installed itself in parallel (and it's up to >> you to turn it on). Everything is in place (startup scripts, config >> file that is a mirror of syslog.conf, etc.) and you just have to >> >> chkconfig syslog off >> chkconfig rsyslog on >> service syslog stop >> service rsyslog start >> >> I guess if you're going to be more permanent: >> >> chkconfig --del syslog >> chkconfig --add rsyslog >> >> And use yum to remove ksyslogd/syslog >> >> >> 2) Will it add itself to my cron jobs? Specifically, I don't mind >> >> (for now) leaving the log rotation alone (don't let rsyslog manage >> my >> >> rotations). If it adds itself to my cron jobs, does that mean it >> >> will remove the logrotate cron job? >> > >> > Not sure sorry. You should grab the src.rpm file from CentOS, install >> it and >> > take a look at the rsyslog.spec and it'll show you what it does on >> the post >> > install section. >> >> That's above my skill level. Instead I tried it out. It also adds >> itself to /etc/logrotate.d/syslog so you don't have to touch any of >> this. Here is the modified file: >> >> /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler >> /var/log/boot.log /var/log/cron { >> sharedscripts >> postrotate >> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> >> /dev/null || true >> /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> >> /dev/null || true >> endscript >> } >> >> As you see, it left syslog there and added rsyslog. Because I have >> turned off syslog, this won't suddenly start it up, will it? >> >> >> 2.5) If I keep using the old logrotate with rsyslog, will that >> create >> >> any conflicts? >> > >> > I don't see how any conflicts will occur with logroate, since rsyslog >> > basically logs to the same files that syslog logs to. It's meant to >> be a >> > drop >> > in replacement. >> > >> > Maybe specific questions about rsyslog with CentOS (or other >> derivatives) >> > would actually be better in the CentOS or Scientific Linux mailing >> lists? >> >> I did, but it didn't help. That's disappointing. >> >> https://www.centos.org/modules/newbb/viewtopic.php?topic_id=21844&start >> =0#forumpost83694 >> >> >> Generally my aim is not to commit 100% to rsyslog yet, so I don't >> >> want to get to a situation where it's a lot of work to get back to >> >> the default syslog setup. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 20:17:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 13:17:50 -0500 Subject: [rsyslog] syslog server and reports Message-ID: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> I have some debian lenny servers sending their logs (via TCP) to a central rsyslog server. Every remote servers has at /etc/rsyslog.conf: *.* @@IP_CENTRAL_SERVER So, I can see in the central syslog server all logs without problems. I'm looking for a single and simple report, like logwatch for example who process all logs and send me in ONE mail or on ONE html page all resume info of all logs. I tried with logwatch and I didn't get this report I'm looking for. My question is? Is there any tool, script, app, etc which I run on the syslog server and give me the information of all servers in a way as simple as possible? Maybe in a single resume mail separated by a line for example? Thanks for your time. -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Sun Sep 6 21:00:46 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Sun, 6 Sep 2009 21:00:46 +0200 Subject: [rsyslog] syslog server and reports References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Probably not exactly what you look for, but maybe worth a try: http://www.phplogcon.org More reporting featueres are being tackled in the next couple of weeks. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Sunday, September 06, 2009 8:18 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] syslog server and reports > > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? > > Thanks for your time. > > -- > Regards; > Israel Garcia > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From igalvarez at gmail.com Sun Sep 6 21:20:34 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 14:20:34 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA706FE09@GRFEXC.intern.adiscon.com> Message-ID: <194a2c240909061220k25cf03e4ycb7dcf379d45ab8f@mail.gmail.com> Hi Rainer, thanks for your soon answer.. On 9/6/09, Rainer Gerhards wrote: > Probably not exactly what you look for, but maybe worth a try: > > http://www.phplogcon.org I have installed phplogcon but, it's not whay I'm looking for. I need an email, a simple daily email with the reports of all my servers. I've tried to setup logwatch and logcheck but I could not get what I want. regards, Israel. > > More reporting featueres are being tackled in the next couple of weeks. > > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia >> Sent: Sunday, September 06, 2009 8:18 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] syslog server and reports >> >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? >> >> Thanks for your time. >> >> -- >> Regards; >> Israel Garcia >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Sun Sep 6 23:19:35 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 14:19:35 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > I have some debian lenny servers sending their logs (via TCP) to a > central rsyslog server. > Every remote servers has at /etc/rsyslog.conf: > > *.* @@IP_CENTRAL_SERVER > > So, I can see in the central syslog server all logs without problems. > I'm looking for a single and simple report, like logwatch for example > who process all logs and send me in ONE mail or on ONE html page all > resume info of all logs. I tried with logwatch and I didn't get this > report I'm looking for. > > My question is? > Is there any tool, script, app, etc which I run on the syslog server > and give me the information of all servers in a way as simple as > possible? Maybe in a single resume mail separated by a line for > example? there are a lot of products and projects out there to analyse logs and generate reports. the problem is that what I am interested in seeing in a report may or may not match what you are interested in seeing. also, most of this effort is taking place within originizations that have large volumes of logs, so distilling it down to a single report or e-mail requires that a lot of detail gets left out (and that goes back to exactly what you are interested in seeing) when you say you want one page that shows you 'everything', what is it that you want to see? are there particular messages that you want to see if they show up even once? or are you interested in simplifying log messages into categories and seeing how many messages in each category you have. do you only care about the logs showing up sometime during the day? or are you interested in the trending of how many logs you get each second throughout the day (or anything in between) unfortunantly the result of all these questions probably means that you will need to customize whatever you use to exactly the report that you want. large companies can spend millions of dollars on systems and software to alert, report, and query their logs. I am currently getting ~300M log messages/day and I distill it down to a single e-mail report that I look at (and generate additional reports with subsets of the data for other people to look at). the best advice I ever got was to use the approach termed 'artificial ignorance' start off with all your logs for any log type that you can categorize create a summary of that log type (even if it's an unimportant log, count it because the number of times an unimportant thing happens can be important) look at what's left and repeat the process after several iterations of this you end up with the vast majority of your logs summarized and a report of "what's left", any new messages that you have never seen before (which usually mean they are important) show up in the "what's left" bucket and tend to stand out you do need to keep on top of this, upgrades to systems, new installs, etc cause new logs to show up, if you categorize and summarize them your final report stays small, if you let things slide for several months the final report can end up very large (and therefor useless) David Lang From igalvarez at gmail.com Mon Sep 7 01:40:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 18:40:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> Message-ID: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> I have some debian lenny servers sending their logs (via TCP) to a >> central rsyslog server. >> Every remote servers has at /etc/rsyslog.conf: >> >> *.* @@IP_CENTRAL_SERVER >> >> So, I can see in the central syslog server all logs without problems. >> I'm looking for a single and simple report, like logwatch for example >> who process all logs and send me in ONE mail or on ONE html page all >> resume info of all logs. I tried with logwatch and I didn't get this >> report I'm looking for. >> >> My question is? >> Is there any tool, script, app, etc which I run on the syslog server >> and give me the information of all servers in a way as simple as >> possible? Maybe in a single resume mail separated by a line for >> example? > > there are a lot of products and projects out there to analyse logs and > generate reports. > > the problem is that what I am interested in seeing in a report may or may > not match what you are interested in seeing. > > also, most of this effort is taking place within originizations that have > large volumes of logs, so distilling it down to a single report or e-mail > requires that a lot of detail gets left out (and that goes back to exactly > what you are interested in seeing) > > when you say you want one page that shows you 'everything', what is it > that you want to see? Hi, David I mean, a report like logwatch use to send me everyday from each server. As I said before, I'm collecting all servers logs (syslog and auth.log) into my central syslog, so I need some tool like logwatch running on the collector which send in one mail or in one html page. . I tried to configure logwatch in the collector without sucess. That's what I need. :-) thanks. regards, Israel > > are there particular messages that you want to see if they show up even > once? or are you interested in simplifying log messages into categories > and seeing how many messages in each category you have. > > do you only care about the logs showing up sometime during the day? or are > you interested in the trending of how many logs you get each second > throughout the day (or anything in between) > > unfortunantly the result of all these questions probably means that you > will need to customize whatever you use to exactly the report that you > want. > > large companies can spend millions of dollars on systems and software to > alert, report, and query their logs. > > I am currently getting ~300M log messages/day and I distill it down to a > single e-mail report that I look at (and generate additional reports with > subsets of the data for other people to look at). > > > the best advice I ever got was to use the approach termed 'artificial > ignorance' > > start off with all your logs > > for any log type that you can categorize create a summary of that log type > (even if it's an unimportant log, count it because the number of times an > unimportant thing happens can be important) > > look at what's left and repeat the process > > after several iterations of this you end up with the vast majority of your > logs summarized and a report of "what's left", any new messages that you > have never seen before (which usually mean they are important) show up in > the "what's left" bucket and tend to stand out > > you do need to keep on top of this, upgrades to systems, new installs, > etc cause new logs to show up, if you categorize and summarize them your > final report stays small, if you let things slide for several months the > final report can end up very large (and therefor useless) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 02:15:40 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 17:15:40 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> I have some debian lenny servers sending their logs (via TCP) to a >>> central rsyslog server. >>> Every remote servers has at /etc/rsyslog.conf: >>> >>> *.* @@IP_CENTRAL_SERVER >>> >>> So, I can see in the central syslog server all logs without problems. >>> I'm looking for a single and simple report, like logwatch for example >>> who process all logs and send me in ONE mail or on ONE html page all >>> resume info of all logs. I tried with logwatch and I didn't get this >>> report I'm looking for. >>> >>> My question is? >>> Is there any tool, script, app, etc which I run on the syslog server >>> and give me the information of all servers in a way as simple as >>> possible? Maybe in a single resume mail separated by a line for >>> example? >> >> there are a lot of products and projects out there to analyse logs and >> generate reports. >> >> the problem is that what I am interested in seeing in a report may or may >> not match what you are interested in seeing. >> >> also, most of this effort is taking place within originizations that have >> large volumes of logs, so distilling it down to a single report or e-mail >> requires that a lot of detail gets left out (and that goes back to exactly >> what you are interested in seeing) >> >> when you say you want one page that shows you 'everything', what is it >> that you want to see? > Hi, David > I mean, a report like logwatch use to send me everyday from each > server. As I said before, I'm collecting all servers logs (syslog and > auth.log) into my central syslog, so I need some tool like logwatch > running on the collector which send in one mail or in one html page. > . > I tried to configure logwatch in the collector without sucess. > > That's what I need. :-) ok, so you want the report that you get from logwatch, that simplifies things. when you say you can't get it to work on the collector box, more info is needed. does logwatch give you the info that you want about the collector box? do you put the logs from all servers in one file? or do you split them by host? (or split them in other ways) how does logwatch fail? does it crash? give you incorrect information? other? David Lang From igalvarez at gmail.com Mon Sep 7 03:47:14 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 20:47:14 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> Message-ID: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> I have some debian lenny servers sending their logs (via TCP) to a >>>> central rsyslog server. >>>> Every remote servers has at /etc/rsyslog.conf: >>>> >>>> *.* @@IP_CENTRAL_SERVER >>>> >>>> So, I can see in the central syslog server all logs without problems. >>>> I'm looking for a single and simple report, like logwatch for example >>>> who process all logs and send me in ONE mail or on ONE html page all >>>> resume info of all logs. I tried with logwatch and I didn't get this >>>> report I'm looking for. >>>> >>>> My question is? >>>> Is there any tool, script, app, etc which I run on the syslog server >>>> and give me the information of all servers in a way as simple as >>>> possible? Maybe in a single resume mail separated by a line for >>>> example? >>> >>> there are a lot of products and projects out there to analyse logs and >>> generate reports. >>> >>> the problem is that what I am interested in seeing in a report may or may >>> not match what you are interested in seeing. >>> >>> also, most of this effort is taking place within originizations that have >>> large volumes of logs, so distilling it down to a single report or e-mail >>> requires that a lot of detail gets left out (and that goes back to >>> exactly >>> what you are interested in seeing) >>> >>> when you say you want one page that shows you 'everything', what is it >>> that you want to see? >> Hi, David >> I mean, a report like logwatch use to send me everyday from each >> server. As I said before, I'm collecting all servers logs (syslog and >> auth.log) into my central syslog, so I need some tool like logwatch >> running on the collector which send in one mail or in one html page. >> . >> I tried to configure logwatch in the collector without sucess. >> >> That's what I need. :-) > > ok, so you want the report that you get from logwatch, that simplifies > things. > > when you say you can't get it to work on the collector box, more info is > needed. > > does logwatch give you the info that you want about the collector box? My scenario: I added this two lines in /etc/rsyslog.conf of all exporting servers: auth,authpriv.* @@xx.xx.xx.xx *.*;auth,authpriv.none @@xx.xx.xx.xx In the collector syslog and auth.log files I see logs coming from those servers. logwatch.conf file is the default. I run logwatch (testing mode) in the collector and it merge logs from all servers, so you can not identify which log output is belongs to. It looks like all logs are from the collector server. here you can see a part of logwatch output: In my case deb2 is the hostname of the collector and debian is the hostname of one exporter. deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) #################### Processing Initiated: Sun Sep 6 21:35:29 2009 Date Range Processed: today ( 2009-Sep-06 ) Period is day. Detail Level of Output: 0 Type of Output/Format: stdout / text Logfiles for Host: deb2 ################################################################## ###This logs are from deb2 Installed: libdate-manip-perl 5.54-1 lockfile-progs 0.1.11-0.1 logtail 1.2.69 logwatch 7.3.6.cvs20080702-2 postfix 2.5.5-1.1 . . . . . --------------------- pam_unix Begin ------------------------ ### All this logs entries from user test123 are from one exporter server (debian). sshd: Authentication Failures: root (localhost): 1 Time(s) su: Authentication Failures: test123(1003) -> root: 2 Time(s) Sessions Opened: root -> logcheck: 17 Time(s) root -> root: 9 Time(s) sudo: Authentication Failures: test123(0) -> test123: 1 Time(s) **Unmatched Entries** useradd: failed adding user `test', data deleted: 1 Time(s) ---------------------- Connections (secure-log) End ------------------------- ============================================================================== ### This is from exporter debian server. test123 => root --------------- /bin/su - 1 Times. ---------------------- Sudo (secure-log) End ------------------------- ## This df output is from deb2 (collector) --------------------- Disk Space Begin ------------------------ Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.5G 2.0G 5.2G 28% / ---------------------- Disk Space End ------------------------- ###################### Logwatch End ################### As you can see, it seems like the report belongs to deb2 server and it's not. I'd be happy if at least logwatch put some tags at the beginning of each line to identify the source. thanks again. regards, israel. > > do you put the logs from all servers in one file? or do you split them by > host? (or split them in other ways) > > how does logwatch fail? does it crash? give you incorrect information? > other? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From david at lang.hm Mon Sep 7 04:23:51 2009 From: david at lang.hm (david at lang.hm) Date: Sun, 6 Sep 2009 19:23:51 -0700 (PDT) Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: On Sun, 6 Sep 2009, Israel Garcia wrote: > On 9/6/09, david at lang.hm wrote: >> On Sun, 6 Sep 2009, Israel Garcia wrote: >> >>> On 9/6/09, david at lang.hm wrote: >>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>> >>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>> central rsyslog server. >>>>> Every remote servers has at /etc/rsyslog.conf: >>>>> >>>>> *.* @@IP_CENTRAL_SERVER >>>>> >>>>> So, I can see in the central syslog server all logs without problems. >>>>> I'm looking for a single and simple report, like logwatch for example >>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>> report I'm looking for. >>>>> >>>>> My question is? >>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>> and give me the information of all servers in a way as simple as >>>>> possible? Maybe in a single resume mail separated by a line for >>>>> example? >>>> >>>> there are a lot of products and projects out there to analyse logs and >>>> generate reports. >>>> >>>> the problem is that what I am interested in seeing in a report may or may >>>> not match what you are interested in seeing. >>>> >>>> also, most of this effort is taking place within originizations that have >>>> large volumes of logs, so distilling it down to a single report or e-mail >>>> requires that a lot of detail gets left out (and that goes back to >>>> exactly >>>> what you are interested in seeing) >>>> >>>> when you say you want one page that shows you 'everything', what is it >>>> that you want to see? >>> Hi, David >>> I mean, a report like logwatch use to send me everyday from each >>> server. As I said before, I'm collecting all servers logs (syslog and >>> auth.log) into my central syslog, so I need some tool like logwatch >>> running on the collector which send in one mail or in one html page. >>> . >>> I tried to configure logwatch in the collector without sucess. >>> >>> That's what I need. :-) >> >> ok, so you want the report that you get from logwatch, that simplifies >> things. >> >> when you say you can't get it to work on the collector box, more info is >> needed. >> >> does logwatch give you the info that you want about the collector box? > > My scenario: > I added this two lines in /etc/rsyslog.conf of all exporting servers: > > auth,authpriv.* @@xx.xx.xx.xx > *.*;auth,authpriv.none @@xx.xx.xx.xx > > In the collector syslog and auth.log files I see logs coming from > those servers. > > logwatch.conf file is the default. > > I run logwatch (testing mode) in the collector and it merge logs from > all servers, so you can not identify which log output is belongs to. > It looks like all logs are from the collector server. ahh, that's the problem. unforutnantly fixing this would take some significant surgury to logwatch. it assumes that all the logs it is dealing with are from the local box and therefor it ignores the server tag in the output. you could use the rsyslog dynafiles feature to create a different file for each server, run logwatch against each of those files, and then combine the reports (including adding text to tell you which server is up next) David Lang > here you can see a part of logwatch output: > > In my case deb2 is the hostname of the collector and debian is the > hostname of one exporter. > > deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today > > ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) > #################### > Processing Initiated: Sun Sep 6 21:35:29 2009 > Date Range Processed: today > ( 2009-Sep-06 ) > Period is day. > Detail Level of Output: 0 > Type of Output/Format: stdout / text > Logfiles for Host: deb2 > ################################################################## > > ###This logs are from deb2 > Installed: > libdate-manip-perl 5.54-1 > lockfile-progs 0.1.11-0.1 > logtail 1.2.69 > logwatch 7.3.6.cvs20080702-2 > postfix 2.5.5-1.1 > . > . > . > . > . > --------------------- pam_unix Begin ------------------------ > ### All this logs entries from user test123 are from one exporter > server (debian). > sshd: > Authentication Failures: > root (localhost): 1 Time(s) > > su: > Authentication Failures: > test123(1003) -> root: 2 Time(s) > Sessions Opened: > root -> logcheck: 17 Time(s) > root -> root: 9 Time(s) > > sudo: > Authentication Failures: > test123(0) -> test123: 1 Time(s) > > **Unmatched Entries** > useradd: failed adding user `test', data deleted: 1 Time(s) > > ---------------------- Connections (secure-log) End ------------------------- > > > ============================================================================== > ### This is from exporter debian server. > test123 => root > --------------- > /bin/su - 1 Times. > > ---------------------- Sudo (secure-log) End ------------------------- > > ## This df output is from deb2 (collector) > --------------------- Disk Space Begin ------------------------ > > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 7.5G 2.0G 5.2G 28% / > > ---------------------- Disk Space End ------------------------- > > ###################### Logwatch End ################### > > As you can see, it seems like the report belongs to deb2 server and it's not. > > I'd be happy if at least logwatch put some tags at the beginning of > each line to identify the source. > > thanks again. > regards, > israel. > > > > > >> >> do you put the logs from all servers in one file? or do you split them by >> host? (or split them in other ways) >> >> how does logwatch fail? does it crash? give you incorrect information? >> other? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > > > From igalvarez at gmail.com Mon Sep 7 04:46:41 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Sun, 6 Sep 2009 21:46:41 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> Message-ID: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> On 9/6/09, david at lang.hm wrote: > On Sun, 6 Sep 2009, Israel Garcia wrote: > >> On 9/6/09, david at lang.hm wrote: >>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>> >>>> On 9/6/09, david at lang.hm wrote: >>>>> On Sun, 6 Sep 2009, Israel Garcia wrote: >>>>> >>>>>> I have some debian lenny servers sending their logs (via TCP) to a >>>>>> central rsyslog server. >>>>>> Every remote servers has at /etc/rsyslog.conf: >>>>>> >>>>>> *.* @@IP_CENTRAL_SERVER >>>>>> >>>>>> So, I can see in the central syslog server all logs without problems. >>>>>> I'm looking for a single and simple report, like logwatch for example >>>>>> who process all logs and send me in ONE mail or on ONE html page all >>>>>> resume info of all logs. I tried with logwatch and I didn't get this >>>>>> report I'm looking for. >>>>>> >>>>>> My question is? >>>>>> Is there any tool, script, app, etc which I run on the syslog server >>>>>> and give me the information of all servers in a way as simple as >>>>>> possible? Maybe in a single resume mail separated by a line for >>>>>> example? >>>>> >>>>> there are a lot of products and projects out there to analyse logs and >>>>> generate reports. >>>>> >>>>> the problem is that what I am interested in seeing in a report may or >>>>> may >>>>> not match what you are interested in seeing. >>>>> >>>>> also, most of this effort is taking place within originizations that >>>>> have >>>>> large volumes of logs, so distilling it down to a single report or >>>>> e-mail >>>>> requires that a lot of detail gets left out (and that goes back to >>>>> exactly >>>>> what you are interested in seeing) >>>>> >>>>> when you say you want one page that shows you 'everything', what is it >>>>> that you want to see? >>>> Hi, David >>>> I mean, a report like logwatch use to send me everyday from each >>>> server. As I said before, I'm collecting all servers logs (syslog and >>>> auth.log) into my central syslog, so I need some tool like logwatch >>>> running on the collector which send in one mail or in one html page. >>>> . >>>> I tried to configure logwatch in the collector without sucess. >>>> >>>> That's what I need. :-) >>> >>> ok, so you want the report that you get from logwatch, that simplifies >>> things. >>> >>> when you say you can't get it to work on the collector box, more info is >>> needed. >>> >>> does logwatch give you the info that you want about the collector box? >> >> My scenario: >> I added this two lines in /etc/rsyslog.conf of all exporting servers: >> >> auth,authpriv.* @@xx.xx.xx.xx >> *.*;auth,authpriv.none @@xx.xx.xx.xx >> >> In the collector syslog and auth.log files I see logs coming from >> those servers. >> >> logwatch.conf file is the default. >> >> I run logwatch (testing mode) in the collector and it merge logs from >> all servers, so you can not identify which log output is belongs to. >> It looks like all logs are from the collector server. > > ahh, that's the problem. > > unforutnantly fixing this would take some significant surgury to logwatch. > it assumes that all the logs it is dealing with are from the local box and > therefor it ignores the server tag in the output. > > you could use the rsyslog dynafiles feature to create a different file for > each server, run logwatch against each of those files, and then combine > the reports (including adding text to tell you which server is up next) Hi David, I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. > > David Lang > >> here you can see a part of logwatch output: >> >> In my case deb2 is the hostname of the collector and debian is the >> hostname of one exporter. >> >> deb2:/etc/cron.daily# /usr/sbin/logwatch --range Today >> >> ################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) >> #################### >> Processing Initiated: Sun Sep 6 21:35:29 2009 >> Date Range Processed: today >> ( 2009-Sep-06 ) >> Period is day. >> Detail Level of Output: 0 >> Type of Output/Format: stdout / text >> Logfiles for Host: deb2 >> ################################################################## >> >> ###This logs are from deb2 >> Installed: >> libdate-manip-perl 5.54-1 >> lockfile-progs 0.1.11-0.1 >> logtail 1.2.69 >> logwatch 7.3.6.cvs20080702-2 >> postfix 2.5.5-1.1 >> . >> . >> . >> . >> . >> --------------------- pam_unix Begin ------------------------ >> ### All this logs entries from user test123 are from one exporter >> server (debian). >> sshd: >> Authentication Failures: >> root (localhost): 1 Time(s) >> >> su: >> Authentication Failures: >> test123(1003) -> root: 2 Time(s) >> Sessions Opened: >> root -> logcheck: 17 Time(s) >> root -> root: 9 Time(s) >> >> sudo: >> Authentication Failures: >> test123(0) -> test123: 1 Time(s) >> >> **Unmatched Entries** >> useradd: failed adding user `test', data deleted: 1 Time(s) >> >> ---------------------- Connections (secure-log) End >> ------------------------- >> >> >> ============================================================================== >> ### This is from exporter debian server. >> test123 => root >> --------------- >> /bin/su - 1 Times. >> >> ---------------------- Sudo (secure-log) End ------------------------- >> >> ## This df output is from deb2 (collector) >> --------------------- Disk Space Begin ------------------------ >> >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda1 7.5G 2.0G 5.2G 28% / >> >> ---------------------- Disk Space End ------------------------- >> >> ###################### Logwatch End ################### >> >> As you can see, it seems like the report belongs to deb2 server and it's >> not. >> >> I'd be happy if at least logwatch put some tags at the beginning of >> each line to identify the source. >> >> thanks again. >> regards, >> israel. >> >> >> >> >> >>> >>> do you put the logs from all servers in one file? or do you split them by >>> host? (or split them in other ways) >>> >>> how does logwatch fail? does it crash? give you incorrect information? >>> other? >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From rgerhards at hq.adiscon.com Mon Sep 7 15:18:13 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 7 Sep 2009 15:18:13 +0200 Subject: [rsyslog] abort in 4.2.1 References: <000401ca25a1$49d004bd$100013ac@intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD87@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD8A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD97@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FD9A@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FDAF@GRFEXC.intern.adiscon.com><1251715849.4897.13.camel@rgf11><1251886461.5821.8.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FDEF@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE1F@GRFEXC.intern.adiscon.com> Hi all, after some struggle, a new status: Thanks to David's data sets, I think I have finally been able to find a code spot that may be troublesome. It also is in an area that we already had under suspicion. While it is too early to say if I finally found the issue, it looks very promising. If I am right, the problem is actually environment-induced, what would also explain why other users did not yet report anything and I did not see anything in my lab so far. The ultimate root cause may even be a formatting error in another rsyslogd instance further up in the relay chain. If so, I'll try to work upward from where to problem currently occurs to the above-it root cause. I just thought I share this new information with you. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Thursday, September 03, 2009 12:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] abort in 4.2.1 > > Hi David, > > > I haven't gone back to the 3.x series, but I did several more runs > with > > 4.2.0 doing the folloiwng > > > > killall syslogd; tcpdump -n -s 0 -w rsyslog.sniff-10 -i eth0 & > > rsyslogd -c4 -x -d >rsyslog.debug-10 2>&1 ; killall tcpdump; syslogd > -r > > -h ; mv /core /core-4.2.0-10 > > > > I have several complete steps, as well as several partial sets of > data. > > I > > will gzip them and attempt to send them to you directly. > > Thanks for the data set, I am right now working on it. Unfortunately, > as I > feared, the core files do not really help. There is a big mismatch > between > your system environment and mine, and so gdb is not able to extract any > useful information. All I see is that there are six threads in the > system, > and the rest is almost only question marks. > > So it would be great if you could issue the gdb commands in your > environment > and let me know the outcome. > > Thanks, > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From DGillies at fairfaxdigital.com.au Tue Sep 8 02:15:29 2009 From: DGillies at fairfaxdigital.com.au (David Gillies) Date: Tue, 8 Sep 2009 10:15:29 +1000 Subject: [rsyslog] syslog server and reports In-Reply-To: <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: Hi Israel, Its been a while since I last used it, but I'm pretty sure that epylog can handle reporting on log files with multiple hosts: https://fedorahosted.org/epylog/ David Gillies Linux Systems engineer Digital Infrastructure Services Fairfax Digital -----Original Message----- From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia Sent: Monday, 7 September 2009 12:47 PM To: rsyslog-users Subject: Re: [rsyslog] syslog server and reports I'll try this way.. but do you know if there another tool more simple to get jmy report? thanks in advance. regards, israel. The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files. From igalvarez at gmail.com Tue Sep 8 05:19:50 2009 From: igalvarez at gmail.com (Israel Garcia) Date: Mon, 7 Sep 2009 22:19:50 -0500 Subject: [rsyslog] syslog server and reports In-Reply-To: References: <194a2c240909061117s63fef477m795cc89287fa6163@mail.gmail.com> <194a2c240909061640y244de062h789025cf51298c1c@mail.gmail.com> <194a2c240909061847g2c0a4281q86452087adbfe14f@mail.gmail.com> <194a2c240909061946h29e900c9vde534620b3a395e5@mail.gmail.com> Message-ID: <194a2c240909072019q618594cay38b7ca7202d13b10@mail.gmail.com> On 9/7/09, David Gillies wrote: > Hi Israel, Hi David, > > Its been a while since I last used it, but I'm pretty sure that epylog can > handle reporting on log files with multiple hosts: > > https://fedorahosted.org/epylog/ umm... sounds good.. I see my rsyslog collector has the latest version of epylog..I'll try right now..:-) thanks regards, israel. > > David Gillies > Linux Systems engineer > Digital Infrastructure Services > Fairfax Digital > > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Israel Garcia > Sent: Monday, 7 September 2009 12:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] syslog server and reports > > > > I'll try this way.. but do you know if there another tool more simple to get > jmy report? > thanks in advance. > > regards, > israel. > The information contained in this e-mail message and any accompanying files > is or may be confidential. If you are not the intended recipient, any use, > dissemination, reliance, forwarding, printing or copying of this e-mail or > any attached files is unauthorised. This e-mail is subject to copyright. No > part of it should be reproduced, adapted or communicated without the written > consent of the copyright owner. If you have received this e-mail in error > please advise the sender immediately by return e-mail or telephone and > delete all copies. Fairfax does not guarantee the accuracy or completeness > of any information contained in this e-mail or attached files. Internet > communications are not secure, therefore Fairfax does not accept legal > responsibility for the contents of this message or attached files. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Regards; Israel Garcia From henry78 at gmx.at Tue Sep 8 09:03:49 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 09:03:49 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <1252092330.924.24.camel@eberhe.office.chipkarte.at> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> Message-ID: <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Hello! Tried it with various log locations (e.g. /tmp/my.log), neither worked. Is this worth ab bug? -- kind regards, Henry On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > Hi! > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > ubuntu: > > -----8<----- > $ModLoad imtcp > $InputTCPServerRun 514 > > # some dynamic templates > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > # log remote local1 to dynamic diretory > if $fromhost-ip != '127.0.0.1' and \ > $syslogfacility-text == 'local1' \ > then -?DYNlocal1 > ----->8----- > > I created /var/log/remote with sufficient privileges. > > Unfortunately this doesn't work. rsyslog crates a folder named after the > remote host (myhostname) and creates the file local1.log (again: > sufficient permissions: syslog:syslog 640). But it doesn't write to that > file, but logs the error: > > -----8<----- > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > discarding message > ----->8----- > > As you might guess my question is: Why isn't rsyslog able to open a file > it is able to create? Any help or hint is really appreciated. > From rgerhards at hq.adiscon.com Tue Sep 8 09:55:21 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 09:55:21 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> can you provide a debug log? > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 9:04 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > Hello! > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > Is this worth ab bug? > > -- > kind regards, Henry > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > Hi! > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > ubuntu: > > > > -----8<----- > > $ModLoad imtcp > > $InputTCPServerRun 514 > > > > # some dynamic templates > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > # log remote local1 to dynamic diretory > > if $fromhost-ip != '127.0.0.1' and \ > > $syslogfacility-text == 'local1' \ > > then -?DYNlocal1 > > ----->8----- > > > > I created /var/log/remote with sufficient privileges. > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > the > > remote host (myhostname) and creates the file local1.log (again: > > sufficient permissions: syslog:syslog 640). But it doesn't write to > that > > file, but logs the error: > > > > -----8<----- > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > discarding message > > ----->8----- > > > > As you might guess my question is: Why isn't rsyslog able to open a > file > > it is able to create? Any help or hint is really appreciated. > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:30:17 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:30:17 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Hi, I got the debug log, it was too big to be sent via the list (but I got it as list admin). I see that you drop privileges to the user "syslog". This probably explains what happens. I think the file is created before you drop privileges, but can then no longer be written when running in the new security context. Could you verify that the user "syslog" can access this file? Also, could you temporarily remove the Privilege drop? Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 9:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From henry78 at gmx.at Tue Sep 8 12:31:35 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:31:35 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> Message-ID: <1252405895.17741.41.camel@eberhe.office.chipkarte.at> Hmm... a simple '-d' debug doesn't seem to give enough information, see attached rsyslogd.debug.full. Attached log starts with processing of the remote logging because the full log is too large for this list. Note, that this was started with an empty /var/log/remote and the file /var/log/remote/myhostname/local1.log got created during debug run. rsysloghost='loghost', remotehost='remotehost'. Thanks for having a look at this, -- regards, Henry On Di, 2009-09-08 at 09:55 +0200, Rainer Gerhards wrote: > can you provide a debug log? > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Henry > > Sent: Tuesday, September 08, 2009 9:04 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > Hello! > > > > Tried it with various log locations (e.g. /tmp/my.log), neither worked. > > Is this worth ab bug? > > > > -- > > kind regards, Henry > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > Hi! > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 on > > > ubuntu: > > > > > > -----8<----- > > > $ModLoad imtcp > > > $InputTCPServerRun 514 > > > > > > # some dynamic templates > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > # log remote local1 to dynamic diretory > > > if $fromhost-ip != '127.0.0.1' and \ > > > $syslogfacility-text == 'local1' \ > > > then -?DYNlocal1 > > > ----->8----- > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named after > > the > > > remote host (myhostname) and creates the file local1.log (again: > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > that > > > file, but logs the error: > > > > > > -----8<----- > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' - > > > discarding message > > > ----->8----- > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > file > > > it is able to create? Any help or hint is really appreciated. > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com -------------- next part -------------- 5054.727606022:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. 5054.727609261:main thread: main queue: EnqueueMsg advised worker start 5054.727630769:imuxsock.c: --------imuxsock calling select, active file descriptors (max 3): 3 5054.727662249:imtcp.c: -------- calling select, active fds (max 4): 4 5054.727668059:main thread: initialization completed, transitioning to regular run mode 5056.400110663:imtcp.c: New connect on NSD 0x65a690. 5056.400240204:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400255357:imtcp.c: netstream 0x683350 with new data 5056.400272148:imtcp.c: logmsg: flags 20, from 'remotehost', msg Sep 8 12:17:36 remotehost root: test by henry 5056.400274671:imtcp.c: Message has legacy syslog format. 5056.400279417:imtcp.c: main queue: entry added, size now 1 entries 5056.400282172:imtcp.c: wtpAdviseMaxWorkers signals busy 5056.400290169:imtcp.c: main queue: EnqueueMsg advised worker start 5056.400301360:imtcp.c: -------- calling select, active fds (max 5): 4 5 5056.400311400:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400328690:main queue:Reg/w0: result of expression evaluation: 0 5056.400337430:main queue:Reg/w0: result of expression evaluation: 0 5056.400346474:main queue:Reg/w0: result of expression evaluation: 0 5056.400351644:main queue:Reg/w0: result of expression evaluation: 0 5056.400358215:main queue:Reg/w0: result of expression evaluation: 0 5056.400366212:main queue:Reg/w0: result of expression evaluation: 0 5056.400371208:main queue:Reg/w0: result of expression evaluation: 0 5056.400381921:main queue:Reg/w0: result of expression evaluation: 0 5056.400388679:main queue:Reg/w0: result of expression evaluation: 0 5056.400395611:main queue:Reg/w0: result of expression evaluation: 0 5056.400403792:main queue:Reg/w0: result of expression evaluation: 0 5056.400420697:main queue:Reg/w0: result of expression evaluation: 0 5056.400430773:main queue:Reg/w0: result of expression evaluation: 1 5056.400434138:main queue:Reg/w0: Called action, logging to builtin-file 5056.400440938:main queue:Reg/w0: (DYNlocal1) 5056.400507577:main queue:Reg/w0: Called LogError, msg: Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400531156:main queue:Reg/w0: logmsg: flags 1, from 'loghost', msg Could not open dynamic file '/var/log/remote/remotehost/local1.log' - discarding message 5056.400537445:main queue:Reg/w0: Message has legacy syslog format. 5056.400540197:main queue:Reg/w0: main queue: entry added, size now 1 entries 5056.400542937:main queue:Reg/w0: wtpAdviseMaxWorkers signals busy 5056.400544844:main queue:Reg/w0: main queue: EnqueueMsg advised worker start 5056.400548374:main queue:Reg/w0: Removed entry 0 for file '[OPEN FAILED]' from dynaCache. 5056.400550633:main queue:Reg/w0: Action requested to be suspended, done that. 5056.400554546:main queue:Reg/w0: main queue: entry deleted, state 0, size now 0 entries 5056.400562730:main queue:Reg/w0: result of expression evaluation: 0 5056.400572312:main queue:Reg/w0: result of expression evaluation: 1 5056.400575746:main queue:Reg/w0: Called action, logging to builtin-file 5056.400580715:main queue:Reg/w0: (/var/log/syslog) 5056.400591557:main queue:Reg/w0: result of expression evaluation: 0 5056.400607572:main queue:Reg/w0: result of expression evaluation: 0 5056.400612225:main queue:Reg/w0: result of expression evaluation: 0 5056.400619969:main queue:Reg/w0: result of expression evaluation: 0 5056.400627299:main queue:Reg/w0: result of expression evaluation: 0 5056.400633371:main queue:Reg/w0: result of expression evaluation: 0 5056.400639122:main queue:Reg/w0: result of expression evaluation: 0 5056.400644767:main queue:Reg/w0: result of expression evaluation: 0 5056.400651994:main queue:Reg/w0: result of expression evaluation: 1 5056.400656433:main queue:Reg/w0: Called action, logging to builtin-file 5056.400668930:main queue:Reg/w0: (/var/log/debug) 5056.400687895:main queue:Reg/w0: result of expression evaluation: 0 5056.400702536:main queue:Reg/w0: result of expression evaluation: 0 5056.400705763:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting for work. From henry78 at gmx.at Tue Sep 8 12:41:37 2009 From: henry78 at gmx.at (Henry) Date: Tue, 08 Sep 2009 12:41:37 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> References: <1252092330.924.24.camel@eberhe.office.chipkarte.at> <1252393429.17741.22.camel@eberhe.office.chipkarte.at> <9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> Message-ID: <1252406497.17741.48.camel@eberhe.office.chipkarte.at> The file (and folder) are created by the syslog user and definitely accessible. But it works if i don't drop privileges. So I'll investigate this further and report back. Thanks for pushing me that far. -- regard, Henry On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > Hi, > > I got the debug log, it was too big to be sent via the list (but I got it as > list admin). I see that you drop privileges to the user "syslog". This > probably explains what happens. I think the file is created before you drop > privileges, but can then no longer be written when running in the new > security context. Could you verify that the user "syslog" can access this > file? Also, could you temporarily remove the Privilege drop? > > Thanks, > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 08, 2009 9:55 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > message > > > > can you provide a debug log? > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > Hello! > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > worked. > > > Is this worth ab bug? > > > > > > -- > > > kind regards, Henry > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > Hi! > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog v4 > > on > > > > ubuntu: > > > > > > > > -----8<----- > > > > $ModLoad imtcp > > > > $InputTCPServerRun 514 > > > > > > > > # some dynamic templates > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > # log remote local1 to dynamic diretory > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > $syslogfacility-text == 'local1' \ > > > > then -?DYNlocal1 > > > > ----->8----- > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > after > > > the > > > > remote host (myhostname) and creates the file local1.log (again: > > > > sufficient permissions: syslog:syslog 640). But it doesn't write to > > > that > > > > file, but logs the error: > > > > > > > > -----8<----- > > > > Could not open dynamic file '/var/log/remote/myhostname/local1.log' > > - > > > > discarding message > > > > ----->8----- > > > > > > > > As you might guess my question is: Why isn't rsyslog able to open a > > > file > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 12:47:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 12:47:05 +0200 Subject: [rsyslog] Could not open dynamic file ... - discarding message References: <1252092330.924.24.camel@eberhe.office.chipkarte.at><1252393429.17741.22.camel@eberhe.office.chipkarte.at><9B6E2A8877C38245BFB15CC491A11DA706FE26@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE2C@GRFEXC.intern.adiscon.com> <1252406497.17741.48.camel@eberhe.office.chipkarte.at> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE2E@GRFEXC.intern.adiscon.com> It is important to know that the PrivDrop directive set was a quick and dirty "let's implement it as far as possible, some is better than nothing" approach. It is expected that a couple of things break if it is used. Of course, if the users has proper rights, what you intend to do should work. I just wanted to alert you on the state of this feature (a mailing list search probably brings up more, but I have no time right now to do this). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Henry > Sent: Tuesday, September 08, 2009 12:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > message > > The file (and folder) are created by the syslog user and definitely > accessible. > > But it works if i don't drop privileges. So I'll investigate this > further and report back. > > Thanks for pushing me that far. > > -- > regard, Henry > > > On Di, 2009-09-08 at 12:30 +0200, Rainer Gerhards wrote: > > Hi, > > > > I got the debug log, it was too big to be sent via the list (but I > got it as > > list admin). I see that you drop privileges to the user "syslog". > This > > probably explains what happens. I think the file is created before > you drop > > privileges, but can then no longer be written when running in the new > > security context. Could you verify that the user "syslog" can access > this > > file? Also, could you temporarily remove the Privilege drop? > > > > Thanks, > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > > Sent: Tuesday, September 08, 2009 9:55 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Could not open dynamic file ... - discarding > > > message > > > > > > can you provide a debug log? > > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > > bounces at lists.adiscon.com] On Behalf Of Henry > > > > Sent: Tuesday, September 08, 2009 9:04 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] Could not open dynamic file ... - > discarding > > > > message > > > > > > > > Hello! > > > > > > > > Tried it with various log locations (e.g. /tmp/my.log), neither > > > worked. > > > > Is this worth ab bug? > > > > > > > > -- > > > > kind regards, Henry > > > > > > > > On Fr, 2009-09-04 at 21:25 +0200, Henry wrote: > > > > > Hi! > > > > > > > > > > This puzzels me: This is my tcprecieve config file for rsyslog > v4 > > > on > > > > > ubuntu: > > > > > > > > > > -----8<----- > > > > > $ModLoad imtcp > > > > > $InputTCPServerRun 514 > > > > > > > > > > # some dynamic templates > > > > > $template DYNlocal1,"/var/log/remote/%HOSTNAME%/local1.log" > > > > > > > > > > # log remote local1 to dynamic diretory > > > > > if $fromhost-ip != '127.0.0.1' and \ > > > > > $syslogfacility-text == 'local1' \ > > > > > then -?DYNlocal1 > > > > > ----->8----- > > > > > > > > > > I created /var/log/remote with sufficient privileges. > > > > > > > > > > Unfortunately this doesn't work. rsyslog crates a folder named > > > after > > > > the > > > > > remote host (myhostname) and creates the file local1.log > (again: > > > > > sufficient permissions: syslog:syslog 640). But it doesn't > write to > > > > that > > > > > file, but logs the error: > > > > > > > > > > -----8<----- > > > > > Could not open dynamic file > '/var/log/remote/myhostname/local1.log' > > > - > > > > > discarding message > > > > > ----->8----- > > > > > > > > > > As you might guess my question is: Why isn't rsyslog able to > open a > > > > file > > > > > it is able to create? Any help or hint is really appreciated. > > > > > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 13:23:10 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 08 Sep 2009 13:23:10 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <1252408990.17679.10.camel@rgf11> Hi all, I am really banging my head on a problem which sounds too easy. I have seen that my systems (and some others as well), seem to not provide more than 1024 bytes on a recvfrom() call. With wireshark, I see that the system itself, at the IP layer, receives more data. I am a bit puzzled, to phrase it lightly. I did not find any information on such a limitation. I have created a strip-down version of a receiver, even built it on top of the Linux man pages samples. Out of desperation, I even set the receivebuf size, which I think has no effect on datagram sockets. Still... I only get 1024 bytes. Code is after my sig. Does anybody have an idea what is going on OR a good place where to ask this question? Thanks, Rainer #include #include #include #include #include #include #include #define BUF_SIZE 2048 int main(int argc, char *argv[]) { struct addrinfo hints; struct addrinfo *result, *rp; int sfd, s; struct sockaddr_storage peer_addr; socklen_t peer_addr_len; ssize_t nread; char buf[BUF_SIZE]; if (argc != 2) { fprintf(stderr, "Usage: %s port\n", argv[0]); exit(EXIT_FAILURE); } memset(&hints, 0, sizeof(struct addrinfo)); hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */ hints.ai_protocol = 0; /* Any protocol */ hints.ai_canonname = NULL; hints.ai_addr = NULL; hints.ai_next = NULL; s = getaddrinfo(NULL, argv[1], &hints, &result); if (s != 0) { fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); exit(EXIT_FAILURE); } /* getaddrinfo() returns a list of address structures. Try each address until we successfully bind(2). If socket(2) (or bind(2)) fails, we (close the socket and) try the next address. */ for (rp = result; rp != NULL; rp = rp->ai_next) { sfd = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (sfd == -1) continue; int result2; int bufSize = 2048; result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, sizeof(bufSize)); printf("result of setsockopt: %d\n", result2); if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) break; /* Success */ close(sfd); } if (rp == NULL) { /* No address succeeded */ fprintf(stderr, "Could not bind\n"); exit(EXIT_FAILURE); } freeaddrinfo(result); /* No longer needed */ /* Read datagrams and echo them back to sender */ for (;;) { peer_addr_len = sizeof(struct sockaddr_storage); memset(buf, 0, BUF_SIZE); nread = recvfrom(sfd, buf, BUF_SIZE, 0, (struct sockaddr *) &peer_addr, &peer_addr_len); if(nread > 1024) printf("NREAD > 1024!"); if (nread == -1) continue; /* Ignore failed request */ char host[NI_MAXHOST], service[NI_MAXSERV]; s = getnameinfo((struct sockaddr *) &peer_addr, peer_addr_len, host, NI_MAXHOST, service, NI_MAXSERV, NI_NUMERICSERV); if (s == 0) printf("Received %ld bytes from %s:%s, msg:'%s'\n", (long) nread, host, service, buf); else fprintf(stderr, "getnameinfo: %s\n", gai_strerror(s)); } } From rgerhards at hq.adiscon.com Tue Sep 8 14:17:05 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 14:17:05 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <1252408990.17679.10.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> oh my... Please disregard this question. I was working on a tcpdump file, and the message length actually *is* 1024 bytes. I was confused by Wireshark's (correct!) indication that the frame is 1066 octets in length. Of course, this is correct, if you take the 42 octets of UDP header into account... I guess the dump file was created with a max of 1K... Sometimes it is sooo easy ... and yet so hard to see ;) Sorry for the interruption, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 08, 2009 1:23 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] Help requested: UDP max message size? > > Hi all, > > I am really banging my head on a problem which sounds too easy. I have > seen that my systems (and some others as well), seem to not provide > more > than 1024 bytes on a recvfrom() call. With wireshark, I see that the > system itself, at the IP layer, receives more data. I am a bit puzzled, > to phrase it lightly. I did not find any information on such a > limitation. > > I have created a strip-down version of a receiver, even built it on top > of the Linux man pages samples. Out of desperation, I even set the > receivebuf size, which I think has no effect on datagram sockets. > Still... I only get 1024 bytes. Code is after my sig. > > Does anybody have an idea what is going on OR a good place where to ask > this question? > > Thanks, > Rainer > > #include > #include > #include > #include > #include > #include > #include > > #define BUF_SIZE 2048 > > int > main(int argc, char *argv[]) > { > struct addrinfo hints; > struct addrinfo *result, *rp; > int sfd, s; > struct sockaddr_storage peer_addr; > socklen_t peer_addr_len; > ssize_t nread; > char buf[BUF_SIZE]; > > if (argc != 2) { > fprintf(stderr, "Usage: %s port\n", argv[0]); > exit(EXIT_FAILURE); > } > > memset(&hints, 0, sizeof(struct addrinfo)); > hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ > hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > hints.ai_flags = AI_PASSIVE; /* For wildcard IP address > */ > hints.ai_protocol = 0; /* Any protocol */ > hints.ai_canonname = NULL; > hints.ai_addr = NULL; > hints.ai_next = NULL; > > s = getaddrinfo(NULL, argv[1], &hints, &result); > if (s != 0) { > fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); > exit(EXIT_FAILURE); > } > > /* getaddrinfo() returns a list of address structures. > Try each address until we successfully bind(2). > If socket(2) (or bind(2)) fails, we (close the socket > and) try the next address. */ > > for (rp = result; rp != NULL; rp = rp->ai_next) { > sfd = socket(rp->ai_family, rp->ai_socktype, > rp->ai_protocol); > if (sfd == -1) > continue; > > > int result2; > int bufSize = 2048; > result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > sizeof(bufSize)); > printf("result of setsockopt: %d\n", result2); > > if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > break; /* Success */ > > close(sfd); > } > > if (rp == NULL) { /* No address succeeded */ > fprintf(stderr, "Could not bind\n"); > exit(EXIT_FAILURE); > } > > freeaddrinfo(result); /* No longer needed */ > > /* Read datagrams and echo them back to sender */ > for (;;) { > peer_addr_len = sizeof(struct sockaddr_storage); > memset(buf, 0, BUF_SIZE); > nread = recvfrom(sfd, buf, BUF_SIZE, 0, > (struct sockaddr *) &peer_addr, &peer_addr_len); > if(nread > 1024) > printf("NREAD > 1024!"); > if (nread == -1) > continue; /* Ignore failed request */ > > char host[NI_MAXHOST], service[NI_MAXSERV]; > > s = getnameinfo((struct sockaddr *) &peer_addr, > peer_addr_len, host, NI_MAXHOST, > service, NI_MAXSERV, NI_NUMERICSERV); > if (s == 0) > printf("Received %ld bytes from %s:%s, msg:'%s'\n", > (long) nread, host, service, buf); > else > fprintf(stderr, "getnameinfo: %s\n", > gai_strerror(s)); > } > } > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mbe_ml at swiss-wireless.com.ar Tue Sep 8 17:24:59 2009 From: mbe_ml at swiss-wireless.com.ar (Beat Meier) Date: Tue, 08 Sep 2009 12:24:59 -0300 Subject: [rsyslog] FailoverSyslogServer: Write buffer immediatly to disk instead to memory option available? Message-ID: <4AA6774B.6070604@swiss-wireless.com.ar> Hello Short: rsyslog V3-V4: Can I write to disk ONLY if the remote rsyslog server is not reachable? Can it be done with the following? $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $ActionQueueType Disk $ActionQueueFileName srvrfwd # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure If the server is reachable there will be nothing written do disk (my problem is using CF card in embedded system see below) or is it written first to disk and than processed by the dispatcher? Long: I use rsyslog on AP which I try now to log remotely to a syslog server because CF card dies if you log often. Now the problem is, that I don't want to loss my syslog messages in the case the syslog server is not available. Now this messages are helt in the memory but if there is a power loss all messages will be lost. We have many power losses here :-( Greetings and thanks Beat From david at lang.hm Tue Sep 8 19:55:21 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 10:55:21 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> References: <1252408990.17679.10.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE32@GRFEXC.intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 8 20:38:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 20:38:12 +0200 Subject: [rsyslog] Help requested: UDP max message size? Message-ID: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) rainer ----- Urspr?ngliche Nachricht ----- Von: "david at lang.hm" An: "rsyslog-users" Gesendet: 08.09.09 19:55 Betreff: Re: [rsyslog] Help requested: UDP max message size? On Tue, 8 Sep 2009, Rainer Gerhards wrote: > oh my... Please disregard this question. I was working on a tcpdump file, and > the message length actually *is* 1024 bytes. I was confused by Wireshark's > (correct!) indication that the frame is 1066 octets in length. Of course, > this is correct, if you take the 42 octets of UDP header into account... > > I guess the dump file was created with a max of 1K... the dump file was set -s 0 (up to 64k packet size), but many/most syslog senders will limit their outbound data to 1k David Lang > Sometimes it is sooo easy ... and yet so hard to see ;) > > Sorry for the interruption, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 08, 2009 1:23 PM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] Help requested: UDP max message size? >> >> Hi all, >> >> I am really banging my head on a problem which sounds too easy. I have >> seen that my systems (and some others as well), seem to not provide >> more >> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >> system itself, at the IP layer, receives more data. I am a bit puzzled, >> to phrase it lightly. I did not find any information on such a >> limitation. >> >> I have created a strip-down version of a receiver, even built it on top >> of the Linux man pages samples. Out of desperation, I even set the >> receivebuf size, which I think has no effect on datagram sockets. >> Still... I only get 1024 bytes. Code is after my sig. >> >> Does anybody have an idea what is going on OR a good place where to ask >> this question? >> >> Thanks, >> Rainer >> >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> #define BUF_SIZE 2048 >> >> int >> main(int argc, char *argv[]) >> { >> struct addrinfo hints; >> struct addrinfo *result, *rp; >> int sfd, s; >> struct sockaddr_storage peer_addr; >> socklen_t peer_addr_len; >> ssize_t nread; >> char buf[BUF_SIZE]; >> >> if (argc != 2) { >> fprintf(stderr, "Usage: %s port\n", argv[0]); >> exit(EXIT_FAILURE); >> } >> >> memset(&hints, 0, sizeof(struct addrinfo)); >> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >> */ >> hints.ai_protocol = 0; /* Any protocol */ >> hints.ai_canonname = NULL; >> hints.ai_addr = NULL; >> hints.ai_next = NULL; >> >> s = getaddrinfo(NULL, argv[1], &hints, &result); >> if (s != 0) { >> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >> exit(EXIT_FAILURE); >> } >> >> /* getaddrinfo() returns a list of address structures. >> Try each address until we successfully bind(2). >> If socket(2) (or bind(2)) fails, we (close the socket >> and) try the next address. */ >> >> for (rp = result; rp != NULL; rp = rp->ai_next) { >> sfd = socket(rp->ai_family, rp->ai_socktype, >> rp->ai_protocol); >> if (sfd == -1) >> continue; >> >> >> int result2; >> int bufSize = 2048; >> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >> sizeof(bufSize)); >> printf("result of setsockopt: %d\n", result2); >> >> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >> break; /* Success */ >> >> close(sfd); >> } >> >> if (rp == NULL) { /* No address succeeded */ >> fprintf(stderr, "Could not bind\n"); >> exit(EXIT_FAILURE); >> } >> >> freeaddrinfo(result); /* No longer needed */ >> >> /* Read datagrams and echo them back to sender */ >> for (;;) { >> peer_addr_len = sizeof(struct sockaddr_storage); >> memset(buf, 0, BUF_SIZE); >> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >> (struct sockaddr *) &peer_addr, &peer_addr_len); >> if(nread > 1024) >> printf("NREAD > 1024!"); >> if (nread == -1) >> continue; /* Ignore failed request */ >> >> char host[NI_MAXHOST], service[NI_MAXSERV]; >> >> s = getnameinfo((struct sockaddr *) &peer_addr, >> peer_addr_len, host, NI_MAXHOST, >> service, NI_MAXSERV, NI_NUMERICSERV); >> if (s == 0) >> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >> (long) nread, host, service, buf); >> else >> fprintf(stderr, "getnameinfo: %s\n", >> gai_strerror(s)); >> } >> } >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com From david at lang.hm Tue Sep 8 20:41:39 2009 From: david at lang.hm (david at lang.hm) Date: Tue, 8 Sep 2009 11:41:39 -0700 (PDT) Subject: [rsyslog] Help requested: UDP max message size? In-Reply-To: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: On Tue, 8 Sep 2009, Rainer Gerhards wrote: > Was there an non-rsyslog relay in the relay chain? If not, it points to the rsyslog forwarding module doing the truncation (what recent v3+ i think should not do...) yes, as far as I know the none of the senders are rsyslog yet. I am working from the central server out. the central server is rsyslog with no problems all but this one relay box are rsyslog things sending to these relay boxes are whatever syslog sender was on the OS/appliance (there may be some acting as relays as well as sending for themselves) David Lang > rainer > > ----- Urspr?ngliche Nachricht ----- > Von: "david at lang.hm" > An: "rsyslog-users" > Gesendet: 08.09.09 19:55 > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > >> oh my... Please disregard this question. I was working on a tcpdump file, and >> the message length actually *is* 1024 bytes. I was confused by Wireshark's >> (correct!) indication that the frame is 1066 octets in length. Of course, >> this is correct, if you take the 42 octets of UDP header into account... >> >> I guess the dump file was created with a max of 1K... > > the dump file was set -s 0 (up to 64k packet size), but many/most syslog > senders will limit their outbound data to 1k > > David Lang > >> Sometimes it is sooo easy ... and yet so hard to see ;) >> >> Sorry for the interruption, >> Rainer >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >>> Sent: Tuesday, September 08, 2009 1:23 PM >>> To: rsyslog at lists.adiscon.com >>> Subject: [rsyslog] Help requested: UDP max message size? >>> >>> Hi all, >>> >>> I am really banging my head on a problem which sounds too easy. I have >>> seen that my systems (and some others as well), seem to not provide >>> more >>> than 1024 bytes on a recvfrom() call. With wireshark, I see that the >>> system itself, at the IP layer, receives more data. I am a bit puzzled, >>> to phrase it lightly. I did not find any information on such a >>> limitation. >>> >>> I have created a strip-down version of a receiver, even built it on top >>> of the Linux man pages samples. Out of desperation, I even set the >>> receivebuf size, which I think has no effect on datagram sockets. >>> Still... I only get 1024 bytes. Code is after my sig. >>> >>> Does anybody have an idea what is going on OR a good place where to ask >>> this question? >>> >>> Thanks, >>> Rainer >>> >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> #define BUF_SIZE 2048 >>> >>> int >>> main(int argc, char *argv[]) >>> { >>> struct addrinfo hints; >>> struct addrinfo *result, *rp; >>> int sfd, s; >>> struct sockaddr_storage peer_addr; >>> socklen_t peer_addr_len; >>> ssize_t nread; >>> char buf[BUF_SIZE]; >>> >>> if (argc != 2) { >>> fprintf(stderr, "Usage: %s port\n", argv[0]); >>> exit(EXIT_FAILURE); >>> } >>> >>> memset(&hints, 0, sizeof(struct addrinfo)); >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */ >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ >>> hints.ai_flags = AI_PASSIVE; /* For wildcard IP address >>> */ >>> hints.ai_protocol = 0; /* Any protocol */ >>> hints.ai_canonname = NULL; >>> hints.ai_addr = NULL; >>> hints.ai_next = NULL; >>> >>> s = getaddrinfo(NULL, argv[1], &hints, &result); >>> if (s != 0) { >>> fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s)); >>> exit(EXIT_FAILURE); >>> } >>> >>> /* getaddrinfo() returns a list of address structures. >>> Try each address until we successfully bind(2). >>> If socket(2) (or bind(2)) fails, we (close the socket >>> and) try the next address. */ >>> >>> for (rp = result; rp != NULL; rp = rp->ai_next) { >>> sfd = socket(rp->ai_family, rp->ai_socktype, >>> rp->ai_protocol); >>> if (sfd == -1) >>> continue; >>> >>> >>> int result2; >>> int bufSize = 2048; >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, >>> sizeof(bufSize)); >>> printf("result of setsockopt: %d\n", result2); >>> >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) >>> break; /* Success */ >>> >>> close(sfd); >>> } >>> >>> if (rp == NULL) { /* No address succeeded */ >>> fprintf(stderr, "Could not bind\n"); >>> exit(EXIT_FAILURE); >>> } >>> >>> freeaddrinfo(result); /* No longer needed */ >>> >>> /* Read datagrams and echo them back to sender */ >>> for (;;) { >>> peer_addr_len = sizeof(struct sockaddr_storage); >>> memset(buf, 0, BUF_SIZE); >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, >>> (struct sockaddr *) &peer_addr, &peer_addr_len); >>> if(nread > 1024) >>> printf("NREAD > 1024!"); >>> if (nread == -1) >>> continue; /* Ignore failed request */ >>> >>> char host[NI_MAXHOST], service[NI_MAXSERV]; >>> >>> s = getnameinfo((struct sockaddr *) &peer_addr, >>> peer_addr_len, host, NI_MAXHOST, >>> service, NI_MAXSERV, NI_NUMERICSERV); >>> if (s == 0) >>> printf("Received %ld bytes from %s:%s, msg:'%s'\n", >>> (long) nread, host, service, buf); >>> else >>> fprintf(stderr, "getnameinfo: %s\n", >>> gai_strerror(s)); >>> } >>> } >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 8 21:24:04 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 8 Sep 2009 21:24:04 +0200 Subject: [rsyslog] Help requested: UDP max message size? References: <002001ca30b3$86e67d46$100013ac@intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE36@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 08, 2009 8:42 PM > To: rsyslog-users > Subject: Re: [rsyslog] Help requested: UDP max message size? > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > Was there an non-rsyslog relay in the relay chain? If not, > it points to the rsyslog forwarding module doing the > truncation (what recent v3+ i think should not do...) > > yes, as far as I know the none of the senders are rsyslog yet. Well, from what I see in the tcpdump logs, the initial sender is rsyslog and the messages originated from imklog. I can point you to the entries in question, but I don't have logs with me now. Rainer > > I am working from the central server out. > > the central server is rsyslog with no problems > > all but this one relay box are rsyslog > > things sending to these relay boxes are whatever syslog > sender was on the > OS/appliance (there may be some acting as relays as well as > sending for > themselves) > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "david at lang.hm" > > An: "rsyslog-users" > > Gesendet: 08.09.09 19:55 > > Betreff: Re: [rsyslog] Help requested: UDP max message size? > > > > On Tue, 8 Sep 2009, Rainer Gerhards wrote: > > > >> oh my... Please disregard this question. I was working on > a tcpdump file, and > >> the message length actually *is* 1024 bytes. I was > confused by Wireshark's > >> (correct!) indication that the frame is 1066 octets in > length. Of course, > >> this is correct, if you take the 42 octets of UDP header > into account... > >> > >> I guess the dump file was created with a max of 1K... > > > > the dump file was set -s 0 (up to 64k packet size), but > many/most syslog > > senders will limit their outbound data to 1k > > > > David Lang > > > >> Sometimes it is sooo easy ... and yet so hard to see ;) > >> > >> Sorry for the interruption, > >> Rainer > >> > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > >>> Sent: Tuesday, September 08, 2009 1:23 PM > >>> To: rsyslog at lists.adiscon.com > >>> Subject: [rsyslog] Help requested: UDP max message size? > >>> > >>> Hi all, > >>> > >>> I am really banging my head on a problem which sounds too > easy. I have > >>> seen that my systems (and some others as well), seem to > not provide > >>> more > >>> than 1024 bytes on a recvfrom() call. With wireshark, I > see that the > >>> system itself, at the IP layer, receives more data. I am > a bit puzzled, > >>> to phrase it lightly. I did not find any information on such a > >>> limitation. > >>> > >>> I have created a strip-down version of a receiver, even > built it on top > >>> of the Linux man pages samples. Out of desperation, I even set the > >>> receivebuf size, which I think has no effect on datagram sockets. > >>> Still... I only get 1024 bytes. Code is after my sig. > >>> > >>> Does anybody have an idea what is going on OR a good > place where to ask > >>> this question? > >>> > >>> Thanks, > >>> Rainer > >>> > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> #include > >>> > >>> #define BUF_SIZE 2048 > >>> > >>> int > >>> main(int argc, char *argv[]) > >>> { > >>> struct addrinfo hints; > >>> struct addrinfo *result, *rp; > >>> int sfd, s; > >>> struct sockaddr_storage peer_addr; > >>> socklen_t peer_addr_len; > >>> ssize_t nread; > >>> char buf[BUF_SIZE]; > >>> > >>> if (argc != 2) { > >>> fprintf(stderr, "Usage: %s port\n", argv[0]); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> memset(&hints, 0, sizeof(struct addrinfo)); > >>> hints.ai_family = AF_UNSPEC; /* Allow IPv4 > or IPv6 */ > >>> hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */ > >>> hints.ai_flags = AI_PASSIVE; /* For > wildcard IP address > >>> */ > >>> hints.ai_protocol = 0; /* Any protocol */ > >>> hints.ai_canonname = NULL; > >>> hints.ai_addr = NULL; > >>> hints.ai_next = NULL; > >>> > >>> s = getaddrinfo(NULL, argv[1], &hints, &result); > >>> if (s != 0) { > >>> fprintf(stderr, "getaddrinfo: %s\n", > gai_strerror(s)); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> /* getaddrinfo() returns a list of address structures. > >>> Try each address until we successfully bind(2). > >>> If socket(2) (or bind(2)) fails, we (close > the socket > >>> and) try the next address. */ > >>> > >>> for (rp = result; rp != NULL; rp = rp->ai_next) { > >>> sfd = socket(rp->ai_family, rp->ai_socktype, > >>> rp->ai_protocol); > >>> if (sfd == -1) > >>> continue; > >>> > >>> > >>> int result2; > >>> int bufSize = 2048; > >>> result2 = setsockopt(sfd, SOL_SOCKET, SO_RCVBUF, &bufSize, > >>> sizeof(bufSize)); > >>> printf("result of setsockopt: %d\n", result2); > >>> > >>> if (bind(sfd, rp->ai_addr, rp->ai_addrlen) == 0) > >>> break; /* Success */ > >>> > >>> close(sfd); > >>> } > >>> > >>> if (rp == NULL) { /* No address > succeeded */ > >>> fprintf(stderr, "Could not bind\n"); > >>> exit(EXIT_FAILURE); > >>> } > >>> > >>> freeaddrinfo(result); /* No longer needed */ > >>> > >>> /* Read datagrams and echo them back to sender */ > >>> for (;;) { > >>> peer_addr_len = sizeof(struct sockaddr_storage); > >>> memset(buf, 0, BUF_SIZE); > >>> nread = recvfrom(sfd, buf, BUF_SIZE, 0, > >>> (struct sockaddr *) &peer_addr, > &peer_addr_len); > >>> if(nread > 1024) > >>> printf("NREAD > 1024!"); > >>> if (nread == -1) > >>> continue; /* Ignore > failed request */ > >>> > >>> char host[NI_MAXHOST], service[NI_MAXSERV]; > >>> > >>> s = getnameinfo((struct sockaddr *) &peer_addr, > >>> peer_addr_len, host, NI_MAXHOST, > >>> service, NI_MAXSERV, > NI_NUMERICSERV); > >>> if (s == 0) > >>> printf("Received %ld bytes from %s:%s, > msg:'%s'\n", > >>> (long) nread, host, service, buf); > >>> else > >>> fprintf(stderr, "getnameinfo: %s\n", > >>> gai_strerror(s)); > >>> } > >>> } > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From corsmith at gmail.com Tue Sep 8 21:46:25 2009 From: corsmith at gmail.com (Corey Smith) Date: Tue, 8 Sep 2009 15:46:25 -0400 Subject: [rsyslog] rsyslog 4.4.1 and solaris In-Reply-To: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> References: <001901ca2cc8$871d16dd$100013ac@intern.adiscon.com> Message-ID: <8061fbee0909081246q6535c9d0s997f814c18e69c83@mail.gmail.com> On Thu, Sep 3, 2009 at 2:58 PM, Rainer Gerhards wrote: > Can you tell me what i need to do to get the recent gcc under solaris? I am quite solaris illiterate, but have a vm where i compile (and upgrade) the solaris branch from time to time. Getting v5 ready, too, would be a big step :) I come from a FreeBSD background so the Solaris package management system leaves much to be desired. The limitations of the default toolset in Solaris are amazing. That is why I started using pkgsrc - a portable package management system originally developed for netbsd. The way I got gcc44 working on Solaris 10/Sparc64: Download, install pkgsrc and bootstrap using the gcc from sunfreeware (3.4) # Check out: http://www.netbsd.org/docs/pkgsrc/platforms.html#solaris Install pkgsrc-wip using a cvs checkout # Check out: http://pkgsrc-wip.sourceforge.net/ Replace the wip/rsyslog port with the one I attached earlier on the thread. Build rsyslog and dependencies using gcc3.4 Install gcc44 from wip/gcc44 and make the changes I described in the first message of the thread cd /usr/pkgsrc/wip/rsyslog && make update # rebuild rsyslog with gcc44 On a side note: I tried building rsyslog-5 from git which compiled but would core every time I started it. BTW: Which virtual machine are you using to emulate sparc64? -Corey Smith From rgerhards at hq.adiscon.com Wed Sep 9 15:00:55 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 9 Sep 2009 15:00:55 +0200 Subject: [rsyslog] epoll-supporting imudp Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE43@GRFEXC.intern.adiscon.com> Hi all, I have finally begun to add some new features concurrently to my analysis of the reported segfault (which seems to be environment-induced and will very seldom show up in practice). I have now created an imudp-epoll branch, based on current master, which provides an imudp module that utilizes epoll() instead of select(). This is my first move towards supporting epoll() where useful. Please note that imudp will not tremendously benefit - on busy servers, select() is very infrequently called, as we read the socket as long there is data. On non-busy servers, there are few calls and I don't expect that epoll vs. select makes any real difference then. Please note that the most benefit from epoll we will gain on tcp based traffic. However, moving to epoll there is far more complicated, because I need to remodel the netstream driver layer. Thus I wanted to gain some experience with easy things first. Probably imuxsock is my next target after I have waited some time for feedback. I would appreciate if some folks could try out the new branch and tell me their experience. I plan to include the new functionality with the next v5-devel release in a couple of days. Rainer From joshsystem at gmail.com Thu Sep 10 07:45:28 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 13:45:28 +0800 Subject: [rsyslog] does rsyslog supports data analytic Message-ID: hi all, i want to receive each syslog msg then input it into my special processing module.after processing the data,output the new data into database.of course,the raw data we must keep it into files. can anyone give me some suggestions? PS: i browse the git source code, but i can't understand why the Experimental-lockfree is not adopted? thanks From david at lang.hm Thu Sep 10 08:26:09 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 9 Sep 2009 23:26:09 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: Message-ID: On Thu, 10 Sep 2009, Josh Zhao wrote: > hi all, i want to receive each syslog msg then input it into my special > processing module.after processing the data,output the new data into > database.of course,the raw data we must keep it into files. can anyone give > me some suggestions? would you not just list two destinations, one to the place you want the raw data archived and one to the processing module? I have a very high volume of logs (>300M/day), so I roll the logs every 5 min with this script #!/bin/sh # PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin umask 022 year=`date +%Y` month=`date +%m` day=`date +%d` fdate=`date +%Y%m%d.%H%M` logroot=/var/log logroll=$logroot/oldlogs cd $logroot mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null mv messages messages.$fdate mv messages.$fdate $logroll/$year/$month/$day/messages.$fdate mv /usr/local/bin/ita/system/itascan1a-p/winlogs /usr/local/bin/ita/system/itascan1a-p/winlogs.0 pkill -HUP syslogd pkill -HUP syslog-ng #pkill win-dump gzip -9 $logroll/$year/$month/$day/messages.$fdate > PS: i browse the git source code, but i can't understand why the > Experimental-lockfree > is > not adopted? I believe that it boils down to complications in being sure that there are no bugs, and the fact that even without that there has been a LOT of room for improvement from the early 3.x timeframe to the current 5.x version. I expect that after the current round of improvements are settled that aspect of things will get reexamined. David Lang From rgerhards at hq.adiscon.com Thu Sep 10 08:32:08 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 08:32:08 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Thursday, September 10, 2009 8:26 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > PS: i browse the git source code, but i can't understand why the > > > Experimental-lockfree > > is > > not adopted? > > I believe that it boils down to complications in being sure > that there are > no bugs, and the fact that even without that there has been a > LOT of room > for improvement from the early 3.x timeframe to the current > 5.x version. > > I expect that after the current round of improvements are > settled that > aspect of things will get reexamined. That branch is mostly there for historical reasons. I keep that branch as a think-tank, but it is is obsoleted. Also, in less polite words than David used, it simply doesn't work. Getting this code with multiple producers and consumers correct is far from being trivial and the literature I browsed indicates that it is probably not possible given the other predicates the code must obey to. Still, optimization is high up on the todo list. Rainer From joshsystem at gmail.com Thu Sep 10 15:25:23 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Thu, 10 Sep 2009 21:25:23 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: Thanks for David and Rainer's reply.I m sorry that I did not explain my question clearly.I m new to rsyslog and want to add a processing module in rsyslog.The rsyslog has input plugins(front-end) and output plugins(back-end).My processing module receives data from input plugins and output the processed data and raw data both into output plugins.So how I add it? 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > Sent: Thursday, September 10, 2009 8:26 AM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > PS: i browse the git source code, but i can't understand why the > > > > > Experimental-lockfree shortlog;h=refs/heads/Experimental-lockfree> > > > is > > > not adopted? > > > > I believe that it boils down to complications in being sure > > that there are > > no bugs, and the fact that even without that there has been a > > LOT of room > > for improvement from the early 3.x timeframe to the current > > 5.x version. > > > > I expect that after the current round of improvements are > > settled that > > aspect of things will get reexamined. > > That branch is mostly there for historical reasons. I keep that branch as a > think-tank, but it is is obsoleted. Also, in less polite words than David > used, it simply doesn't work. Getting this code with multiple producers and > consumers correct is far from being trivial and the literature I browsed > indicates that it is probably not possible given the other predicates the > code must obey to. Still, optimization is high up on the todo list. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Thu Sep 10 17:06:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Thu, 10 Sep 2009 17:06:33 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Thursday, September 10, 2009 3:25 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module > in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins > and > output the processed data and raw data both into output plugins.So how > I add > it? What you are looking for is a library plugin. Unfortunaley, library plugins will work together with the scripting engine. In other words: there currently is no in-proc method available. What you can do, however, is chain two rsyslog instances, pipe data to your plugin and send that data to the other instance. Far from perfect and easy to do, but maybe a workable work-around... Rainer > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that > branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > used, it simply doesn't work. Getting this code with multiple > producers and > > consumers correct is far from being trivial and the literature I > browsed > > indicates that it is probably not possible given the other predicates > the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Thu Sep 10 22:50:30 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:50:30 +0200 Subject: [rsyslog] use snmp as source Message-ID: <4AA96696.9080906@irontec.com> Hi! I have a SNMP capable VoIP gateway, and I want to be able to log in syslog, the messages received by SNMP. Is this possible? I have read that in the other direction, it is possible. http://www.rsyslog.com/doc-omsnmp.html Thanks From mikel at irontec.com Thu Sep 10 22:52:50 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Thu, 10 Sep 2009 22:52:50 +0200 Subject: [rsyslog] use snmp as source In-Reply-To: <4AA96696.9080906@irontec.com> References: <4AA96696.9080906@irontec.com> Message-ID: <4AA96722.4090307@irontec.com> The solution is snmptrapd Thanks!! Mikel Jimenez wrote: > Hi! > > I have a SNMP capable VoIP gateway, and I want to be able to log in > syslog, the messages received by SNMP. > > > Is this possible? > > I have read that in the other direction, it is possible. > http://www.rsyslog.com/doc-omsnmp.html > > Thanks > > From joshsystem at gmail.com Fri Sep 11 02:13:35 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 08:13:35 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: You mean I have to rewrite the processing module in rainerscript.where can i find the detailed documents related to the scripting engine? Thank you! 2009/9/10 Rainer Gerhards > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Thursday, September 10, 2009 3:25 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > > and > > output the processed data and raw data both into output plugins.So how > > I add > > it? > > What you are looking for is a library plugin. Unfortunaley, library plugins > will work together with the scripting engine. In other words: there > currently > is no in-proc method available. > > What you can do, however, is chain two rsyslog instances, pipe data to your > plugin and send that data to the other instance. Far from perfect and easy > to > do, but maybe a workable work-around... > > Rainer > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that > > branch as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > > David > > > used, it simply doesn't work. Getting this code with multiple > > producers and > > > consumers correct is far from being trivial and the literature I > > browsed > > > indicates that it is probably not possible given the other predicates > > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 02:26:46 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 17:26:46 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > You mean I have to rewrite the processing module in rainerscript.where can i > find the detailed documents related to the scripting engine? right now rainerscript is as much an idea as an implementation. it can be used for a few things, but mostly just for filter 'does this log match X' type of things. David Lang > Thank you! > 2009/9/10 Rainer Gerhards > >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>> Sent: Thursday, September 10, 2009 3:25 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>> >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>> question clearly.I m new to rsyslog and want to add a processing module >>> in >>> rsyslog.The rsyslog has input plugins(front-end) and output >>> plugins(back-end).My processing module receives data from input plugins >>> and >>> output the processed data and raw data both into output plugins.So how >>> I add >>> it? >> >> What you are looking for is a library plugin. Unfortunaley, library plugins >> will work together with the scripting engine. In other words: there >> currently >> is no in-proc method available. >> >> What you can do, however, is chain two rsyslog instances, pipe data to your >> plugin and send that data to the other instance. Far from perfect and easy >> to >> do, but maybe a workable work-around... >> >> Rainer >> >>> >>> >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>> david at lang.hm >>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>> >>>>>> PS: i browse the git source code, but i can't understand why the >>>>>> >>>>> Experimental-lockfree>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>> is >>>>>> not adopted? >>>>> >>>>> I believe that it boils down to complications in being sure >>>>> that there are >>>>> no bugs, and the fact that even without that there has been a >>>>> LOT of room >>>>> for improvement from the early 3.x timeframe to the current >>>>> 5.x version. >>>>> >>>>> I expect that after the current round of improvements are >>>>> settled that >>>>> aspect of things will get reexamined. >>>> >>>> That branch is mostly there for historical reasons. I keep that >>> branch as a >>>> think-tank, but it is is obsoleted. Also, in less polite words than >>> David >>>> used, it simply doesn't work. Getting this code with multiple >>> producers and >>>> consumers correct is far from being trivial and the literature I >>> browsed >>>> indicates that it is probably not possible given the other predicates >>> the >>>> code must obey to. Still, optimization is high up on the todo list. >>>> >>>> Rainer >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 03:39:01 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 09:39:01 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Is rsyslog no way to reslove problem, What about syslog-ng? What I think about,rsyslog's multi-thread archititure is better for my mulit-core hardware. The logs data is very high volume too. Could you give me any suggestion on this matter? Thank you! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > You mean I have to rewrite the processing module in rainerscript.where > can i > > find the detailed documents related to the scripting engine? > > right now rainerscript is as much an idea as an implementation. it can be > used for a few things, but mostly just for filter 'does this log match X' > type of things. > > David Lang > > > Thank you! > > 2009/9/10 Rainer Gerhards > > > >>> -----Original Message----- > >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>> Sent: Thursday, September 10, 2009 3:25 PM > >>> To: rsyslog-users > >>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>> > >>> Thanks for David and Rainer's reply.I m sorry that I did not explain my > >>> question clearly.I m new to rsyslog and want to add a processing module > >>> in > >>> rsyslog.The rsyslog has input plugins(front-end) and output > >>> plugins(back-end).My processing module receives data from input plugins > >>> and > >>> output the processed data and raw data both into output plugins.So how > >>> I add > >>> it? > >> > >> What you are looking for is a library plugin. Unfortunaley, library > plugins > >> will work together with the scripting engine. In other words: there > >> currently > >> is no in-proc method available. > >> > >> What you can do, however, is chain two rsyslog instances, pipe data to > your > >> plugin and send that data to the other instance. Far from perfect and > easy > >> to > >> do, but maybe a workable work-around... > >> > >> Rainer > >> > >>> > >>> > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>> david at lang.hm > >>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>> > >>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>> > >>>>> Experimental-lockfree >>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>> is > >>>>>> not adopted? > >>>>> > >>>>> I believe that it boils down to complications in being sure > >>>>> that there are > >>>>> no bugs, and the fact that even without that there has been a > >>>>> LOT of room > >>>>> for improvement from the early 3.x timeframe to the current > >>>>> 5.x version. > >>>>> > >>>>> I expect that after the current round of improvements are > >>>>> settled that > >>>>> aspect of things will get reexamined. > >>>> > >>>> That branch is mostly there for historical reasons. I keep that > >>> branch as a > >>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>> David > >>>> used, it simply doesn't work. Getting this code with multiple > >>> producers and > >>>> consumers correct is far from being trivial and the literature I > >>> browsed > >>>> indicates that it is probably not possible given the other predicates > >>> the > >>>> code must obey to. Still, optimization is high up on the todo list. > >>>> > >>>> Rainer > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Fri Sep 11 06:28:59 2009 From: david at lang.hm (david at lang.hm) Date: Thu, 10 Sep 2009 21:28:59 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > about,rsyslog's multi-thread archititure is better for my mulit-core > hardware. The logs data is very high volume too. Could you give me any > suggestion on this matter? my experiance with syslog-ng was not good, so I'm not the right person to talk about doing this sort of thing with it. but I am not aware of any syslog daemon that lets you insert your own logic in the middle of the processing. rsyslog has the concept, but it has not been implemented (fixing bugs and speeding it up has taken priority) what sort of volume do you consider 'high'? (it's amazing the range that this can span, so I've learned to ask rather than assume ;-) since you are needing to get your final data into a database, I think that you will find that rsyslog will (or will soon) suit your needs far better than alternate approaches. the ability to process multiple messages in one transaction that is being developed will be a huge improvement in terms of database interaction. I would look at what rainer suggested for now. have one copy of rsyslog that receives the messages, does whatever formatting/cleanup is needed on them, then passes the logs to one or more instances of your code to do additional processing, which can then feed the results into another instance of rsyslog to forward them on, insert them into a database, etc. when rainerscript gains the capability to alter the fields (instead of just testing them), then there will be a lot more that can be done inside rsyslog. David Lang > Thank you! > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> You mean I have to rewrite the processing module in rainerscript.where >> can i >>> find the detailed documents related to the scripting engine? >> >> right now rainerscript is as much an idea as an implementation. it can be >> used for a few things, but mostly just for filter 'does this log match X' >> type of things. >> >> David Lang >> >>> Thank you! >>> 2009/9/10 Rainer Gerhards >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>> >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain my >>>>> question clearly.I m new to rsyslog and want to add a processing module >>>>> in >>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>> plugins(back-end).My processing module receives data from input plugins >>>>> and >>>>> output the processed data and raw data both into output plugins.So how >>>>> I add >>>>> it? >>>> >>>> What you are looking for is a library plugin. Unfortunaley, library >> plugins >>>> will work together with the scripting engine. In other words: there >>>> currently >>>> is no in-proc method available. >>>> >>>> What you can do, however, is chain two rsyslog instances, pipe data to >> your >>>> plugin and send that data to the other instance. Far from perfect and >> easy >>>> to >>>> do, but maybe a workable work-around... >>>> >>>> Rainer >>>> >>>>> >>>>> >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>> david at lang.hm >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>> >>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>> >>>>>>> Experimental-lockfree>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>> is >>>>>>>> not adopted? >>>>>>> >>>>>>> I believe that it boils down to complications in being sure >>>>>>> that there are >>>>>>> no bugs, and the fact that even without that there has been a >>>>>>> LOT of room >>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>> 5.x version. >>>>>>> >>>>>>> I expect that after the current round of improvements are >>>>>>> settled that >>>>>>> aspect of things will get reexamined. >>>>>> >>>>>> That branch is mostly there for historical reasons. I keep that >>>>> branch as a >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>> David >>>>>> used, it simply doesn't work. Getting this code with multiple >>>>> producers and >>>>>> consumers correct is far from being trivial and the literature I >>>>> browsed >>>>>> indicates that it is probably not possible given the other predicates >>>>> the >>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>> >>>>>> Rainer >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 08:16:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 08:16:43 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE6D@GRFEXC.intern.adiscon.com> Hi Davd, I think you hit it right on the nail. But I have also thought a bit more about the idea. Actually, I think, one can implement processing modules right now. Especially the configuration is a bit tricky, but it should really work. The rough outline is to use an output module for that. Output modules may do whatever they want as long as they use the provided interfaces. As such, they can also inject messages. So the idea is to define an output module, that accepts the message, does any processing necessary, indicated RS_RET_DISCARD to the rule engine (to prevent the message from being further processed) and inject the "newly generated" message back into the main message queue. That would also be much faster than whatever RainerScript will have to offer, because RainerScript relies on VM execution. I just don't have time to elaborately talk someone through this approach... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Friday, September 11, 2009 6:29 AM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? > What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you > give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the > right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the > concept, but it has > not been implemented (fixing bugs and speeding it up has > taken priority) > > what sort of volume do you consider 'high'? (it's amazing the > range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, > I think that > you will find that rsyslog will (or will soon) suit your > needs far better > than alternate approaches. the ability to process multiple > messages in one > transaction that is being developed will be a huge > improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to > one or more > instances of your code to do additional processing, which can > then feed > the results into another instance of rsyslog to forward them > on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields > (instead of > just testing them), then there will be a lot more that can be > done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in > rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an > implementation. it can be > >> used for a few things, but mostly just for filter 'does > this log match X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I > did not explain my > >>>>> question clearly.I m new to rsyslog and want to add a > processing module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data > from input plugins > >>>>> and > >>>>> output the processed data and raw data both into output > plugins.So how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. > Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other > words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog > instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far > from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't > understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less > polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the > literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the > other predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on > the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 10:17:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 10:17:12 +0200 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> Message-ID: <1252657032.17679.12.camel@rgf11> Now that I got an idea of how this could be implemented with current rsyslog technology, I would be interested in some more details of what you intend to do with the processing module. What exactly will it do with the message? I am asking because I would like to see a real use case. Thinking about the scenario I have proposed in my last mail, I think I see some pitfalls and I am not sure if they will cause any trouble in real projects. So I would appreciate if you could provide more in-depth info. Thanks, Rainer On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > Thanks for David and Rainer's reply.I m sorry that I did not explain my > question clearly.I m new to rsyslog and want to add a processing module in > rsyslog.The rsyslog has input plugins(front-end) and output > plugins(back-end).My processing module receives data from input plugins and > output the processed data and raw data both into output plugins.So how I add > it? > > > 2009/9/10 Rainer Gerhards > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > > > Sent: Thursday, September 10, 2009 8:26 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > Experimental-lockfree > shortlog;h=refs/heads/Experimental-lockfree> > > > > is > > > > not adopted? > > > > > > I believe that it boils down to complications in being sure > > > that there are > > > no bugs, and the fact that even without that there has been a > > > LOT of room > > > for improvement from the early 3.x timeframe to the current > > > 5.x version. > > > > > > I expect that after the current round of improvements are > > > settled that > > > aspect of things will get reexamined. > > > > That branch is mostly there for historical reasons. I keep that branch as a > > think-tank, but it is is obsoleted. Also, in less polite words than David > > used, it simply doesn't work. Getting this code with multiple producers and > > consumers correct is far from being trivial and the literature I browsed > > indicates that it is probably not possible given the other predicates the > > code must obey to. Still, optimization is high up on the todo list. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From thomas.mieslinger at 1und1.de Fri Sep 11 11:47:28 2009 From: thomas.mieslinger at 1und1.de (Thomas Mieslinger) Date: Fri, 11 Sep 2009 11:47:28 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelp transports In-Reply-To: <1236002254.28865.46.camel@rf10up.intern.adiscon.com> References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com> <1236002254.28865.46.camel@rf10up.intern.adiscon.com> Message-ID: <4AAA1CB0.90106@1und1.de> Hi, I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. One machine (logsender) has: $ModLoad omrelp.so $WorkDirectory /var/spool/rsyslog $ActionQueueType LinkedList $ActionQueueFileName srvrfwd $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on user.* :omrelp:loghost:2514 or user.* @@loghost:1514 and the other machine (loghost) has $ModLoad imrelp.so $UDPServerRun 514 $InputTCPServerRun 1514 $InputRELPServerRun 2514 *.* -/some/logfile If I restart rsyslog on loghost without restarting rsyslog on logsender, the logs produced on logsender never appear on loghost. Is this working as designed? Is there a kind of syslog.debug facility where I can monitor the reconnect activity of rsyslog? Thanks in advance Thomas From rgerhards at hq.adiscon.com Fri Sep 11 12:13:12 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 12:13:12 +0200 Subject: [rsyslog] rsyslogd not reconnecting when using tcp or omrelptransports References: <577465F99B41C842AAFBE9ED71E70ABA44FB9E@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBAF@grfint2.intern.adiscon.com> <577465F99B41C842AAFBE9ED71E70ABA44FBFE@grfint2.intern.adiscon.com> <49993125.2060603@ecker-software.de> <4255c2570902161448i731aa22as2b43e34feb049b55@mail.gmail.com> <577465F99B41C842AAFBE9ED71E70ABA44FC12@grfint2.intern.adiscon.com> <4255c2570902171211u26bc267brd13cdfb01728df70@mail.gmail.com> <4255c2570902260753u53ab4c46le86afe27437d2ed9@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA71E99@GRFEXC.intern.adiscon.com><1236002254.28865.46.camel@rf10up.intern.adiscon.com> <4AAA1CB0.90106@1und1.de> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE79@GRFEXC.intern.adiscon.com> I suggest to turn on debug logging on both the client and sender: http://www.rsyslog.com/doc-troubleshoot.html Often, the debug log points to an obvious problem source. If it does not, feel free to mail me the logs to rgerhards at gmail.com BUT let me know you did so - I usually do not monitor this account. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Thomas Mieslinger > Sent: Friday, September 11, 2009 11:47 AM > To: rsyslog-users > Subject: [rsyslog] rsyslogd not reconnecting when using tcp or > omrelptransports > > Hi, > > I've setup rsyslog on CentOS 5.3 (rsyslog-3.21.3-4) on two machines. > One > machine (logsender) has: > > $ModLoad omrelp.so > $WorkDirectory /var/spool/rsyslog > $ActionQueueType LinkedList > $ActionQueueFileName srvrfwd > $ActionResumeRetryCount -1 > $ActionQueueSaveOnShutdown on > > user.* :omrelp:loghost:2514 > or > user.* @@loghost:1514 > > and the other machine (loghost) > has > > $ModLoad imrelp.so > $UDPServerRun 514 > $InputTCPServerRun 1514 > $InputRELPServerRun 2514 > > *.* -/some/logfile > > If I restart rsyslog on loghost without restarting rsyslog on > logsender, > the logs produced on logsender never appear on loghost. Is this working > as designed? > > Is there a kind of syslog.debug facility where I can monitor the > reconnect activity of rsyslog? > > Thanks in advance Thomas > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From tbergfeld at hq.adiscon.com Fri Sep 11 15:21:34 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Fri, 11 Sep 2009 15:21:34 +0200 Subject: [rsyslog] rsyslog 5.1.5 (v5-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE82@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.1.5. This is the first public beta of the v5 branch. As such, it is an important milestone on the way to an even more powerful rsyslogd. As of our usual policies, this means that the first v5-stable will probably available within two to three month, so before the end of the year. Please note that this also means we are shifting our development efforts primarily to v5 for any new functionality (but we keep the option open to add some enhancements to v4-devel). Feedback and bug reports on the new v5-bea branch would be deeply appreciated. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-174.phtml Changelog: http://www.rsyslog.com/Article400.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html . From joshsystem at gmail.com Fri Sep 11 16:17:14 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 22:17:14 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: Our raw data is "high" volume that means to prcocess data about 100M/min. Yes, I want to improve the system performance as soon as possibe. As you said,rsyslog has a concept that inserts my logic module into it ,but it was not been implemented. Could you point out in detail? The rainerscript seems not that strong,otherwise, it is a good idea for user interface. 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Is rsyslog no way to reslove problem, What about syslog-ng? What I think > > about,rsyslog's multi-thread archititure is better for my mulit-core > > hardware. The logs data is very high volume too. Could you give me any > > suggestion on this matter? > > my experiance with syslog-ng was not good, so I'm not the right person to > talk about doing this sort of thing with it. > > but I am not aware of any syslog daemon that lets you insert your own > logic in the middle of the processing. rsyslog has the concept, but it has > not been implemented (fixing bugs and speeding it up has taken priority) > > what sort of volume do you consider 'high'? (it's amazing the range that > this can span, so I've learned to ask rather than assume ;-) > > since you are needing to get your final data into a database, I think that > you will find that rsyslog will (or will soon) suit your needs far better > than alternate approaches. the ability to process multiple messages in one > transaction that is being developed will be a huge improvement in terms of > database interaction. > > I would look at what rainer suggested for now. > > have one copy of rsyslog that receives the messages, does whatever > formatting/cleanup is needed on them, then passes the logs to one or more > instances of your code to do additional processing, which can then feed > the results into another instance of rsyslog to forward them on, insert > them into a database, etc. > > when rainerscript gains the capability to alter the fields (instead of > just testing them), then there will be a lot more that can be done inside > rsyslog. > > David Lang > > > Thank you! > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> You mean I have to rewrite the processing module in rainerscript.where > >> can i > >>> find the detailed documents related to the scripting engine? > >> > >> right now rainerscript is as much an idea as an implementation. it can > be > >> used for a few things, but mostly just for filter 'does this log match > X' > >> type of things. > >> > >> David Lang > >> > >>> Thank you! > >>> 2009/9/10 Rainer Gerhards > >>> > >>>>> -----Original Message----- > >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>> > >>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain > my > >>>>> question clearly.I m new to rsyslog and want to add a processing > module > >>>>> in > >>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>> plugins(back-end).My processing module receives data from input > plugins > >>>>> and > >>>>> output the processed data and raw data both into output plugins.So > how > >>>>> I add > >>>>> it? > >>>> > >>>> What you are looking for is a library plugin. Unfortunaley, library > >> plugins > >>>> will work together with the scripting engine. In other words: there > >>>> currently > >>>> is no in-proc method available. > >>>> > >>>> What you can do, however, is chain two rsyslog instances, pipe data to > >> your > >>>> plugin and send that data to the other instance. Far from perfect and > >> easy > >>>> to > >>>> do, but maybe a workable work-around... > >>>> > >>>> Rainer > >>>> > >>>>> > >>>>> > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>> david at lang.hm > >>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>> > >>>>>>>> PS: i browse the git source code, but i can't understand why the > >>>>>>>> > >>>>>>> Experimental-lockfree >>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>> is > >>>>>>>> not adopted? > >>>>>>> > >>>>>>> I believe that it boils down to complications in being sure > >>>>>>> that there are > >>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>> LOT of room > >>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>> 5.x version. > >>>>>>> > >>>>>>> I expect that after the current round of improvements are > >>>>>>> settled that > >>>>>>> aspect of things will get reexamined. > >>>>>> > >>>>>> That branch is mostly there for historical reasons. I keep that > >>>>> branch as a > >>>>>> think-tank, but it is is obsoleted. Also, in less polite words than > >>>>> David > >>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>> producers and > >>>>>> consumers correct is far from being trivial and the literature I > >>>>> browsed > >>>>>> indicates that it is probably not possible given the other > predicates > >>>>> the > >>>>>> code must obey to. Still, optimization is high up on the todo list. > >>>>>> > >>>>>> Rainer > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Fri Sep 11 17:09:32 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Fri, 11 Sep 2009 23:09:32 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <1252657032.17679.12.camel@rgf11> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> Message-ID: Thanks Rainer. The basic purpose is statistics,which can accumulate some fields of msgs,but I think the customers have more weird requirements. 2009/9/11 Rainer Gerhards > Now that I got an idea of how this could be implemented with current > rsyslog technology, I would be interested in some more details of what > you intend to do with the processing module. What exactly will it do > with the message? I am asking because I would like to see a real use > case. Thinking about the scenario I have proposed in my last mail, I > think I see some pitfalls and I am not sure if they will cause any > trouble in real projects. > > So I would appreciate if you could provide more in-depth info. > > Thanks, > Rainer > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > Thanks for David and Rainer's reply.I m sorry that I did not explain my > > question clearly.I m new to rsyslog and want to add a processing module > in > > rsyslog.The rsyslog has input plugins(front-end) and output > > plugins(back-end).My processing module receives data from input plugins > and > > output the processed data and raw data both into output plugins.So how I > add > > it? > > > > > > 2009/9/10 Rainer Gerhards > > > > > > -----Original Message----- > > > > From: rsyslog-bounces at lists.adiscon.com > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > PS: i browse the git source code, but i can't understand why the > > > > > > > > > Experimental-lockfree > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > is > > > > > not adopted? > > > > > > > > I believe that it boils down to complications in being sure > > > > that there are > > > > no bugs, and the fact that even without that there has been a > > > > LOT of room > > > > for improvement from the early 3.x timeframe to the current > > > > 5.x version. > > > > > > > > I expect that after the current round of improvements are > > > > settled that > > > > aspect of things will get reexamined. > > > > > > That branch is mostly there for historical reasons. I keep that branch > as a > > > think-tank, but it is is obsoleted. Also, in less polite words than > David > > > used, it simply doesn't work. Getting this code with multiple producers > and > > > consumers correct is far from being trivial and the literature I > browsed > > > indicates that it is probably not possible given the other predicates > the > > > code must obey to. Still, optimization is high up on the todo list. > > > > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:18:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:18:27 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><1252657032.17679.12.camel@rgf11> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> OK, that's not very precise, but I have also thought a bit about this. What I have proposed this morning should be possible. But you should be warned, it requires a lot of reading and understanding the source code. A good place to start is the template input and output modules as well as some actual output modules. I think imdiag would be useful (because it is simple) and probably also either omstdout (simple) and omoracle (complex, but utilizes the vector interface which may be the best choice for what you intend to acomplish). As a side-note, if this is paid work you may want to think about purchasing some development help from Adiscon, which may dramatically reduce the time you need to get started (just a thought, omoracle was crafted very well without any such help - thanks again!). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > Sent: Friday, September 11, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] does rsyslog supports data analytic > > Thanks Rainer. The basic purpose is statistics,which can accumulate > some > fields of msgs,but I think the customers have more weird requirements. > > 2009/9/11 Rainer Gerhards > > > Now that I got an idea of how this could be implemented with current > > rsyslog technology, I would be interested in some more details of > what > > you intend to do with the processing module. What exactly will it do > > with the message? I am asking because I would like to see a real use > > case. Thinking about the scenario I have proposed in my last mail, I > > think I see some pitfalls and I am not sure if they will cause any > > trouble in real projects. > > > > So I would appreciate if you could provide more in-depth info. > > > > Thanks, > > Rainer > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > Thanks for David and Rainer's reply.I m sorry that I did not > explain my > > > question clearly.I m new to rsyslog and want to add a processing > module > > in > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > plugins(back-end).My processing module receives data from input > plugins > > and > > > output the processed data and raw data both into output plugins.So > how I > > add > > > it? > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > -----Original Message----- > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > david at lang.hm > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > PS: i browse the git source code, but i can't understand why > the > > > > > > > > > > > Experimental-lockfree > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > is > > > > > > not adopted? > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > that there are > > > > > no bugs, and the fact that even without that there has been a > > > > > LOT of room > > > > > for improvement from the early 3.x timeframe to the current > > > > > 5.x version. > > > > > > > > > > I expect that after the current round of improvements are > > > > > settled that > > > > > aspect of things will get reexamined. > > > > > > > > That branch is mostly there for historical reasons. I keep that > branch > > as a > > > > think-tank, but it is is obsoleted. Also, in less polite words > than > > David > > > > used, it simply doesn't work. Getting this code with multiple > producers > > and > > > > consumers correct is far from being trivial and the literature I > > browsed > > > > indicates that it is probably not possible given the other > predicates > > the > > > > code must obey to. Still, optimization is high up on the todo > list. > > > > > > > > Rainer > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Fri Sep 11 17:24:04 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 08:24:04 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Fri, 11 Sep 2009, Josh Zhao wrote: > Our raw data is "high" volume that means to prcocess data about 100M/min. is this 100M log records, or 100MB of log data (if the latter, approximatly how large are the recors, of how many log records/min) I'm currently processing ~300K messages averaging ~256 bytes/message for a total of ~75MB of logs/min. in my testing v4 will support up to about 6x this volume before it runs into problems (it can receive them faster, up to gig-E wire speed, the limit is in the output, which is ~80K records a sec if doing trivial work like writing them to disk or ~30K records/sec if doing more complex things like forwarding them elsewhere) improvements in V5 include a batch mode that lets an output module process up to N records at a time. I expect this to provide close to a Nx speedup to the output capabilities (with single log per action much of the overhead is in the queue locking, so multiple output workers doesn't help much, with batches not only is much more getting done per pass, but you have the possibility of each output thread taking long enough to get it's work done that it's effective to run more of them without locking contention being the bottleneck) this batch mode will be especially useful for database work as it will let you insert multiple messages in the database in a single transaction. what transport are you using to deliver the logs to your server? > Yes, I want to improve the system performance as soon as possibe. what is the bottleneck you are running into today (what syslog system are you using, etc)? > As you > said,rsyslog has a concept that inserts my logic module into it ,but it was > not been implemented. Could you point out in detail? The rainerscript seems > not that strong,otherwise, it is a good idea for user interface. if you are looking at the source look for imtemplate and omtemplate, basicly he is suggesting creating a custom output module that rsyslog thinks is delivering the messages somewhere, have it be given the log, do it's processing, then acting like an input module and delivering the result to rsyslog as if it was a new message that just arrived. you will need to put some filters in rsyslog to keep your output module from seeing the logs that it creates, and either use discard or filters to keep the other output modules from seeing the raw input that your module is looking for. David Lang > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I think >>> about,rsyslog's multi-thread archititure is better for my mulit-core >>> hardware. The logs data is very high volume too. Could you give me any >>> suggestion on this matter? >> >> my experiance with syslog-ng was not good, so I'm not the right person to >> talk about doing this sort of thing with it. >> >> but I am not aware of any syslog daemon that lets you insert your own >> logic in the middle of the processing. rsyslog has the concept, but it has >> not been implemented (fixing bugs and speeding it up has taken priority) >> >> what sort of volume do you consider 'high'? (it's amazing the range that >> this can span, so I've learned to ask rather than assume ;-) >> >> since you are needing to get your final data into a database, I think that >> you will find that rsyslog will (or will soon) suit your needs far better >> than alternate approaches. the ability to process multiple messages in one >> transaction that is being developed will be a huge improvement in terms of >> database interaction. >> >> I would look at what rainer suggested for now. >> >> have one copy of rsyslog that receives the messages, does whatever >> formatting/cleanup is needed on them, then passes the logs to one or more >> instances of your code to do additional processing, which can then feed >> the results into another instance of rsyslog to forward them on, insert >> them into a database, etc. >> >> when rainerscript gains the capability to alter the fields (instead of >> just testing them), then there will be a lot more that can be done inside >> rsyslog. >> >> David Lang >> >>> Thank you! >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> You mean I have to rewrite the processing module in rainerscript.where >>>> can i >>>>> find the detailed documents related to the scripting engine? >>>> >>>> right now rainerscript is as much an idea as an implementation. it can >> be >>>> used for a few things, but mostly just for filter 'does this log match >> X' >>>> type of things. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> 2009/9/10 Rainer Gerhards >>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>> >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not explain >> my >>>>>>> question clearly.I m new to rsyslog and want to add a processing >> module >>>>>>> in >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>> plugins(back-end).My processing module receives data from input >> plugins >>>>>>> and >>>>>>> output the processed data and raw data both into output plugins.So >> how >>>>>>> I add >>>>>>> it? >>>>>> >>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>> plugins >>>>>> will work together with the scripting engine. In other words: there >>>>>> currently >>>>>> is no in-proc method available. >>>>>> >>>>>> What you can do, however, is chain two rsyslog instances, pipe data to >>>> your >>>>>> plugin and send that data to the other instance. Far from perfect and >>>> easy >>>>>> to >>>>>> do, but maybe a workable work-around... >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>> david at lang.hm >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>> >>>>>>>>>> PS: i browse the git source code, but i can't understand why the >>>>>>>>>> >>>>>>>>> Experimental-lockfree>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>> is >>>>>>>>>> not adopted? >>>>>>>>> >>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>> that there are >>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>> LOT of room >>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>> 5.x version. >>>>>>>>> >>>>>>>>> I expect that after the current round of improvements are >>>>>>>>> settled that >>>>>>>>> aspect of things will get reexamined. >>>>>>>> >>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>> branch as a >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words than >>>>>>> David >>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>> producers and >>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>> browsed >>>>>>>> indicates that it is probably not possible given the other >> predicates >>>>>>> the >>>>>>>> code must obey to. Still, optimization is high up on the todo list. >>>>>>>> >>>>>>>> Rainer >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 11 17:30:25 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 17:30:25 +0200 Subject: [rsyslog] does rsyslog supports data analytic References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE89@GRFEXC.intern.adiscon.com> > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, > do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. I think I did not state one important fact: this is not a dirty trick, but something that the engine was designed for. This mechanism was originally designed and is (somewhat) actually used to report back error conditions. It's used sparsely, because of the circular loop potential. But it is something the engine can handle and is designed to - so no abuse. Actually, I have begun to think if for some feature requests (string replacements before finally writing to an output) this may be good alternative approach. But it seems to involve more overhead than necessary for the job. > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters > to > keep the other output modules from seeing the raw input that your > module > is looking for. Returning RS_RET_DISCARD would solve this, as it stops processing. You just need to make sure that the newly injected messages don't go back into the same rule. With multiple rulesets we now have, this is trivial. But while all this is interesting, I unfortunately have more pressing things to do ;) Rainer > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit- > core > >>> hardware. The logs data is very high volume too. Could you give me > any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right > person to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your > own > >> logic in the middle of the processing. rsyslog has the concept, but > it has > >> not been implemented (fixing bugs and speeding it up has taken > priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range > that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I > think that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages > in one > >> transaction that is being developed will be a huge improvement in > terms of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then > feed > >> the results into another instance of rsyslog to forward them on, > insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead > of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it > can > >> be > >>>> used for a few things, but mostly just for filter 'does this log > match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a > processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output > plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, > library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: > there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe > data to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from > perfect and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand > why the > >>>>>>>>>> > >>>>>>>>> Experimental- > lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep > that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature > I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:39:30 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:39:30 -0600 Subject: [rsyslog] How to increase message size maximum Message-ID: Hello, I replied to an older thread on the forums but wanted to bring this up here. I have an application logging to rsyslog (version 3.22.0) which sends very large messages. We are trying to migrate logging from syslog-ng to rsyslog and I'm running into a problem where messages appear to be truncated or split across lines (I'm seeing both behaviors but I'm not sure if they are both the same problem). In syslog-ng we had to increase the maximum message size with the parameter "log_msg_size(65536);" within the options section. I'm trying to do the equivalent in rsyslog. I saw mention in the forums that this was possibly configurable via a #define but no mention of where I might find this. I realize this is probably outside the typical syslog spec but unfortunately it's a situation I have to deal with for rsyslog to be suitable in our environment. Unfortunately I cannot post the log messages publicly but I can probably provide sanitized samples if an individual was willing to help. Thank you, Aaron From rgerhards at hq.adiscon.com Fri Sep 11 18:48:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 11 Sep 2009 18:48:39 +0200 Subject: [rsyslog] How to increase message size maximum References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, search for MAXLINE inside the code, change that, and recompile. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Friday, September 11, 2009 6:40 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] How to increase message size maximum > > Hello, > I replied to an older thread on the forums but wanted to bring this > up > here. I have an application logging to rsyslog (version 3.22.0) which > sends > very large messages. We are trying to migrate logging from syslog-ng to > rsyslog and I'm running into a problem where messages appear to be > truncated > or split across lines (I'm seeing both behaviors but I'm not sure if > they > are both the same problem). In syslog-ng we had to increase the maximum > message size with the parameter "log_msg_size(65536);" within the > options > section. I'm trying to do the equivalent in rsyslog. I saw mention in > the > forums that this was possibly configurable via a #define but no mention > of > where I might find this. > > I realize this is probably outside the typical syslog spec but > unfortunately > it's a situation I have to deal with for rsyslog to be suitable in our > environment. Unfortunately I cannot post the log messages publicly but > I can > probably provide sanitized samples if an individual was willing to > help. > > Thank you, > Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From anichols at trumped.org Fri Sep 11 18:59:38 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 10:59:38 -0600 Subject: [rsyslog] How to increase message size maximum In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE8B@GRFEXC.intern.adiscon.com> Message-ID: On Fri, Sep 11, 2009 at 10:48 AM, Rainer Gerhards wrote: > $MaxMessageSize 64k - not sure if v3 supports it, check changelog. If not, > search for MAXLINE inside the code, change that, and recompile. > > Excellent - thank you, that is supported as of 3.21.4 per ChangeLog. From anichols at trumped.org Fri Sep 11 23:29:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Fri, 11 Sep 2009 15:29:39 -0600 Subject: [rsyslog] Beginning of log messages being removed Message-ID: Hello, Log messages are now being delivered correctly after raising the messagesize value - now I seem to be having some issue with parsing. I am trying to log only the %msg% portion of the log message however the beginning of that message seems to be removed. Below are the two templates I used to log in both the rawmsg and then the value of %msg% so you can see what is being removed. I cannot post the entire %msg% value, but the two are the same with the exception of the beginning value. I just need to be able to log the message portion without the timestamp which is being delivered from the client. Thinking this may have been fixed with some of the parsing problems I have updated to the latest 4.x stable release - this problem has been observed on 3.22.1 & 4.4.1. I am currently running against 4.4.1. Two templates: $template ServerXML, "%timestamp% || %hostname% || %msg%\n" $template ServerXMLraw, "%rawmsg%\n" Using the first template the message looks like this: Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder of message removed for brevity, but it is intact in the logs) Using the second template the raw message looks like this: <142>Sep 11 21:15:01 localhost References: Message-ID: On Fri, 11 Sep 2009, Aaron Nichols wrote: > Hello, > Log messages are now being delivered correctly after raising the > messagesize value - now I seem to be having some issue with parsing. I am > trying to log only the %msg% portion of the log message however the > beginning of that message seems to be removed. Below are the two templates I > used to log in both the rawmsg and then the value of %msg% so you can see > what is being removed. I cannot post the entire %msg% value, but the two are > the same with the exception of the beginning value. I just need to be able > to log the message portion without the timestamp which is being delivered > from the client. > > Thinking this may have been fixed with some of the parsing problems I have > updated to the latest 4.x stable release - this problem has been observed on > 3.22.1 & 4.4.1. I am currently running against 4.4.1. > > Two templates: > $template ServerXML, "%timestamp% || %hostname% || %msg%\n" > $template ServerXMLraw, "%rawmsg%\n" > > Using the first template the message looks like this: > Sep 11 21:15:01 || localhost || time="1252703701.94" userId=... (remainder > of message removed for brevity, but it is intact in the logs) > > Using the second template the raw message looks like this: > <142>Sep 11 21:15:01 localhost userId= > > I'm trying to understand why the value " from %msg%. because it's being put in %syslogtag% as the program name. David Lang From joshsystem at gmail.com Sat Sep 12 07:36:11 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:36:11 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <1252657032.17679.12.camel@rgf11> <9B6E2A8877C38245BFB15CC491A11DA706FE88@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/11 Rainer Gerhards > OK, that's not very precise, but I have also thought a bit about this. What > I > have proposed this morning should be possible. But you should be warned, it > requires a lot of reading and understanding the source code. A good place > to > start is the template input and output modules as well as some actual > output > modules. I think imdiag would be useful (because it is simple) and probably > also either omstdout (simple) and omoracle (complex, but utilizes the > vector > interface which may be the best choice for what you intend to acomplish). > > As a side-note, if this is paid work you may want to think about purchasing > some development help from Adiscon, which may dramatically reduce the time > you need to get started (just a thought, omoracle was crafted very well > without any such help - thanks again!). > Where can I find the purchased development help in details? Thanks > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Josh Zhao > > Sent: Friday, September 11, 2009 5:10 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > Thanks Rainer. The basic purpose is statistics,which can accumulate > > some > > fields of msgs,but I think the customers have more weird requirements. > > > > 2009/9/11 Rainer Gerhards > > > > > Now that I got an idea of how this could be implemented with current > > > rsyslog technology, I would be interested in some more details of > > what > > > you intend to do with the processing module. What exactly will it do > > > with the message? I am asking because I would like to see a real use > > > case. Thinking about the scenario I have proposed in my last mail, I > > > think I see some pitfalls and I am not sure if they will cause any > > > trouble in real projects. > > > > > > So I would appreciate if you could provide more in-depth info. > > > > > > Thanks, > > > Rainer > > > > > > On Thu, 2009-09-10 at 21:25 +0800, Josh Zhao wrote: > > > > Thanks for David and Rainer's reply.I m sorry that I did not > > explain my > > > > question clearly.I m new to rsyslog and want to add a processing > > module > > > in > > > > rsyslog.The rsyslog has input plugins(front-end) and output > > > > plugins(back-end).My processing module receives data from input > > plugins > > > and > > > > output the processed data and raw data both into output plugins.So > > how I > > > add > > > > it? > > > > > > > > > > > > 2009/9/10 Rainer Gerhards > > > > > > > > > > -----Original Message----- > > > > > > From: rsyslog-bounces at lists.adiscon.com > > > > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > david at lang.hm > > > > > > Sent: Thursday, September 10, 2009 8:26 AM > > > > > > To: rsyslog-users > > > > > > Subject: Re: [rsyslog] does rsyslog supports data analytic > > > > > > > > > > > > PS: i browse the git source code, but i can't understand why > > the > > > > > > > > > > > > > Experimental-lockfree > > > > shortlog;h=refs/heads/Experimental-lockfree> > > > > > > > is > > > > > > > not adopted? > > > > > > > > > > > > I believe that it boils down to complications in being sure > > > > > > that there are > > > > > > no bugs, and the fact that even without that there has been a > > > > > > LOT of room > > > > > > for improvement from the early 3.x timeframe to the current > > > > > > 5.x version. > > > > > > > > > > > > I expect that after the current round of improvements are > > > > > > settled that > > > > > > aspect of things will get reexamined. > > > > > > > > > > That branch is mostly there for historical reasons. I keep that > > branch > > > as a > > > > > think-tank, but it is is obsoleted. Also, in less polite words > > than > > > David > > > > > used, it simply doesn't work. Getting this code with multiple > > producers > > > and > > > > > consumers correct is far from being trivial and the literature I > > > browsed > > > > > indicates that it is probably not possible given the other > > predicates > > > the > > > > > code must obey to. Still, optimization is high up on the todo > > list. > > > > > > > > > > Rainer > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From joshsystem at gmail.com Sat Sep 12 07:39:36 2009 From: joshsystem at gmail.com (Josh Zhao) Date: Sat, 12 Sep 2009 13:39:36 +0800 Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck is output module.it relies on storage disk. I currently test the splunk,but not strong enough.of course,the client delivers the message to server via ethernet. >if you are looking at the source look for imtemplate and omtemplate, >basicly he is suggesting creating a custom output module that rsyslog >thinks is delivering the messages somewhere, have it be given the log, do >it's processing, then acting like an input module and delivering the >result to rsyslog as if it was a new message that just arrived. . This approach as Rainer may be overhead :( ; >That would also be much faster than whatever RainerScript will have to offer, >because RainerScript relies on VM execution. As Rainer said that RainerScrpt is not easy to be extended,but I think it is the perfect approach. I can't find any documents about it:(; It's really hard to start it! 2009/9/11 > On Fri, 11 Sep 2009, Josh Zhao wrote: > > > Our raw data is "high" volume that means to prcocess data about 100M/min. > > is this 100M log records, or 100MB of log data (if the latter, > approximatly how large are the recors, of how many log records/min) > > I'm currently processing ~300K messages averaging ~256 bytes/message for a > total of ~75MB of logs/min. > > in my testing v4 will support up to about 6x this volume before it runs > into problems (it can receive them faster, up to gig-E wire speed, the > limit is in the output, which is ~80K records a sec if doing trivial work > like writing them to disk or ~30K records/sec if doing more complex things > like forwarding them elsewhere) > > improvements in V5 include a batch mode that lets an output module process > up to N records at a time. I expect this to provide close to a Nx speedup > to the output capabilities (with single log per action much of the > overhead is in the queue locking, so multiple output workers doesn't help > much, with batches not only is much more getting done per pass, but you > have the possibility of each output thread taking long enough to get it's > work done that it's effective to run more of them without locking > contention being the bottleneck) > > this batch mode will be especially useful for database work as it will let > you insert multiple messages in the database in a single transaction. > > what transport are you using to deliver the logs to your server? > > > Yes, I want to improve the system performance as soon as possibe. > > what is the bottleneck you are running into today (what syslog system are > you using, etc)? > > > As you > > said,rsyslog has a concept that inserts my logic module into it ,but it > was > > not been implemented. Could you point out in detail? The rainerscript > seems > > not that strong,otherwise, it is a good idea for user interface. > > if you are looking at the source look for imtemplate and omtemplate, > basicly he is suggesting creating a custom output module that rsyslog > thinks is delivering the messages somewhere, have it be given the log, do > it's processing, then acting like an input module and delivering the > result to rsyslog as if it was a new message that just arrived. > > you will need to put some filters in rsyslog to keep your output module > from seeing the logs that it creates, and either use discard or filters to > keep the other output modules from seeing the raw input that your module > is looking for. > > David Lang > > > > > 2009/9/11 > > > >> On Fri, 11 Sep 2009, Josh Zhao wrote: > >> > >>> Is rsyslog no way to reslove problem, What about syslog-ng? What I > think > >>> about,rsyslog's multi-thread archititure is better for my mulit-core > >>> hardware. The logs data is very high volume too. Could you give me any > >>> suggestion on this matter? > >> > >> my experiance with syslog-ng was not good, so I'm not the right person > to > >> talk about doing this sort of thing with it. > >> > >> but I am not aware of any syslog daemon that lets you insert your own > >> logic in the middle of the processing. rsyslog has the concept, but it > has > >> not been implemented (fixing bugs and speeding it up has taken priority) > >> > >> what sort of volume do you consider 'high'? (it's amazing the range that > >> this can span, so I've learned to ask rather than assume ;-) > >> > >> since you are needing to get your final data into a database, I think > that > >> you will find that rsyslog will (or will soon) suit your needs far > better > >> than alternate approaches. the ability to process multiple messages in > one > >> transaction that is being developed will be a huge improvement in terms > of > >> database interaction. > >> > >> I would look at what rainer suggested for now. > >> > >> have one copy of rsyslog that receives the messages, does whatever > >> formatting/cleanup is needed on them, then passes the logs to one or > more > >> instances of your code to do additional processing, which can then feed > >> the results into another instance of rsyslog to forward them on, insert > >> them into a database, etc. > >> > >> when rainerscript gains the capability to alter the fields (instead of > >> just testing them), then there will be a lot more that can be done > inside > >> rsyslog. > >> > >> David Lang > >> > >>> Thank you! > >>> > >>> 2009/9/11 > >>> > >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: > >>>> > >>>>> You mean I have to rewrite the processing module in > rainerscript.where > >>>> can i > >>>>> find the detailed documents related to the scripting engine? > >>>> > >>>> right now rainerscript is as much an idea as an implementation. it can > >> be > >>>> used for a few things, but mostly just for filter 'does this log match > >> X' > >>>> type of things. > >>>> > >>>> David Lang > >>>> > >>>>> Thank you! > >>>>> 2009/9/10 Rainer Gerhards > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao > >>>>>>> Sent: Thursday, September 10, 2009 3:25 PM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>> > >>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not > explain > >> my > >>>>>>> question clearly.I m new to rsyslog and want to add a processing > >> module > >>>>>>> in > >>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output > >>>>>>> plugins(back-end).My processing module receives data from input > >> plugins > >>>>>>> and > >>>>>>> output the processed data and raw data both into output plugins.So > >> how > >>>>>>> I add > >>>>>>> it? > >>>>>> > >>>>>> What you are looking for is a library plugin. Unfortunaley, library > >>>> plugins > >>>>>> will work together with the scripting engine. In other words: there > >>>>>> currently > >>>>>> is no in-proc method available. > >>>>>> > >>>>>> What you can do, however, is chain two rsyslog instances, pipe data > to > >>>> your > >>>>>> plugin and send that data to the other instance. Far from perfect > and > >>>> easy > >>>>>> to > >>>>>> do, but maybe a workable work-around... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> 2009/9/10 Rainer Gerhards > >>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com > >>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > >>>>>>> david at lang.hm > >>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM > >>>>>>>>> To: rsyslog-users > >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic > >>>>>>>> > >>>>>>>>>> PS: i browse the git source code, but i can't understand why > the > >>>>>>>>>> > >>>>>>>>> Experimental-lockfree >>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> > >>>>>>>>>> is > >>>>>>>>>> not adopted? > >>>>>>>>> > >>>>>>>>> I believe that it boils down to complications in being sure > >>>>>>>>> that there are > >>>>>>>>> no bugs, and the fact that even without that there has been a > >>>>>>>>> LOT of room > >>>>>>>>> for improvement from the early 3.x timeframe to the current > >>>>>>>>> 5.x version. > >>>>>>>>> > >>>>>>>>> I expect that after the current round of improvements are > >>>>>>>>> settled that > >>>>>>>>> aspect of things will get reexamined. > >>>>>>>> > >>>>>>>> That branch is mostly there for historical reasons. I keep that > >>>>>>> branch as a > >>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words > than > >>>>>>> David > >>>>>>>> used, it simply doesn't work. Getting this code with multiple > >>>>>>> producers and > >>>>>>>> consumers correct is far from being trivial and the literature I > >>>>>>> browsed > >>>>>>>> indicates that it is probably not possible given the other > >> predicates > >>>>>>> the > >>>>>>>> code must obey to. Still, optimization is high up on the todo > list. > >>>>>>>> > >>>>>>>> Rainer > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Sat Sep 12 08:18:17 2009 From: david at lang.hm (david at lang.hm) Date: Fri, 11 Sep 2009 23:18:17 -0700 (PDT) Subject: [rsyslog] does rsyslog supports data analytic In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA706FE4D@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FE68@GRFEXC.intern.adiscon.com> Message-ID: On Sat, 12 Sep 2009, Josh Zhao wrote: > It's about 100MB of log data,~5k-6k messages/sec. As you said the bottleneck > is output module.it relies on storage disk. I currently test the splunk,but > not strong enough. I am also using splunk. at that volume I would expect to be able to handle everything with a single server. My old system has one box receiving logs from many sources, archiving them, and forwarding them on to several other systems for event correlation, reporting, etc (one of which is my old splunk box). it is comfortably handling about your volume (averaged over a few min, my peak seconds top 10K logs). things are spread across multiple systems less due to current load than in preperation for increasing the load (I am gearing up to handle ~10x my current load) I've done a fair bit of stress testing of the various components and applications. what sort of problems are you having? > of course,the client delivers the message to server via > ethernet. I was meaning are you using TCP syslog, UDP syslog, or something else? David Lang >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. > . > This approach as Rainer may be overhead :( ; > >> That would also be much faster than whatever RainerScript will have to > offer, >> because RainerScript relies on VM execution. > > As Rainer said that RainerScrpt is not easy to be extended,but I think it is > the perfect approach. I can't find any documents about it:(; It's really > hard to start it! > > > > > > 2009/9/11 > >> On Fri, 11 Sep 2009, Josh Zhao wrote: >> >>> Our raw data is "high" volume that means to prcocess data about 100M/min. >> >> is this 100M log records, or 100MB of log data (if the latter, >> approximatly how large are the recors, of how many log records/min) >> >> I'm currently processing ~300K messages averaging ~256 bytes/message for a >> total of ~75MB of logs/min. >> >> in my testing v4 will support up to about 6x this volume before it runs >> into problems (it can receive them faster, up to gig-E wire speed, the >> limit is in the output, which is ~80K records a sec if doing trivial work >> like writing them to disk or ~30K records/sec if doing more complex things >> like forwarding them elsewhere) >> >> improvements in V5 include a batch mode that lets an output module process >> up to N records at a time. I expect this to provide close to a Nx speedup >> to the output capabilities (with single log per action much of the >> overhead is in the queue locking, so multiple output workers doesn't help >> much, with batches not only is much more getting done per pass, but you >> have the possibility of each output thread taking long enough to get it's >> work done that it's effective to run more of them without locking >> contention being the bottleneck) >> >> this batch mode will be especially useful for database work as it will let >> you insert multiple messages in the database in a single transaction. >> >> what transport are you using to deliver the logs to your server? >> >>> Yes, I want to improve the system performance as soon as possibe. >> >> what is the bottleneck you are running into today (what syslog system are >> you using, etc)? >> >>> As you >>> said,rsyslog has a concept that inserts my logic module into it ,but it >> was >>> not been implemented. Could you point out in detail? The rainerscript >> seems >>> not that strong,otherwise, it is a good idea for user interface. >> >> if you are looking at the source look for imtemplate and omtemplate, >> basicly he is suggesting creating a custom output module that rsyslog >> thinks is delivering the messages somewhere, have it be given the log, do >> it's processing, then acting like an input module and delivering the >> result to rsyslog as if it was a new message that just arrived. >> >> you will need to put some filters in rsyslog to keep your output module >> from seeing the logs that it creates, and either use discard or filters to >> keep the other output modules from seeing the raw input that your module >> is looking for. >> >> David Lang >> >>> >>> 2009/9/11 >>> >>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>> >>>>> Is rsyslog no way to reslove problem, What about syslog-ng? What I >> think >>>>> about,rsyslog's multi-thread archititure is better for my mulit-core >>>>> hardware. The logs data is very high volume too. Could you give me any >>>>> suggestion on this matter? >>>> >>>> my experiance with syslog-ng was not good, so I'm not the right person >> to >>>> talk about doing this sort of thing with it. >>>> >>>> but I am not aware of any syslog daemon that lets you insert your own >>>> logic in the middle of the processing. rsyslog has the concept, but it >> has >>>> not been implemented (fixing bugs and speeding it up has taken priority) >>>> >>>> what sort of volume do you consider 'high'? (it's amazing the range that >>>> this can span, so I've learned to ask rather than assume ;-) >>>> >>>> since you are needing to get your final data into a database, I think >> that >>>> you will find that rsyslog will (or will soon) suit your needs far >> better >>>> than alternate approaches. the ability to process multiple messages in >> one >>>> transaction that is being developed will be a huge improvement in terms >> of >>>> database interaction. >>>> >>>> I would look at what rainer suggested for now. >>>> >>>> have one copy of rsyslog that receives the messages, does whatever >>>> formatting/cleanup is needed on them, then passes the logs to one or >> more >>>> instances of your code to do additional processing, which can then feed >>>> the results into another instance of rsyslog to forward them on, insert >>>> them into a database, etc. >>>> >>>> when rainerscript gains the capability to alter the fields (instead of >>>> just testing them), then there will be a lot more that can be done >> inside >>>> rsyslog. >>>> >>>> David Lang >>>> >>>>> Thank you! >>>>> >>>>> 2009/9/11 >>>>> >>>>>> On Fri, 11 Sep 2009, Josh Zhao wrote: >>>>>> >>>>>>> You mean I have to rewrite the processing module in >> rainerscript.where >>>>>> can i >>>>>>> find the detailed documents related to the scripting engine? >>>>>> >>>>>> right now rainerscript is as much an idea as an implementation. it can >>>> be >>>>>> used for a few things, but mostly just for filter 'does this log match >>>> X' >>>>>> type of things. >>>>>> >>>>>> David Lang >>>>>> >>>>>>> Thank you! >>>>>>> 2009/9/10 Rainer Gerhards >>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Josh Zhao >>>>>>>>> Sent: Thursday, September 10, 2009 3:25 PM >>>>>>>>> To: rsyslog-users >>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>> >>>>>>>>> Thanks for David and Rainer's reply.I m sorry that I did not >> explain >>>> my >>>>>>>>> question clearly.I m new to rsyslog and want to add a processing >>>> module >>>>>>>>> in >>>>>>>>> rsyslog.The rsyslog has input plugins(front-end) and output >>>>>>>>> plugins(back-end).My processing module receives data from input >>>> plugins >>>>>>>>> and >>>>>>>>> output the processed data and raw data both into output plugins.So >>>> how >>>>>>>>> I add >>>>>>>>> it? >>>>>>>> >>>>>>>> What you are looking for is a library plugin. Unfortunaley, library >>>>>> plugins >>>>>>>> will work together with the scripting engine. In other words: there >>>>>>>> currently >>>>>>>> is no in-proc method available. >>>>>>>> >>>>>>>> What you can do, however, is chain two rsyslog instances, pipe data >> to >>>>>> your >>>>>>>> plugin and send that data to the other instance. Far from perfect >> and >>>>>> easy >>>>>>>> to >>>>>>>> do, but maybe a workable work-around... >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2009/9/10 Rainer Gerhards >>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: rsyslog-bounces at lists.adiscon.com >>>>>>>>>>> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of >>>>>>>>> david at lang.hm >>>>>>>>>>> Sent: Thursday, September 10, 2009 8:26 AM >>>>>>>>>>> To: rsyslog-users >>>>>>>>>>> Subject: Re: [rsyslog] does rsyslog supports data analytic >>>>>>>>>> >>>>>>>>>>>> PS: i browse the git source code, but i can't understand why >> the >>>>>>>>>>>> >>>>>>>>>>> Experimental-lockfree>>>>>>>>> shortlog;h=refs/heads/Experimental-lockfree> >>>>>>>>>>>> is >>>>>>>>>>>> not adopted? >>>>>>>>>>> >>>>>>>>>>> I believe that it boils down to complications in being sure >>>>>>>>>>> that there are >>>>>>>>>>> no bugs, and the fact that even without that there has been a >>>>>>>>>>> LOT of room >>>>>>>>>>> for improvement from the early 3.x timeframe to the current >>>>>>>>>>> 5.x version. >>>>>>>>>>> >>>>>>>>>>> I expect that after the current round of improvements are >>>>>>>>>>> settled that >>>>>>>>>>> aspect of things will get reexamined. >>>>>>>>>> >>>>>>>>>> That branch is mostly there for historical reasons. I keep that >>>>>>>>> branch as a >>>>>>>>>> think-tank, but it is is obsoleted. Also, in less polite words >> than >>>>>>>>> David >>>>>>>>>> used, it simply doesn't work. Getting this code with multiple >>>>>>>>> producers and >>>>>>>>>> consumers correct is far from being trivial and the literature I >>>>>>>>> browsed >>>>>>>>>> indicates that it is probably not possible given the other >>>> predicates >>>>>>>>> the >>>>>>>>>> code must obey to. Still, optimization is high up on the todo >> list. >>>>>>>>>> >>>>>>>>>> Rainer >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Mon Sep 14 10:14:39 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Mon, 14 Sep 2009 10:14:39 +0200 Subject: [rsyslog] DNS Cache Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FE9D@GRFEXC.intern.adiscon.com> Hi all, I just wanted to let you know that in parallel to my bughunt (which involves a lot of waiting for lab results), I will now begin to implement a real DNS cache. That will be a v5-exclusive feature (too much trouble to do it in v4 and v5, code base has changed too much). Together with the case, I will probably also implement a feature to override reverse DNS resolution via a file - simply by loading non-expiring entries from that file. I just thought I share this plan, if someone has feature requests in that regard. It would be good to know them, as now is a good time to integrate them into the design. Tech side-note: I'll be using AVL trees for the cache, as I don't outrule many entries and this hopefully speeds up cache searches for larger caches. Once the avl tree class is there, I can probably speed up a few other things that currently rely on simple linked lists). I will probably do two or even three releases until the full functionality is there. I also plan to do a "pre-cache" v5-devel today, so that all new features (including imudp epoll) are rolled out and can be tested. Rainer From tbergfeld at hq.adiscon.com Mon Sep 14 17:05:35 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 14 Sep 2009 17:05:35 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Hi all, We have just released rsyslog 5.3.0. This release starts a new v5-development branch. This release offers a lot of new features like the use of epoll, when possible, in imudp, which provides greater performance and is a pilot to more such enhancements. Further more there are also some bug fixes. See Changelog for more details. This is a recommended update for all users of the devel branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml Changelog: http://www.rsyslog.com/Article402.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld From david at lang.hm Tue Sep 15 02:51:19 2009 From: david at lang.hm (david at lang.hm) Date: Mon, 14 Sep 2009 17:51:19 -0700 (PDT) Subject: [rsyslog] rsyslog 5.3.0 (devel) released In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: I do not see this tagged in git. David Lang On Mon, 14 Sep 2009, Tom Bergfeld wrote: > Date: Mon, 14 Sep 2009 17:05:35 +0200 > From: Tom Bergfeld > Reply-To: rsyslog-users > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog 5.3.0 (devel) released > > Hi all, > > We have just released rsyslog 5.3.0. This release starts a new v5-development > branch. > This release offers a lot of new features like the use of epoll, when > possible, in imudp, which provides greater performance and is a pilot to more > such enhancements. Further more there are also some bug fixes. See Changelog > for more details. This is a recommended update for all users of the devel > branch. > > Download: > > http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-175.phtml > > Changelog: > > http://www.rsyslog.com/Article402.phtml > > As always, feedback is appreciated. > > Best regards, > Tom Bergfeld > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mikel at irontec.com Tue Sep 15 10:52:37 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 10:52:37 +0200 Subject: [rsyslog] server frozen when remote logging Message-ID: <4AAF55D5.20807@irontec.com> Hi!! I have 80 servers logging to a centralized rsyslog, and I have experimented the kaos!! Accidentaly the central server shutdowns, and one hour later, all the 80 servers frezze. Can not access ssh, ping... I use Debian in central server, and suse in nodes. Thanks! -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 10:56:28 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:56:28 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> This sounds like you are overdoing "reliable delivery". But I need configs and version information to tell you what may be the case. If it is an older v3 version, this may also be a bug in rsyslog. HTH Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 10:53 AM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] server frozen when remote logging > > Hi!! > > I have 80 servers logging to a centralized rsyslog, and I have > experimented the kaos!! > > Accidentaly the central server shutdowns, and one hour later, all the > 80 > servers frezze. > > Can not access ssh, ping... > > I use Debian in central server, and suse in nodes. > > Thanks! > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 10:58:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 10:58:44 +0200 Subject: [rsyslog] rsyslog 5.3.0 (devel) released References: <9B6E2A8877C38245BFB15CC491A11DA706FEBA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FECC@GRFEXC.intern.adiscon.com> Thanks - I had forgotten to push the tags (but thankfully this time not the tagging itself ;)) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Tuesday, September 15, 2009 2:51 AM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog 5.3.0 (devel) released > > I do not see this tagged in git. From mikel at irontec.com Tue Sep 15 11:56:26 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:56:26 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF64CA.8020002@irontec.com> Ok Rainer In the clients: OS= opensuse 10.0 rsyslog version: 3.19.7 In the server OS=Debian 4.0 rsyslog version: 3.18.2 I attach the configuration files of the clients and the servers. The remote server is 192.1.4.215. Thanks Rainer Gerhards wrote: > This sounds like you are overdoing "reliable delivery". But I need configs > and version information to tell you what may be the case. If it is an older > v3 version, this may also be a bug in rsyslog. > > HTH > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 10:53 AM >> To: rsyslog at lists.adiscon.com >> Subject: [rsyslog] server frozen when remote logging >> >> Hi!! >> >> I have 80 servers logging to a centralized rsyslog, and I have >> experimented the kaos!! >> >> Accidentaly the central server shutdowns, and one hour later, all the >> 80 >> servers frezze. >> >> Can not access ssh, ping... >> >> I use Debian in central server, and suse in nodes. >> >> Thanks! >> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-client.conf URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rsyslog-server.conf URL: From mikel at irontec.com Tue Sep 15 11:58:52 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 11:58:52 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <4A9E6A72.8080202@irontec.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> Message-ID: <4AAF655C.6050601@irontec.com> Hi!! We are very interested, how much do you estimate? Thanks Mikel Jimenez wrote: > Ok, I will comunicate you if we decide. > > Is the development of phplogcon frezzed? the last version is of > January 27 ... > > Thanks > > Mikel Jimenez wrote: >> hi >> >> Some news about this? >> >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >> Maybe with a bounty? >> >> thanks >> > > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:01:02 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:01:02 +0200 Subject: [rsyslog] milliseconds timestamp References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:59 AM > To: rsyslog-users > Subject: Re: [rsyslog] milliseconds timestamp > > Hi!! > > We are very interested, how much do you estimate? Thanks. I've just asked the right people, please expect a reply either today or (depending on discussion) tomorrow, must probably via private mail. Rainer > Thanks > > Mikel Jimenez wrote: > > Ok, I will comunicate you if we decide. > > > > Is the development of phplogcon frezzed? the last version is of > > January 27 ... > > > > Thanks > > > > Mikel Jimenez wrote: > >> hi > >> > >> Some news about this? > >> > >> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html > >> Maybe with a bounty? > >> > >> thanks > >> > > > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:05:40 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:05:40 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Ok, there are errors in the config files. I've stopped looking at them when I saw EST=... ... @@$EST This does not work in rsyslog (yet). Please make sure that your configs are OK. With the versions you have, you either need to start rsyslogd interactively in debug mode OR simply look at the syslogd logs (those with syslog facility). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 11:56 AM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Ok Rainer > > In the clients: > > OS= opensuse 10.0 > rsyslog version: 3.19.7 > > In the server > OS=Debian 4.0 > rsyslog version: 3.18.2 > > I attach the configuration files of the clients and the servers. > > The remote server is 192.1.4.215. > > Thanks > > Rainer Gerhards wrote: > > This sounds like you are overdoing "reliable delivery". But I need > configs > > and version information to tell you what may be the case. If it is an > older > > v3 version, this may also be a bug in rsyslog. > > > > HTH > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 10:53 AM > >> To: rsyslog at lists.adiscon.com > >> Subject: [rsyslog] server frozen when remote logging > >> > >> Hi!! > >> > >> I have 80 servers logging to a centralized rsyslog, and I have > >> experimented the kaos!! > >> > >> Accidentaly the central server shutdowns, and one hour later, all > the > >> 80 > >> servers frezze. > >> > >> Can not access ssh, ping... > >> > >> I use Debian in central server, and suse in nodes. > >> > >> Thanks! > >> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > From mikel at irontec.com Tue Sep 15 12:10:16 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:10:16 +0200 Subject: [rsyslog] milliseconds timestamp In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> References: <4A9D1193.4090806@irontec.com> <4A9E6A72.8080202@irontec.com> <4AAF655C.6050601@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED2@GRFEXC.intern.adiscon.com> Message-ID: <4AAF6808.2080402@irontec.com> Yeah!!! :) Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:59 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] milliseconds timestamp >> >> Hi!! >> >> We are very interested, how much do you estimate? >> > > Thanks. I've just asked the right people, please expect a reply either today > or (depending on discussion) tomorrow, must probably via private mail. > > Rainer > > > > >> Thanks >> >> Mikel Jimenez wrote: >> >>> Ok, I will comunicate you if we decide. >>> >>> Is the development of phplogcon frezzed? the last version is of >>> January 27 ... >>> >>> Thanks >>> >>> Mikel Jimenez wrote: >>> >>>> hi >>>> >>>> Some news about this? >>>> >>>> http://lists.adiscon.net/pipermail/rsyslog/2008-November/001391.html >>>> Maybe with a bounty? >>>> >>>> thanks >>>> >>>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From mikel at irontec.com Tue Sep 15 12:14:20 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:14:20 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> Message-ID: <4AAF68FC.1040205@irontec.com> But rsyslog starts... If I use UDP instead TCP? Rainer Gerhards wrote: > Ok, there are errors in the config files. I've stopped looking at them when I > saw > > EST=... > > ... @@$EST > > This does not work in rsyslog (yet). Please make sure that your configs are > OK. With the versions you have, you either need to start rsyslogd > interactively in debug mode OR simply look at the syslogd logs (those with > syslog facility). > > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 11:56 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> Ok Rainer >> >> In the clients: >> >> OS= opensuse 10.0 >> rsyslog version: 3.19.7 >> >> In the server >> OS=Debian 4.0 >> rsyslog version: 3.18.2 >> >> I attach the configuration files of the clients and the servers. >> >> The remote server is 192.1.4.215. >> >> Thanks >> >> Rainer Gerhards wrote: >> >>> This sounds like you are overdoing "reliable delivery". But I need >>> >> configs >> >>> and version information to tell you what may be the case. If it is an >>> >> older >> >>> v3 version, this may also be a bug in rsyslog. >>> >>> HTH >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>> To: rsyslog at lists.adiscon.com >>>> Subject: [rsyslog] server frozen when remote logging >>>> >>>> Hi!! >>>> >>>> I have 80 servers logging to a centralized rsyslog, and I have >>>> experimented the kaos!! >>>> >>>> Accidentaly the central server shutdowns, and one hour later, all >>>> >> the >> >>>> 80 >>>> servers frezze. >>>> >>>> Can not access ssh, ping... >>>> >>>> I use Debian in central server, and suse in nodes. >>>> >>>> Thanks! >>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:18:30 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:18:30 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > But rsyslog starts... > > If I use UDP instead TCP? sure, because the messages are thrown away, at least I think. The point is: it doesn't make sense to hunt for a problem as long as we know that the config is incorrect. Better get the config clean first, then see if the problem even persists and then look at it. Bluntly and not meant to be embarrassing: I've set aside some time to do this kind of support, but if you need more "full service" help, it would probably be a good idea to purchase one of the support packages. They exists so that we can look at issues in depth. This is often an excellent values, as it may safe you hours and hours of work. And, really, I can't develop all this and provide this kind of full-service support ;) Rainer > > Rainer Gerhards wrote: > > Ok, there are errors in the config files. I've stopped looking at > them when I > > saw > > > > EST=... > > > > ... @@$EST > > > > This does not work in rsyslog (yet). Please make sure that your > configs are > > OK. With the versions you have, you either need to start rsyslogd > > interactively in debug mode OR simply look at the syslogd logs (those > with > > syslog facility). > > > > Rainer > > > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 11:56 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> Ok Rainer > >> > >> In the clients: > >> > >> OS= opensuse 10.0 > >> rsyslog version: 3.19.7 > >> > >> In the server > >> OS=Debian 4.0 > >> rsyslog version: 3.18.2 > >> > >> I attach the configuration files of the clients and the servers. > >> > >> The remote server is 192.1.4.215. > >> > >> Thanks > >> > >> Rainer Gerhards wrote: > >> > >>> This sounds like you are overdoing "reliable delivery". But I need > >>> > >> configs > >> > >>> and version information to tell you what may be the case. If it is > an > >>> > >> older > >> > >>> v3 version, this may also be a bug in rsyslog. > >>> > >>> HTH > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>> To: rsyslog at lists.adiscon.com > >>>> Subject: [rsyslog] server frozen when remote logging > >>>> > >>>> Hi!! > >>>> > >>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>> experimented the kaos!! > >>>> > >>>> Accidentaly the central server shutdowns, and one hour later, all > >>>> > >> the > >> > >>>> 80 > >>>> servers frezze. > >>>> > >>>> Can not access ssh, ping... > >>>> > >>>> I use Debian in central server, and suse in nodes. > >>>> > >>>> Thanks! > >>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 12:47:31 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 12:47:31 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> Message-ID: <4AAF70C3.1000800@irontec.com> Hi I will delete this line of the config. I will make probes. About the comercial support, I think that this issue is "basic" for the proper working of a production and seriour enviroment of rsyslog. In the future, if we want an especialezed support we call you for support, sure!! :) So any solution for this? UDP? Rainer Gerhards wrote: >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >> Sent: Tuesday, September 15, 2009 12:14 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> But rsyslog starts... >> >> If I use UDP instead TCP? >> > > sure, because the messages are thrown away, at least I think. The point is: > it doesn't make sense to hunt for a problem as long as we know that the > config is incorrect. Better get the config clean first, then see if the > problem even persists and then look at it. > > Bluntly and not meant to be embarrassing: I've set aside some time to do this > kind of support, but if you need more "full service" help, it would probably > be a good idea to purchase one of the support packages. They exists so that > we can look at issues in depth. This is often an excellent values, as it may > safe you hours and hours of work. And, really, I can't develop all this and > provide this kind of full-service support ;) > > Rainer > > >> Rainer Gerhards wrote: >> >>> Ok, there are errors in the config files. I've stopped looking at >>> >> them when I >> >>> saw >>> >>> EST=... >>> >>> ... @@$EST >>> >>> This does not work in rsyslog (yet). Please make sure that your >>> >> configs are >> >>> OK. With the versions you have, you either need to start rsyslogd >>> interactively in debug mode OR simply look at the syslogd logs (those >>> >> with >> >>> syslog facility). >>> >>> Rainer >>> >>> >>> >>>> -----Original Message----- >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] server frozen when remote logging >>>> >>>> Ok Rainer >>>> >>>> In the clients: >>>> >>>> OS= opensuse 10.0 >>>> rsyslog version: 3.19.7 >>>> >>>> In the server >>>> OS=Debian 4.0 >>>> rsyslog version: 3.18.2 >>>> >>>> I attach the configuration files of the clients and the servers. >>>> >>>> The remote server is 192.1.4.215. >>>> >>>> Thanks >>>> >>>> Rainer Gerhards wrote: >>>> >>>> >>>>> This sounds like you are overdoing "reliable delivery". But I need >>>>> >>>>> >>>> configs >>>> >>>> >>>>> and version information to tell you what may be the case. If it is >>>>> >> an >> >>>> older >>>> >>>> >>>>> v3 version, this may also be a bug in rsyslog. >>>>> >>>>> HTH >>>>> Rainer >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>> To: rsyslog at lists.adiscon.com >>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>> >>>>>> Hi!! >>>>>> >>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>> experimented the kaos!! >>>>>> >>>>>> Accidentaly the central server shutdowns, and one hour later, all >>>>>> >>>>>> >>>> the >>>> >>>> >>>>>> 80 >>>>>> servers frezze. >>>>>> >>>>>> Can not access ssh, ping... >>>>>> >>>>>> I use Debian in central server, and suse in nodes. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> Mikel Jimenez Fernandez >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>> >>>>>> >>>> http://www.irontec.com >>>> >>>> >>>>>> +34 94.404.81.82 >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>>> >>>> -- >>>> Mikel Jimenez Fernandez >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>> >> http://www.irontec.com >> >>>> +34 94.404.81.82 >>>> >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >>> >> -- >> Mikel Jimenez Fernandez >> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >> +34 94.404.81.82 >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 12:52:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:52:27 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com> <4AAF70C3.1000800@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > Sent: Tuesday, September 15, 2009 12:48 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Hi > > I will delete this line of the config. > I will make probes. > About the comercial support, I think that this issue is "basic" for > the > proper working of a production and seriour enviroment of rsyslog. Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in bug reports is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. > > In the future, if we want an especialezed support we call you for > support, sure!! :) > > > So any solution for this? > UDP? Anyhow, does that mean your config is now error-free and the problem still persists? Rainer > > Rainer Gerhards wrote: > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >> Sent: Tuesday, September 15, 2009 12:14 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] server frozen when remote logging > >> > >> But rsyslog starts... > >> > >> If I use UDP instead TCP? > >> > > > > sure, because the messages are thrown away, at least I think. The > point is: > > it doesn't make sense to hunt for a problem as long as we know that > the > > config is incorrect. Better get the config clean first, then see if > the > > problem even persists and then look at it. > > > > Bluntly and not meant to be embarrassing: I've set aside some time to > do this > > kind of support, but if you need more "full service" help, it would > probably > > be a good idea to purchase one of the support packages. They exists > so that > > we can look at issues in depth. This is often an excellent values, as > it may > > safe you hours and hours of work. And, really, I can't develop all > this and > > provide this kind of full-service support ;) > > > > Rainer > > > > > >> Rainer Gerhards wrote: > >> > >>> Ok, there are errors in the config files. I've stopped looking at > >>> > >> them when I > >> > >>> saw > >>> > >>> EST=... > >>> > >>> ... @@$EST > >>> > >>> This does not work in rsyslog (yet). Please make sure that your > >>> > >> configs are > >> > >>> OK. With the versions you have, you either need to start rsyslogd > >>> interactively in debug mode OR simply look at the syslogd logs > (those > >>> > >> with > >> > >>> syslog facility). > >>> > >>> Rainer > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] server frozen when remote logging > >>>> > >>>> Ok Rainer > >>>> > >>>> In the clients: > >>>> > >>>> OS= opensuse 10.0 > >>>> rsyslog version: 3.19.7 > >>>> > >>>> In the server > >>>> OS=Debian 4.0 > >>>> rsyslog version: 3.18.2 > >>>> > >>>> I attach the configuration files of the clients and the servers. > >>>> > >>>> The remote server is 192.1.4.215. > >>>> > >>>> Thanks > >>>> > >>>> Rainer Gerhards wrote: > >>>> > >>>> > >>>>> This sounds like you are overdoing "reliable delivery". But I > need > >>>>> > >>>>> > >>>> configs > >>>> > >>>> > >>>>> and version information to tell you what may be the case. If it > is > >>>>> > >> an > >> > >>>> older > >>>> > >>>> > >>>>> v3 version, this may also be a bug in rsyslog. > >>>>> > >>>>> HTH > >>>>> Rainer > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > >>>>>> To: rsyslog at lists.adiscon.com > >>>>>> Subject: [rsyslog] server frozen when remote logging > >>>>>> > >>>>>> Hi!! > >>>>>> > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > >>>>>> experimented the kaos!! > >>>>>> > >>>>>> Accidentaly the central server shutdowns, and one hour later, > all > >>>>>> > >>>>>> > >>>> the > >>>> > >>>> > >>>>>> 80 > >>>>>> servers frezze. > >>>>>> > >>>>>> Can not access ssh, ping... > >>>>>> > >>>>>> I use Debian in central server, and suse in nodes. > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> -- > >>>>>> Mikel Jimenez Fernandez > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>>>> > >>>>>> > >>>> http://www.irontec.com > >>>> > >>>> > >>>>>> +34 94.404.81.82 > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>>> > >>>>>> > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>>> > >>>>> > >>>> -- > >>>> Mikel Jimenez Fernandez > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > >>>> > >> http://www.irontec.com > >> > >>>> +34 94.404.81.82 > >>>> > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >>> > >> -- > >> Mikel Jimenez Fernandez > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > >> +34 94.404.81.82 > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > > > -- > Mikel Jimenez Fernandez > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > +34 94.404.81.82 > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:54:24 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:54:24 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Used the wrong words ;) Of course, this should read: Sure, but let me phrase it that way: My interest is finding bugs, support questions often lead to that. If there is no bug involved, my personal interest in support is *extremely limited*. Still, there is the rest of the community, and they often provide advice. So Adiscon created the commercial support for corporations that want to have a solution and save time while doing so. And: why is my interest limited? Support to get someone else going contributes almost nothing back to the project... Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:52 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > Sent: Tuesday, September 15, 2009 12:48 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > Hi > > > > I will delete this line of the config. > > I will make probes. > > About the comercial support, I think that this issue is "basic" for > > the > > proper working of a production and seriour enviroment of rsyslog. > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in bug reports is *extremely limited*. Still, there is the > rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > > > > In the future, if we want an especialezed support we call you for > > support, sure!! :) > > > > > > So any solution for this? > > UDP? > > Anyhow, does that mean your config is now error-free and the problem > still > persists? > > Rainer > > > > Rainer Gerhards wrote: > > >> -----Original Message----- > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] server frozen when remote logging > > >> > > >> But rsyslog starts... > > >> > > >> If I use UDP instead TCP? > > >> > > > > > > sure, because the messages are thrown away, at least I think. The > > point is: > > > it doesn't make sense to hunt for a problem as long as we know that > > the > > > config is incorrect. Better get the config clean first, then see if > > the > > > problem even persists and then look at it. > > > > > > Bluntly and not meant to be embarrassing: I've set aside some time > to > > do this > > > kind of support, but if you need more "full service" help, it would > > probably > > > be a good idea to purchase one of the support packages. They exists > > so that > > > we can look at issues in depth. This is often an excellent values, > as > > it may > > > safe you hours and hours of work. And, really, I can't develop all > > this and > > > provide this kind of full-service support ;) > > > > > > Rainer > > > > > > > > >> Rainer Gerhards wrote: > > >> > > >>> Ok, there are errors in the config files. I've stopped looking at > > >>> > > >> them when I > > >> > > >>> saw > > >>> > > >>> EST=... > > >>> > > >>> ... @@$EST > > >>> > > >>> This does not work in rsyslog (yet). Please make sure that your > > >>> > > >> configs are > > >> > > >>> OK. With the versions you have, you either need to start rsyslogd > > >>> interactively in debug mode OR simply look at the syslogd logs > > (those > > >>> > > >> with > > >> > > >>> syslog facility). > > >>> > > >>> Rainer > > >>> > > >>> > > >>> > > >>>> -----Original Message----- > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > >>>> To: rsyslog-users > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > >>>> > > >>>> Ok Rainer > > >>>> > > >>>> In the clients: > > >>>> > > >>>> OS= opensuse 10.0 > > >>>> rsyslog version: 3.19.7 > > >>>> > > >>>> In the server > > >>>> OS=Debian 4.0 > > >>>> rsyslog version: 3.18.2 > > >>>> > > >>>> I attach the configuration files of the clients and the servers. > > >>>> > > >>>> The remote server is 192.1.4.215. > > >>>> > > >>>> Thanks > > >>>> > > >>>> Rainer Gerhards wrote: > > >>>> > > >>>> > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > need > > >>>>> > > >>>>> > > >>>> configs > > >>>> > > >>>> > > >>>>> and version information to tell you what may be the case. If it > > is > > >>>>> > > >> an > > >> > > >>>> older > > >>>> > > >>>> > > >>>>> v3 version, this may also be a bug in rsyslog. > > >>>>> > > >>>>> HTH > > >>>>> Rainer > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > >>>>>> To: rsyslog at lists.adiscon.com > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > >>>>>> > > >>>>>> Hi!! > > >>>>>> > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I have > > >>>>>> experimented the kaos!! > > >>>>>> > > >>>>>> Accidentaly the central server shutdowns, and one hour later, > > all > > >>>>>> > > >>>>>> > > >>>> the > > >>>> > > >>>> > > >>>>>> 80 > > >>>>>> servers frezze. > > >>>>>> > > >>>>>> Can not access ssh, ping... > > >>>>>> > > >>>>>> I use Debian in central server, and suse in nodes. > > >>>>>> > > >>>>>> Thanks! > > >>>>>> > > >>>>>> -- > > >>>>>> Mikel Jimenez Fernandez > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>>>> > > >>>>>> > > >>>> http://www.irontec.com > > >>>> > > >>>> > > >>>>>> +34 94.404.81.82 > > >>>>>> > > >>>>>> > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>> _______________________________________________ > > >>>>> rsyslog mailing list > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>> http://www.rsyslog.com > > >>>>> > > >>>>> > > >>>>> > > >>>> -- > > >>>> Mikel Jimenez Fernandez > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > >>>> > > >> http://www.irontec.com > > >> > > >>>> +34 94.404.81.82 > > >>>> > > >>>> > > >>>> > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com > > >>> > > >>> > > >> -- > > >> Mikel Jimenez Fernandez > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > http://www.irontec.com > > >> +34 94.404.81.82 > > >> > > >> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com > > >> > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > > > > > -- > > Mikel Jimenez Fernandez > > Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com > > +34 94.404.81.82 > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From rgerhards at hq.adiscon.com Tue Sep 15 12:55:29 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 12:55:29 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDB@GRFEXC.intern.adiscon.com> and a bit more on the philosophy, I knew I wrote it down ;) http://www.rsyslog.com/doc-free_support.html > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > Sent: Tuesday, September 15, 2009 12:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] server frozen when remote logging > > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, > support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and > save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards > > Sent: Tuesday, September 15, 2009 12:52 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > Sent: Tuesday, September 15, 2009 12:48 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] server frozen when remote logging > > > > > > Hi > > > > > > I will delete this line of the config. > > > I will make probes. > > > About the comercial support, I think that this issue is "basic" > for > > > the > > > proper working of a production and seriour enviroment of rsyslog. > > > > Sure, but let me phrase it that way: My interest is finding bugs, > > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in bug reports is *extremely limited*. Still, there is the > > rest of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > > save > > time while doing so. > > > > > > > > In the future, if we want an especialezed support we call you for > > > support, sure!! :) > > > > > > > > > So any solution for this? > > > UDP? > > > > Anyhow, does that mean your config is now error-free and the problem > > still > > persists? > > > > Rainer > > > > > > Rainer Gerhards wrote: > > > >> -----Original Message----- > > > >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >> Sent: Tuesday, September 15, 2009 12:14 PM > > > >> To: rsyslog-users > > > >> Subject: Re: [rsyslog] server frozen when remote logging > > > >> > > > >> But rsyslog starts... > > > >> > > > >> If I use UDP instead TCP? > > > >> > > > > > > > > sure, because the messages are thrown away, at least I think. The > > > point is: > > > > it doesn't make sense to hunt for a problem as long as we know > that > > > the > > > > config is incorrect. Better get the config clean first, then see > if > > > the > > > > problem even persists and then look at it. > > > > > > > > Bluntly and not meant to be embarrassing: I've set aside some > time > > to > > > do this > > > > kind of support, but if you need more "full service" help, it > would > > > probably > > > > be a good idea to purchase one of the support packages. They > exists > > > so that > > > > we can look at issues in depth. This is often an excellent > values, > > as > > > it may > > > > safe you hours and hours of work. And, really, I can't develop > all > > > this and > > > > provide this kind of full-service support ;) > > > > > > > > Rainer > > > > > > > > > > > >> Rainer Gerhards wrote: > > > >> > > > >>> Ok, there are errors in the config files. I've stopped looking > at > > > >>> > > > >> them when I > > > >> > > > >>> saw > > > >>> > > > >>> EST=... > > > >>> > > > >>> ... @@$EST > > > >>> > > > >>> This does not work in rsyslog (yet). Please make sure that your > > > >>> > > > >> configs are > > > >> > > > >>> OK. With the versions you have, you either need to start > rsyslogd > > > >>> interactively in debug mode OR simply look at the syslogd logs > > > (those > > > >>> > > > >> with > > > >> > > > >>> syslog facility). > > > >>> > > > >>> Rainer > > > >>> > > > >>> > > > >>> > > > >>>> -----Original Message----- > > > >>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>> Sent: Tuesday, September 15, 2009 11:56 AM > > > >>>> To: rsyslog-users > > > >>>> Subject: Re: [rsyslog] server frozen when remote logging > > > >>>> > > > >>>> Ok Rainer > > > >>>> > > > >>>> In the clients: > > > >>>> > > > >>>> OS= opensuse 10.0 > > > >>>> rsyslog version: 3.19.7 > > > >>>> > > > >>>> In the server > > > >>>> OS=Debian 4.0 > > > >>>> rsyslog version: 3.18.2 > > > >>>> > > > >>>> I attach the configuration files of the clients and the > servers. > > > >>>> > > > >>>> The remote server is 192.1.4.215. > > > >>>> > > > >>>> Thanks > > > >>>> > > > >>>> Rainer Gerhards wrote: > > > >>>> > > > >>>> > > > >>>>> This sounds like you are overdoing "reliable delivery". But I > > > need > > > >>>>> > > > >>>>> > > > >>>> configs > > > >>>> > > > >>>> > > > >>>>> and version information to tell you what may be the case. If > it > > > is > > > >>>>> > > > >> an > > > >> > > > >>>> older > > > >>>> > > > >>>> > > > >>>>> v3 version, this may also be a bug in rsyslog. > > > >>>>> > > > >>>>> HTH > > > >>>>> Rainer > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > > > >>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez > > > >>>>>> Sent: Tuesday, September 15, 2009 10:53 AM > > > >>>>>> To: rsyslog at lists.adiscon.com > > > >>>>>> Subject: [rsyslog] server frozen when remote logging > > > >>>>>> > > > >>>>>> Hi!! > > > >>>>>> > > > >>>>>> I have 80 servers logging to a centralized rsyslog, and I > have > > > >>>>>> experimented the kaos!! > > > >>>>>> > > > >>>>>> Accidentaly the central server shutdowns, and one hour > later, > > > all > > > >>>>>> > > > >>>>>> > > > >>>> the > > > >>>> > > > >>>> > > > >>>>>> 80 > > > >>>>>> servers frezze. > > > >>>>>> > > > >>>>>> Can not access ssh, ping... > > > >>>>>> > > > >>>>>> I use Debian in central server, and suse in nodes. > > > >>>>>> > > > >>>>>> Thanks! > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Mikel Jimenez Fernandez > > > >>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>>>> > > > >>>>>> > > > >>>> http://www.irontec.com > > > >>>> > > > >>>> > > > >>>>>> +34 94.404.81.82 > > > >>>>>> > > > >>>>>> > > > >>>>>> _______________________________________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>> http://www.rsyslog.com > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> _______________________________________________ > > > >>>>> rsyslog mailing list > > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>> http://www.rsyslog.com > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>> -- > > > >>>> Mikel Jimenez Fernandez > > > >>>> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > >>>> > > > >> http://www.irontec.com > > > >> > > > >>>> +34 94.404.81.82 > > > >>>> > > > >>>> > > > >>>> > > > >>> _______________________________________________ > > > >>> rsyslog mailing list > > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>> http://www.rsyslog.com > > > >>> > > > >>> > > > >> -- > > > >> Mikel Jimenez Fernandez > > > >> Irontec, Internet y Sistemas sobre GNU/LinuX - > > > http://www.irontec.com > > > >> +34 94.404.81.82 > > > >> > > > >> > > > >> _______________________________________________ > > > >> rsyslog mailing list > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> http://www.rsyslog.com > > > >> > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > > > > > > > > > > > -- > > > Mikel Jimenez Fernandez > > > Irontec, Internet y Sistemas sobre GNU/LinuX - > http://www.irontec.com > > > +34 94.404.81.82 > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From mikel at irontec.com Tue Sep 15 13:09:02 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:09:02 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> Message-ID: <4AAF75CE.7090605@irontec.com> Rainer Gerhards wrote: > Used the wrong words ;) Of course, this should read: > > Sure, but let me phrase it that way: My interest is finding bugs, support > questions often lead to that. If there is no bug involved, my personal > interest in support is *extremely limited*. Still, there is the rest of > the community, and they often provide advice. So Adiscon created the > commercial support for corporations that want to have a solution and save > time while doing so. > > And: why is my interest limited? Support to get someone else going > contributes almost nothing back to the project... > Im going to make probes with deleting the config line EST=... When we have coclusion I will tell you, and I will back my concluison to this magic project. :) Thanks > Rainer > > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of Rainer Gerhards >> Sent: Tuesday, September 15, 2009 12:52 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] server frozen when remote logging >> >> >>> -----Original Message----- >>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>> Sent: Tuesday, September 15, 2009 12:48 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] server frozen when remote logging >>> >>> Hi >>> >>> I will delete this line of the config. >>> I will make probes. >>> About the comercial support, I think that this issue is "basic" for >>> the >>> proper working of a production and seriour enviroment of rsyslog. >>> >> Sure, but let me phrase it that way: My interest is finding bugs, >> support >> questions often lead to that. If there is no bug involved, my personal >> interest in bug reports is *extremely limited*. Still, there is the >> rest of >> the community, and they often provide advice. So Adiscon created the >> commercial support for corporations that want to have a solution and >> save >> time while doing so. >> >> >>> In the future, if we want an especialezed support we call you for >>> support, sure!! :) >>> >>> >>> So any solution for this? >>> UDP? >>> >> Anyhow, does that mean your config is now error-free and the problem >> still >> persists? >> >> Rainer >> >>> Rainer Gerhards wrote: >>> >>>>> -----Original Message----- >>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>> Sent: Tuesday, September 15, 2009 12:14 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>> >>>>> But rsyslog starts... >>>>> >>>>> If I use UDP instead TCP? >>>>> >>>>> >>>> sure, because the messages are thrown away, at least I think. The >>>> >>> point is: >>> >>>> it doesn't make sense to hunt for a problem as long as we know that >>>> >>> the >>> >>>> config is incorrect. Better get the config clean first, then see if >>>> >>> the >>> >>>> problem even persists and then look at it. >>>> >>>> Bluntly and not meant to be embarrassing: I've set aside some time >>>> >> to >> >>> do this >>> >>>> kind of support, but if you need more "full service" help, it would >>>> >>> probably >>> >>>> be a good idea to purchase one of the support packages. They exists >>>> >>> so that >>> >>>> we can look at issues in depth. This is often an excellent values, >>>> >> as >> >>> it may >>> >>>> safe you hours and hours of work. And, really, I can't develop all >>>> >>> this and >>> >>>> provide this kind of full-service support ;) >>>> >>>> Rainer >>>> >>>> >>>> >>>>> Rainer Gerhards wrote: >>>>> >>>>> >>>>>> Ok, there are errors in the config files. I've stopped looking at >>>>>> >>>>>> >>>>> them when I >>>>> >>>>> >>>>>> saw >>>>>> >>>>>> EST=... >>>>>> >>>>>> ... @@$EST >>>>>> >>>>>> This does not work in rsyslog (yet). Please make sure that your >>>>>> >>>>>> >>>>> configs are >>>>> >>>>> >>>>>> OK. With the versions you have, you either need to start rsyslogd >>>>>> interactively in debug mode OR simply look at the syslogd logs >>>>>> >>> (those >>> >>>>> with >>>>> >>>>> >>>>>> syslog facility). >>>>>> >>>>>> Rainer >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>> Sent: Tuesday, September 15, 2009 11:56 AM >>>>>>> To: rsyslog-users >>>>>>> Subject: Re: [rsyslog] server frozen when remote logging >>>>>>> >>>>>>> Ok Rainer >>>>>>> >>>>>>> In the clients: >>>>>>> >>>>>>> OS= opensuse 10.0 >>>>>>> rsyslog version: 3.19.7 >>>>>>> >>>>>>> In the server >>>>>>> OS=Debian 4.0 >>>>>>> rsyslog version: 3.18.2 >>>>>>> >>>>>>> I attach the configuration files of the clients and the servers. >>>>>>> >>>>>>> The remote server is 192.1.4.215. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Rainer Gerhards wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> This sounds like you are overdoing "reliable delivery". But I >>>>>>>> >>> need >>> >>>>>>>> >>>>>>> configs >>>>>>> >>>>>>> >>>>>>> >>>>>>>> and version information to tell you what may be the case. If it >>>>>>>> >>> is >>> >>>>> an >>>>> >>>>> >>>>>>> older >>>>>>> >>>>>>> >>>>>>> >>>>>>>> v3 version, this may also be a bug in rsyslog. >>>>>>>> >>>>>>>> HTH >>>>>>>> Rainer >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >>>>>>>>> bounces at lists.adiscon.com] On Behalf Of Mikel Jimenez >>>>>>>>> Sent: Tuesday, September 15, 2009 10:53 AM >>>>>>>>> To: rsyslog at lists.adiscon.com >>>>>>>>> Subject: [rsyslog] server frozen when remote logging >>>>>>>>> >>>>>>>>> Hi!! >>>>>>>>> >>>>>>>>> I have 80 servers logging to a centralized rsyslog, and I have >>>>>>>>> experimented the kaos!! >>>>>>>>> >>>>>>>>> Accidentaly the central server shutdowns, and one hour later, >>>>>>>>> >>> all >>> >>>>>>>>> >>>>>>> the >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> 80 >>>>>>>>> servers frezze. >>>>>>>>> >>>>>>>>> Can not access ssh, ping... >>>>>>>>> >>>>>>>>> I use Debian in central server, and suse in nodes. >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Mikel Jimenez Fernandez >>>>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> http://www.irontec.com >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> +34 94.404.81.82 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Mikel Jimenez Fernandez >>>>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>>>> >>>>>>> >>>>> http://www.irontec.com >>>>> >>>>> >>>>>>> +34 94.404.81.82 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Mikel Jimenez Fernandez >>>>> Irontec, Internet y Sistemas sobre GNU/LinuX - >>>>> >>> http://www.irontec.com >>> >>>>> +34 94.404.81.82 >>>>> >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com >>>> >>>> >>> -- >>> Mikel Jimenez Fernandez >>> Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com >>> +34 94.404.81.82 >>> >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From rgerhards at hq.adiscon.com Tue Sep 15 13:44:06 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 15 Sep 2009 13:44:06 +0200 Subject: [rsyslog] server frozen when remote logging References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> > Rainer Gerhards wrote: > > Used the wrong words ;) Of course, this should read: > > > > Sure, but let me phrase it that way: My interest is finding bugs, > support > > questions often lead to that. If there is no bug involved, my > personal > > interest in support is *extremely limited*. Still, there is the rest > of > > the community, and they often provide advice. So Adiscon created the > > commercial support for corporations that want to have a solution and > save > > time while doing so. > > > > And: why is my interest limited? Support to get someone else going > > contributes almost nothing back to the project... > > > Im going to make probes with deleting the config line EST=... > > When we have coclusion I will tell you, and I will back my concluison > to > this magic project. :) I have taken another look at the log files in the meantime, assuming that $EST were not present ;) However, I do not see anything obviously wrong. But I think I remember there was a condition that caused messages to be processed to slowly. Probably the best idea is to see if the issue persists with the current v3-stable release. If it does, we should go to v4 and if it still persists we need to obtain debug logs. But I think chances are extremely high that the current v3-stable will solve it. however, those forwarding rules to $EST can never have worked, and may actually be overruning the retry mechanism after a while... Rainer From mikel at irontec.com Tue Sep 15 13:46:27 2009 From: mikel at irontec.com (Mikel Jimenez) Date: Tue, 15 Sep 2009 13:46:27 +0200 Subject: [rsyslog] server frozen when remote logging In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> References: <4AAF55D5.20807@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FECA@GRFEXC.intern.adiscon.com> <4AAF64CA.8020002@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED4@GRFEXC.intern.adiscon.com> <4AAF68FC.1040205@irontec.com><9B6E2A8877C38245BFB15CC491A11DA706FED7@GRFEXC.intern.adiscon.com><4AAF70C3.1000800@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FED9@GRFEXC.intern.adiscon.com><9B6E2A8877C38245BFB15CC491A11DA706FEDA@GRFEXC.intern.adiscon.com> <4AAF75CE.7090605@irontec.com> <9B6E2A8877C38245BFB15CC491A11DA706FEDC@GRFEXC.intern.adiscon.com> Message-ID: <4AAF7E93.1080309@irontec.com> Thanks Rainer!! Rainer Gerhards wrote: >> Rainer Gerhards wrote: >> >>> Used the wrong words ;) Of course, this should read: >>> >>> Sure, but let me phrase it that way: My interest is finding bugs, >>> >> support >> >>> questions often lead to that. If there is no bug involved, my >>> >> personal >> >>> interest in support is *extremely limited*. Still, there is the rest >>> >> of >> >>> the community, and they often provide advice. So Adiscon created the >>> commercial support for corporations that want to have a solution and >>> >> save >> >>> time while doing so. >>> >>> And: why is my interest limited? Support to get someone else going >>> contributes almost nothing back to the project... >>> >>> >> Im going to make probes with deleting the config line EST=... >> >> When we have coclusion I will tell you, and I will back my concluison >> to >> this magic project. :) >> > > I have taken another look at the log files in the meantime, assuming that > $EST were not present ;) However, I do not see anything obviously wrong. But > I think I remember there was a condition that caused messages to be processed > to slowly. Probably the best idea is to see if the issue persists with the > current v3-stable release. If it does, we should go to v4 and if it still > persists we need to obtain debug logs. But I think chances are extremely high > that the current v3-stable will solve it. > > however, those forwarding rules to $EST can never have worked, and may > actually be overruning the retry mechanism after a while... > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > -- Mikel Jimenez Fernandez Irontec, Internet y Sistemas sobre GNU/LinuX - http://www.irontec.com +34 94.404.81.82 From tbergfeld at hq.adiscon.com Mon Sep 21 08:12:22 2009 From: tbergfeld at hq.adiscon.com (Tom Bergfeld) Date: Mon, 21 Sep 2009 08:12:22 +0200 Subject: [rsyslog] rsyslog 4.5.3 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FF73@GRFEXC.intern.adiscon.com> Rsyslog 4.5.3, a member of the v4- beta branch, has been released. It is a bug-fixing release. Most importantly, a bug that repeated messages were incorrectly processed by what it could lead to loss of the repeated message content, was fixed. As a side- effect, it could probably also be possible that some segfault occurs (quite unlikely). The root cause was that some counters introduced during the malloc optimizations were not properly duplicated in MsgDup(). Note that repeated message processing is not enabled by default. See Changelog for more details. This is a recommended update for all users of the beta branch. Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-176.phtml Changelog: http://www.rsyslog.com/Article404.phtml As always, feedback is appreciated. Best regards, Tom Bergfeld -- Support ======= Improving rsyslog is costly, but you can help! We are looking for organizations that find rsyslog useful and wish to contribute back. You can contribute by reporting bugs, improve the software, or donate money or equipment. Commercial support contracts for rsyslog are available, and they help finance continued maintenance. Adiscon GmbH, a privately held German company, is currently funding rsyslog development. We are always looking for interesting development projects. For details on how to help, please see http://www.rsyslog.com/doc-how2help.html. From anichols at trumped.org Tue Sep 22 03:50:39 2009 From: anichols at trumped.org (Aaron Nichols) Date: Mon, 21 Sep 2009 19:50:39 -0600 Subject: [rsyslog] Improving filter performance & general performance Message-ID: Hi Everyone, I have rsyslog 4.4.1 chugging along reasonably well but am looking for ways to improve performance and optimize the filter ruleset. Unfortunately I have to create fairly extensive rulesets to filter on hostname, programname, facility, priority, etc. Some log sources generate a high volume of logs (a few Mbytes/sec) across multiple machines and others generate a fairly routine amount of log data - maybe 5 meg per day. Many filters have duplicate conditions for some values but there is always variance. I have tried to order the rules so that the highest volume logs match first and then are discarded. I've included a sample of the rules used for my highest volume logs (names changed to protect the innocent). If there are ways to chain or nest rules so that I can take advantage of matches already made against a log entry to filter it minimally that would be great. For example, most of the below rules filter on the same facility & list of hostnames but look for different values in the 'rawmsg'. If I could filter on the facility & hostname once and then rawmsg to sort to different destinations I'm guessing it would be lower overhead but I don't really know how the processing logic works. Also - if a condition is not met, are other parts of the filter evaluated? For example, if a message was received on local0, would any conditions beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more efficient to filter on the undecoded value syslogfacility vs. syslogfacility-text? I'm looking for suggestions or general techniques for optimizing rule performance under these circumstances. $template XMLFormat, "%syslogtag%%msg%\n" if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'protocolLogRecord' then -/log/syslog/collated/server/protocol.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'messageLogRecord' then -/log/syslog/collated/server/message.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'clientLogRecord' then -/log/syslog/collated/server/client.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local2' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) then -/log/syslog/collated/server/usage.log;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'WAP Page Service ID' then -/log/syslog/collated/server/customer-service;XMLFormat & ~ # discard after match if $syslogfacility-text == 'local1' and ( \ $fromhost startswith 'hosta' or \ $fromhost startswith 'hostb' or \ $fromhost startswith 'hostc' or \ $fromhost startswith 'hostd' \ ) and $rawmsg contains 'locationlogrecord' then -/log/syslog/collated/server/lbs.log;XMLFormat & ~ # discard after match From rgerhards at hq.adiscon.com Tue Sep 22 07:23:43 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 22 Sep 2009 07:23:43 +0200 Subject: [rsyslog] Improving filter performance & general performance References: Message-ID: <9B6E2A8877C38245BFB15CC491A11DA706FFA1@GRFEXC.intern.adiscon.com> Sorry, I am swamped with fixing an important segfault issue we see in one environment, so I do not have time for an more in-depth answer (other list members may have). But I suggest to look into multiple ruleset support, which is in its infancy, but may help. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Aaron Nichols > Sent: Tuesday, September 22, 2009 3:51 AM > To: rsyslog-users > Subject: [rsyslog] Improving filter performance & general performance > > Hi Everyone, > > I have rsyslog 4.4.1 chugging along reasonably well but am > looking for ways > to improve performance and optimize the filter ruleset. > Unfortunately I have > to create fairly extensive rulesets to filter on hostname, > programname, > facility, priority, etc. Some log sources generate a high > volume of logs (a > few Mbytes/sec) across multiple machines and others generate a fairly > routine amount of log data - maybe 5 meg per day. Many filters have > duplicate conditions for some values but there is always > variance. I have > tried to order the rules so that the highest volume logs > match first and > then are discarded. I've included a sample of the rules used > for my highest > volume logs (names changed to protect the innocent). > > If there are ways to chain or nest rules so that I can take > advantage of > matches already made against a log entry to filter it > minimally that would > be great. For example, most of the below rules filter on the > same facility & > list of hostnames but look for different values in the > 'rawmsg'. If I could > filter on the facility & hostname once and then rawmsg to > sort to different > destinations I'm guessing it would be lower overhead but I > don't really know > how the processing logic works. > > Also - if a condition is not met, are other parts of the > filter evaluated? > For example, if a message was received on local0, would any conditions > beyond "if $syslogfacility-text == 'local1'" be evaluated? Is it more > efficient to filter on the undecoded value syslogfacility vs. > syslogfacility-text? > > I'm looking for suggestions or general techniques for optimizing rule > performance under these circumstances. > > $template XMLFormat, "%syslogtag%%msg%\n" > > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'protocolLogRecord' then > -/log/syslog/collated/server/protocol.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'messageLogRecord' then > -/log/syslog/collated/server/message.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'clientLogRecord' then > -/log/syslog/collated/server/client.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local2' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) then -/log/syslog/collated/server/usage.log;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'WAP Page Service ID' then > -/log/syslog/collated/server/customer-service;XMLFormat > & ~ # discard after match > if $syslogfacility-text == 'local1' and ( \ > $fromhost startswith 'hosta' or \ > $fromhost startswith 'hostb' or \ > $fromhost startswith 'hostc' or \ > $fromhost startswith 'hostd' \ > ) and $rawmsg contains 'locationlogrecord' then > -/log/syslog/collated/server/lbs.log;XMLFormat > & ~ # discard after match > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 14:54:38 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 07:54:38 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error Message-ID: <20090925125437.GA28679@it.is.rice.edu> I just looked at our PostgreSQL DB for our rsyslog system and the following error was logged: ERROR: value too long for type character varying(60) STATEMENT: insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, '////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////') The problem is not so much the error but that it stopped logging to the database. I had to restart rsyslog to get it to start logging once more. Should rsyslog check that its values match the schema or should I need to setup a trigger in the DB to handle off-the-wall input. Regards, Ken From rgerhards at hq.adiscon.com Fri Sep 25 15:30:58 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:30:58 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Actually, it should have dropped this message, but that depends on the configuration. In general, rsyslog does not know about the schema. And to be more precise, we are not really talking about rsyslogd itself but rather the output plugin. Every output plugin can perform its own checks. But the best answer probably is to use a trigger ;) Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 2:55 PM > To: rsyslog at lists.adiscon.com > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > I just looked at our PostgreSQL DB for our rsyslog system and > the following error was logged: > > ERROR: value too long for type character varying(60) > STATEMENT: insert into SystemEvents (Message, Facility, > FromHost, Priority, DeviceReportedTime, ReceivedAt, > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > '///////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > ////////////////////////////////////////////////////////////// > /////////////////////////////////////////////////////////////////') > > The problem is not so much the error but that it stopped logging > to the database. I had to restart rsyslog to get it to start logging > once more. Should rsyslog check that its values match the schema or > should I need to setup a trigger in the DB to handle off-the-wall > input. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From ktm at rice.edu Fri Sep 25 15:41:28 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Fri, 25 Sep 2009 08:41:28 -0500 Subject: [rsyslog] rsyslog bug - logging stops after a DB error In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> References: <20090925125437.GA28679@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> Message-ID: <20090925134128.GB28679@it.is.rice.edu> Okay, I will take a look at the output plugin to see where it makes the most sense to fix this. A trigger will always work, but would require every DB to setup and maybe having the plugin perform the truncation would be better. Thank you for the recommendation. Regards, Ken On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > Actually, it should have dropped this message, but that depends on the > configuration. In general, rsyslog does not know about the schema. And to be > more precise, we are not really talking about rsyslogd itself but rather the > output plugin. Every output plugin can perform its own checks. > > But the best answer probably is to use a trigger ;) > > Rainer > > > -----Original Message----- > > From: rsyslog-bounces at lists.adiscon.com > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > Kenneth Marshall > > Sent: Friday, September 25, 2009 2:55 PM > > To: rsyslog at lists.adiscon.com > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > the following error was logged: > > > > ERROR: value too long for type character varying(60) > > STATEMENT: insert into SystemEvents (Message, Facility, > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > '///////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > ////////////////////////////////////////////////////////////// > > /////////////////////////////////////////////////////////////////') > > > > The problem is not so much the error but that it stopped logging > > to the database. I had to restart rsyslog to get it to start logging > > once more. Should rsyslog check that its values match the schema or > > should I need to setup a trigger in the DB to handle off-the-wall > > input. > > > > Regards, > > Ken > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Fri Sep 25 15:43:36 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Fri, 25 Sep 2009 15:43:36 +0200 Subject: [rsyslog] rsyslog bug - logging stops after a DB error References: <20090925125437.GA28679@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA71030BC@GRFEXC.intern.adiscon.com> <20090925134128.GB28679@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030BE@GRFEXC.intern.adiscon.com> Ken, The postgres output is quite simple. You may also want to have a look at omoracle, just to see how flexible an output plugin is (postgres was contributed, as was oracle, btw). Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > Kenneth Marshall > Sent: Friday, September 25, 2009 3:41 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog bug - logging stops after a DB error > > Okay, I will take a look at the output plugin to see where it > makes the most sense to fix this. A trigger will always work, > but would require every DB to setup and maybe having the plugin > perform the truncation would be better. Thank you for the > recommendation. > > Regards, > Ken > > On Fri, Sep 25, 2009 at 03:30:58PM +0200, Rainer Gerhards wrote: > > Actually, it should have dropped this message, but that > depends on the > > configuration. In general, rsyslog does not know about the > schema. And to be > > more precise, we are not really talking about rsyslogd > itself but rather the > > output plugin. Every output plugin can perform its own checks. > > > > But the best answer probably is to use a trigger ;) > > > > Rainer > > > > > -----Original Message----- > > > From: rsyslog-bounces at lists.adiscon.com > > > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > > > Kenneth Marshall > > > Sent: Friday, September 25, 2009 2:55 PM > > > To: rsyslog at lists.adiscon.com > > > Subject: [rsyslog] rsyslog bug - logging stops after a DB error > > > > > > I just looked at our PostgreSQL DB for our rsyslog system and > > > the following error was logged: > > > > > > ERROR: value too long for type character varying(60) > > > STATEMENT: insert into SystemEvents (Message, Facility, > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('', 1, 'mh2.mail.rice.edu', 5, > > > '2009-09-25 00:11:39', '2009-09-25 00:11:39', 1, > > > '///////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > ////////////////////////////////////////////////////////////// > > > > /////////////////////////////////////////////////////////////////') > > > > > > The problem is not so much the error but that it stopped logging > > > to the database. I had to restart rsyslog to get it to > start logging > > > once more. Should rsyslog check that its values match the > schema or > > > should I need to setup a trigger in the DB to handle off-the-wall > > > input. > > > > > > Regards, > > > Ken > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Tue Sep 29 09:59:14 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Tue, 29 Sep 2009 09:59:14 +0200 Subject: [rsyslog] rsyslog 4.5.4 (v4-beta) released Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71030E3@GRFEXC.intern.adiscon.com> Hi all, I have just released 4.5.4, a member of the v4-beta branch. This beta contains an important fix that can lead to a segfault when the gzip output writer is used. It also contains some other fixes. Users of v4-beta are strongly advised to upgrade to that version. ChangeLog: http://www.rsyslog.com/Article406.phtml Download: http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-177.phtml I hope this release is useful, Rainer From rgerhards at hq.adiscon.com Wed Sep 30 16:31:27 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 16:31:27 +0200 Subject: [rsyslog] DNS cache and expiration Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Hi all, I think we had discussed this some time in the past, but I cannot find a record of it. So I thought I ask (again?): After my bughunt looks almost completed, I have come back to implementing the name lookup cache. However, I just found out that obtaining the expiration period of the name lookup seems not to be covered by the "usual" socket calls. Or did I just miss them? Any advise, comments and hints regarding name caching and expiration would deeply be appreciated. Rainer From aland at freeradius.org Wed Sep 30 17:44:26 2009 From: aland at freeradius.org (Alan T DeKok) Date: Wed, 30 Sep 2009 17:44:26 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4AC37CDA.9040707@freeradius.org> Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? You didn't miss anything. They're not available. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. You'll have to use a more powerful DNS library, like adns. Alan DeKok. From aoz.syn at gmail.com Wed Sep 30 17:55:09 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 09:55:09 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? Unfortunately not - most resolver libraries provide only what the programmer usually wants - the symbolic (name) or numeric (IP) result of a query. I've not looked carefully at APIs like res_query, though, and that might bring what you need. > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. This was my greatest concern with doing *good* internal caching in rsyslog - you're almost guaranteed to use and/or implement a large chunk of proper resolver functionality. Depending on how readable you find Perl, the Net::DNS infrastructure may provide some good pointers on implementing custom resolution toolkits. The djbdns 'dnscache' program (and perhaps the djbdns client resolver library itself) could also be good pointers. From rgerhards at hq.adiscon.com Wed Sep 30 18:52:44 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 18:52:44 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Thanks for the quick responses, this was what I feared. In the mean time, I have thought a bit about the design. I think I will start not with the cache, but rather by checking to see if I can move the reverse name resolution further down in the processing flow AND move it to one central location. That makes it easier and more efficient to do caching. One drawback when doing so is that the name resolution potentially happens much later than the message reception. Just think about a busy system, or even one waiting for an upstream server to come online again, that lacks behind some minutes or even some hours. When I do the name resolution in the backend thread, the reverse entries may have changed since the message was received :( For many cases, this may be acceptable, for some not. I will probably need to at least define a config value which enables direct queries vs. deferred ones. Any comments on that issue would also be most welcome. Thanks, Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- > bounces at lists.adiscon.com] On Behalf Of RB > Sent: Wednesday, September 30, 2009 5:55 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards > wrote: > > After my bughunt looks almost completed, I have come back to > implementing the > > name lookup cache. However, I just found out that obtaining the > expiration > > period of the name lookup seems not to be covered by the "usual" > socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration > would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com From david at lang.hm Wed Sep 30 19:00:58 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:00:58 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Hi all, > > I think we had discussed this some time in the past, but I cannot find a > record of it. So I thought I ask (again?): > > After my bughunt looks almost completed, I have come back to implementing the > name lookup cache. However, I just found out that obtaining the expiration > period of the name lookup seems not to be covered by the "usual" socket > calls. Or did I just miss them? no they don't, they are name resolution calls, not DNS calls. there are many sources of name resolution (/etc/hosts, LDAP, wins, NIS, etc) and most of them do not have a concept of expiration, and those that do have specific rules for how expiration works (for DNS you have a time after which you are supposed to try and re-resolve it, but can continue to use the name, and a different time after which you are not supposed to use the name for example) > Any advise, comments and hints regarding name caching and expiration would > deeply be appreciated. going back to basics here. Why is this feature desired? why not just use a caching nameserver listening on localhost? 1. doing a name lookup can cause lost logs as you saw when you were doing testing recently, doing name lookups on each received UDP log can cause you to loose log messages when the OS can no longer queue them up. 2. throughput in a high volume site, the cost of doing a name lookup for each log message can be high enough to be a problem. even a local nameserver can be expensive if you are dealing with 10's of thousands of messages/sec what if you were to move the name resolution from the input module to the output module? that would solve problem #1 immediatly by just eliminating any lookups as the messages are received. note: this may not be possible due to name based rules for what hosts to accept logs from, although the answer here may be to lookup the names when you startup and do the filtering by IP while running. if you delay the name resolution until the output module, you may be able to only do it if the output module needs it (if it uses a name property in the template or ruleset), and if it doesn't you skip the work entirely. in anything short of a very high volume site a local caching nameserver will satisfy the throughput issue nicely (especially if the name resolution is delayed to the output as I mentioned above). in a high volume site I really think that it can be good enough to just throw away the name cache when you do a HUP. a high volume site is going to be doing a HUP on a frequent basis anyway to rotate the logs. This avoids a LOT of overhead and complications in managing expirations. In my site I send rsyslog a HUP every 5 min currently (and have some cases where I plan to change this to every 1 min in the near future) David Lang From david at lang.hm Wed Sep 30 19:05:04 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:05:04 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <9B6E2A8877C38245BFB15CC491A11DA710310D@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the quick responses, this was what I feared. > > In the mean time, I have thought a bit about the design. I think I will start > not with the cache, but rather by checking to see if I can move the reverse > name resolution further down in the processing flow AND move it to one > central location. That makes it easier and more efficient to do caching. > > One drawback when doing so is that the name resolution potentially happens > much later than the message reception. Just think about a busy system, or > even one waiting for an upstream server to come online again, that lacks > behind some minutes or even some hours. When I do the name resolution in the > backend thread, the reverse entries may have changed since the message was > received :( > > For many cases, this may be acceptable, for some not. I will probably need to > at least define a config value which enables direct queries vs. deferred > ones. > > Any comments on that issue would also be most welcome. it is actually pretty unusual for the source of logs to change it's name. remember that DNS takes time to propogate changes, so even if you do queries immediatly the data may be out of date if you are in an environemnt where it changes. David Lang > Thanks, > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog- >> bounces at lists.adiscon.com] On Behalf Of RB >> Sent: Wednesday, September 30, 2009 5:55 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards >> wrote: >>> After my bughunt looks almost completed, I have come back to >> implementing the >>> name lookup cache. However, I just found out that obtaining the >> expiration >>> period of the name lookup seems not to be covered by the "usual" >> socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration >> would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From mbiebl at gmail.com Wed Sep 30 19:36:20 2009 From: mbiebl at gmail.com (Michael Biebl) Date: Wed, 30 Sep 2009 19:36:20 +0200 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: 2009/9/30 : > > going back to basics here. > > Why is this feature desired? why not just use a caching nameserver > listening on localhost? > Was wondering about this myself. There are small caching nameservers like dnsmasq which will do all the hard work for you. Rainer, have you evaluated such an option? Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? From david at lang.hm Wed Sep 30 19:52:07 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 10:52:07 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Michael Biebl wrote: > 2009/9/30 : >> >> going back to basics here. >> >> Why is this feature desired? why not just use a caching nameserver >> listening on localhost? >> > > Was wondering about this myself. There are small caching nameservers > like dnsmasq which will do all the hard work for you. > Rainer, have you evaluated such an option? by itself, this would not solve the problems 1. with the current situation where the lookups are done as the message is being received, the time taken to do the lookup (especially in the case where the lookup is not yet in the cache) can take long enough that log messages get lost 2. when you are talking message rates of 100K logs/sec and up the overhead of doing a DNS query, even to a server running on localhost that has the info cached in it, becomes a signficant amount of the total time you have to process that message before the next message arrives. David Lang From ktm at rice.edu Wed Sep 30 19:54:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 12:54:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> Message-ID: <20090930175456.GD6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > > After my bughunt looks almost completed, I have come back to implementing the > > name lookup cache. However, I just found out that obtaining the expiration > > period of the name lookup seems not to be covered by the "usual" socket > > calls. Or did I just miss them? > > Unfortunately not - most resolver libraries provide only what the > programmer usually wants - the symbolic (name) or numeric (IP) result > of a query. I've not looked carefully at APIs like res_query, though, > and that might bring what you need. > > > Any advise, comments and hints regarding name caching and expiration would > > deeply be appreciated. > > This was my greatest concern with doing *good* internal caching in > rsyslog - you're almost guaranteed to use and/or implement a large > chunk of proper resolver functionality. Depending on how readable you > find Perl, the Net::DNS infrastructure may provide some good pointers > on implementing custom resolution toolkits. The djbdns 'dnscache' > program (and perhaps the djbdns client resolver library itself) could > also be good pointers. I do not think that the goal of this feature in rsyslog is to re-implement resolver functionality but to provide a fast-path mechanism to map IP addresses to names for the purposes of logging error messages. As such, pretty much the only piece that needs to be tracked within rsyslog is the TTL for the entry and the ip -> name mapping. A thread would be responsible for expiring entries from the cache (or refreshing the timeout) after validating the correctness of the mapping. I think the DNS lookups should be handled by a good resolver like pdns-recursor, djbdns,... The goal here is to allow names in the log entries and not just IP addresses and in a very high performance logging environment. Regards, Ken From david at lang.hm Wed Sep 30 20:15:42 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:15:42 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: >>> After my bughunt looks almost completed, I have come back to implementing the >>> name lookup cache. However, I just found out that obtaining the expiration >>> period of the name lookup seems not to be covered by the "usual" socket >>> calls. Or did I just miss them? >> >> Unfortunately not - most resolver libraries provide only what the >> programmer usually wants - the symbolic (name) or numeric (IP) result >> of a query. I've not looked carefully at APIs like res_query, though, >> and that might bring what you need. >> >>> Any advise, comments and hints regarding name caching and expiration would >>> deeply be appreciated. >> >> This was my greatest concern with doing *good* internal caching in >> rsyslog - you're almost guaranteed to use and/or implement a large >> chunk of proper resolver functionality. Depending on how readable you >> find Perl, the Net::DNS infrastructure may provide some good pointers >> on implementing custom resolution toolkits. The djbdns 'dnscache' >> program (and perhaps the djbdns client resolver library itself) could >> also be good pointers. > > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. As such, pretty much the only piece that needs to > be tracked within rsyslog is the TTL for the entry and the ip -> > name mapping. A thread would be responsible for expiring entries > from the cache (or refreshing the timeout) after validating the > correctness of the mapping. I think the DNS lookups should be > handled by a good resolver like pdns-recursor, djbdns,... The > goal here is to allow names in the log entries and not just IP > addresses and in a very high performance logging environment. the trouble is that doing _proper_ TTL expiration isn't as simple as it sounds. and if you are willing to back away from 'proper' expiration to something that will work in practice, why not go much further (as I have detailed in the other messages) David Lang From aoz.syn at gmail.com Wed Sep 30 20:18:37 2009 From: aoz.syn at gmail.com (RB) Date: Wed, 30 Sep 2009 12:18:37 -0600 Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930175456.GD6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: > I do not think that the goal of this feature in rsyslog is to > re-implement resolver functionality but to provide a fast-path > mechanism to map IP addresses to names for the purposes of logging > error messages. Although I agree with your assessment of the goal, the only difference I see between the two is wording semantics. An RFC-compliant DNS cache will, for all intents and purposes, look an awful lot like any other caching, recursive-only DNS resolver (like dnscache). The only major difference would be that it would accept requests via an API as opposed to through a socket interface. Regardless, I have to sit on the same side as David and Michael - in very high-performance environments, I doubt the difference between an internal cache and an external one is going to be significant. From ktm at rice.edu Wed Sep 30 20:25:56 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Wed, 30 Sep 2009 13:25:56 -0500 Subject: [rsyslog] DNS cache and expiration In-Reply-To: References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> Message-ID: <20090930182556.GE6749@it.is.rice.edu> On Wed, Sep 30, 2009 at 11:15:42AM -0700, david at lang.hm wrote: > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > > On Wed, Sep 30, 2009 at 09:55:09AM -0600, RB wrote: > >> On Wed, Sep 30, 2009 at 08:31, Rainer Gerhards wrote: > >>> After my bughunt looks almost completed, I have come back to implementing the > >>> name lookup cache. However, I just found out that obtaining the expiration > >>> period of the name lookup seems not to be covered by the "usual" socket > >>> calls. Or did I just miss them? > >> > >> Unfortunately not - most resolver libraries provide only what the > >> programmer usually wants - the symbolic (name) or numeric (IP) result > >> of a query. I've not looked carefully at APIs like res_query, though, > >> and that might bring what you need. > >> > >>> Any advise, comments and hints regarding name caching and expiration would > >>> deeply be appreciated. > >> > >> This was my greatest concern with doing *good* internal caching in > >> rsyslog - you're almost guaranteed to use and/or implement a large > >> chunk of proper resolver functionality. Depending on how readable you > >> find Perl, the Net::DNS infrastructure may provide some good pointers > >> on implementing custom resolution toolkits. The djbdns 'dnscache' > >> program (and perhaps the djbdns client resolver library itself) could > >> also be good pointers. > > > > I do not think that the goal of this feature in rsyslog is to > > re-implement resolver functionality but to provide a fast-path > > mechanism to map IP addresses to names for the purposes of logging > > error messages. As such, pretty much the only piece that needs to > > be tracked within rsyslog is the TTL for the entry and the ip -> > > name mapping. A thread would be responsible for expiring entries > > from the cache (or refreshing the timeout) after validating the > > correctness of the mapping. I think the DNS lookups should be > > handled by a good resolver like pdns-recursor, djbdns,... The > > goal here is to allow names in the log entries and not just IP > > addresses and in a very high performance logging environment. > > the trouble is that doing _proper_ TTL expiration isn't as simple as it > sounds. > > and if you are willing to back away from 'proper' expiration to something > that will work in practice, why not go much further (as I have detailed in > the other messages) > > David Lang I agree. I only mention TTL values as a reasonable upperbound of the refresh check. The advantage is to remove the large pause in logging due to a DNS refresh after a HUP to rsyslog, if that were the only method to flush/refresh. Like you mention, the IPs/names of systems being logged change rarely so we should tune this for speed and not worry about expiration correctness. I do not agree so other statements that an external cache will perform as well as an internal cache. Too many software products that I work with have needed exactly that functionality to support very high levels of performance. Regards, Ken From david at lang.hm Wed Sep 30 20:51:28 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:51:28 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <4255c2570909301118kf069e03n6d1a73076d20a891@mail.gmail.com> Message-ID: On Wed, 30 Sep 2009, RB wrote: > On Wed, Sep 30, 2009 at 11:54, Kenneth Marshall wrote: >> I do not think that the goal of this feature in rsyslog is to >> re-implement resolver functionality but to provide a fast-path >> mechanism to map IP addresses to names for the purposes of logging >> error messages. > > Although I agree with your assessment of the goal, the only difference > I see between the two is wording semantics. An RFC-compliant DNS > cache will, for all intents and purposes, look an awful lot like any > other caching, recursive-only DNS resolver (like dnscache). The only > major difference would be that it would accept requests via an API as > opposed to through a socket interface. > > Regardless, I have to sit on the same side as David and Michael - in > very high-performance environments, I doubt the difference between an > internal cache and an external one is going to be significant. actually, I am thinking that in a high-performance environment, the difference between an internal name cache and an external one _is_ significant I just don't think the internal one should be a DNS RFC complient one. if you put everything in /etc/hosts it is faster than doing a query against a local caching server, but it's still significantly slower than looking it up in memory. remember that when you make the gethostbyname() call it has to do a lot of checking to see which name resolver libraries you have configured (which includes checking for the existance of multiple files), then call them in order until it finds the name. if you do a strace of this sometime you will see how much stuff goes on under the covers. skipping all of this is significant at high log rates. David Lang From david at lang.hm Wed Sep 30 20:54:12 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 11:54:12 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <20090930182556.GE6749@it.is.rice.edu> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com> <4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com> <20090930175456.GD6749@it.is.rice.edu> <20090930182556.GE6749@it.is.rice.edu> Message-ID: On Wed, 30 Sep 2009, Kenneth Marshall wrote: >>> mechanism to map IP addresses to names for the purposes of logging >>> error messages. As such, pretty much the only piece that needs to >>> be tracked within rsyslog is the TTL for the entry and the ip -> >>> name mapping. A thread would be responsible for expiring entries >>> from the cache (or refreshing the timeout) after validating the >>> correctness of the mapping. I think the DNS lookups should be >>> handled by a good resolver like pdns-recursor, djbdns,... The >>> goal here is to allow names in the log entries and not just IP >>> addresses and in a very high performance logging environment. >> >> the trouble is that doing _proper_ TTL expiration isn't as simple as it >> sounds. >> >> and if you are willing to back away from 'proper' expiration to something >> that will work in practice, why not go much further (as I have detailed in >> the other messages) >> >> David Lang > > I agree. I only mention TTL values as a reasonable upperbound of the > refresh check. The advantage is to remove the large pause in logging > due to a DNS refresh after a HUP to rsyslog, if that were the only > method to flush/refresh. Like you mention, the IPs/names of systems > being logged change rarely so we should tune this for speed and > not worry about expiration correctness. I do not agree so other > statements that an external cache will perform as well as an internal > cache. Too many software products that I work with have needed exactly > that functionality to support very high levels of performance. actually, you could have the cache be configurable in three modes 1. no caching 2. blank the cache on HUP 3. never blank the cache (i.e. require a full restart to clear it) David Lang From rgerhards at hq.adiscon.com Wed Sep 30 21:45:00 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:45:00 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Thanks for the good discussion. I am lacking somewhat behind, but will review it in depth tomorrow morning. I just wanted to stress the point that an external cache does not really help, much for the reason David mentioned: if you process messages at very high data rates, the context switch overhead involved with any external solution is extremely costly. Also, in the usual cases, I may do several million queries within a few seconds for just a handful of hosts. With an internal cache, the overhead in doing so is very minimal. With an external solution, the overhead in calling the external cache causes a lot of performance degredation, what in the case of UDP also implies (heavy!) message loss. Rainer > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 8:54 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Kenneth Marshall wrote: > > >>> mechanism to map IP addresses to names for the purposes of logging > >>> error messages. As such, pretty much the only piece that needs to > >>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>> name mapping. A thread would be responsible for expiring entries > >>> from the cache (or refreshing the timeout) after validating the > >>> correctness of the mapping. I think the DNS lookups should be > >>> handled by a good resolver like pdns-recursor, djbdns,... The > >>> goal here is to allow names in the log entries and not just IP > >>> addresses and in a very high performance logging environment. > >> > >> the trouble is that doing _proper_ TTL expiration isn't as > simple as it > >> sounds. > >> > >> and if you are willing to back away from 'proper' > expiration to something > >> that will work in practice, why not go much further (as I > have detailed in > >> the other messages) > >> > >> David Lang > > > > I agree. I only mention TTL values as a reasonable upperbound of the > > refresh check. The advantage is to remove the large pause in logging > > due to a DNS refresh after a HUP to rsyslog, if that were the only > > method to flush/refresh. Like you mention, the IPs/names of systems > > being logged change rarely so we should tune this for speed and > > not worry about expiration correctness. I do not agree so other > > statements that an external cache will perform as well as > an internal > > cache. Too many software products that I work with have > needed exactly > > that functionality to support very high levels of performance. > > actually, you could have the cache be configurable in three modes > > 1. no caching > > 2. blank the cache on HUP > > 3. never blank the cache (i.e. require a full restart to clear it) > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From david at lang.hm Wed Sep 30 21:53:00 2009 From: david at lang.hm (david at lang.hm) Date: Wed, 30 Sep 2009 12:53:00 -0700 (PDT) Subject: [rsyslog] DNS cache and expiration In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu> <9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: On Wed, 30 Sep 2009, Rainer Gerhards wrote: > Thanks for the good discussion. I am lacking somewhat behind, but will review > it in depth tomorrow morning. > > I just wanted to stress the point that an external cache does not really > help, much for the reason David mentioned: if you process messages at very > high data rates, the context switch overhead involved with any external > solution is extremely costly. Also, in the usual cases, I may do several > million queries within a few seconds for just a handful of hosts. With an > internal cache, the overhead in doing so is very minimal. With an external > solution, the overhead in calling the external cache causes a lot of > performance degredation, what in the case of UDP also implies (heavy!) > message loss. the message loss problem with UDP will not be solved completely by an internal cache. when the source is not in the cache and you have to go out to find it the lookup can take several seconds. moving the lookup out of the input module and into the output module would address this, anything else would leave you with losses as the cache gets populated. David Lang > Rainer > >> -----Original Message----- >> From: rsyslog-bounces at lists.adiscon.com >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm >> Sent: Wednesday, September 30, 2009 8:54 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] DNS cache and expiration >> >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: >> >>>>> mechanism to map IP addresses to names for the purposes of logging >>>>> error messages. As such, pretty much the only piece that needs to >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> >>>>> name mapping. A thread would be responsible for expiring entries >>>>> from the cache (or refreshing the timeout) after validating the >>>>> correctness of the mapping. I think the DNS lookups should be >>>>> handled by a good resolver like pdns-recursor, djbdns,... The >>>>> goal here is to allow names in the log entries and not just IP >>>>> addresses and in a very high performance logging environment. >>>> >>>> the trouble is that doing _proper_ TTL expiration isn't as >> simple as it >>>> sounds. >>>> >>>> and if you are willing to back away from 'proper' >> expiration to something >>>> that will work in practice, why not go much further (as I >> have detailed in >>>> the other messages) >>>> >>>> David Lang >>> >>> I agree. I only mention TTL values as a reasonable upperbound of the >>> refresh check. The advantage is to remove the large pause in logging >>> due to a DNS refresh after a HUP to rsyslog, if that were the only >>> method to flush/refresh. Like you mention, the IPs/names of systems >>> being logged change rarely so we should tune this for speed and >>> not worry about expiration correctness. I do not agree so other >>> statements that an external cache will perform as well as >> an internal >>> cache. Too many software products that I work with have >> needed exactly >>> that functionality to support very high levels of performance. >> >> actually, you could have the cache be configurable in three modes >> >> 1. no caching >> >> 2. blank the cache on HUP >> >> 3. never blank the cache (i.e. require a full restart to clear it) >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > From rgerhards at hq.adiscon.com Wed Sep 30 21:56:33 2009 From: rgerhards at hq.adiscon.com (Rainer Gerhards) Date: Wed, 30 Sep 2009 21:56:33 +0200 Subject: [rsyslog] DNS cache and expiration References: <9B6E2A8877C38245BFB15CC491A11DA7103108@GRFEXC.intern.adiscon.com><4255c2570909300855o5ad64302o1b54539fca4781b0@mail.gmail.com><20090930175456.GD6749@it.is.rice.edu><20090930182556.GE6749@it.is.rice.edu><9B6E2A8877C38245BFB15CC491A11DA710310F@GRFEXC.intern.adiscon.com> Message-ID: <9B6E2A8877C38245BFB15CC491A11DA7103110@GRFEXC.intern.adiscon.com> > -----Original Message----- > From: rsyslog-bounces at lists.adiscon.com > [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of david at lang.hm > Sent: Wednesday, September 30, 2009 9:53 PM > To: rsyslog-users > Subject: Re: [rsyslog] DNS cache and expiration > > On Wed, 30 Sep 2009, Rainer Gerhards wrote: > > > Thanks for the good discussion. I am lacking somewhat > behind, but will review > > it in depth tomorrow morning. > > > > I just wanted to stress the point that an external cache > does not really > > help, much for the reason David mentioned: if you process > messages at very > > high data rates, the context switch overhead involved with > any external > > solution is extremely costly. Also, in the usual cases, I > may do several > > million queries within a few seconds for just a handful of > hosts. With an > > internal cache, the overhead in doing so is very minimal. > With an external > > solution, the overhead in calling the external cache causes a lot of > > performance degredation, what in the case of UDP also > implies (heavy!) > > message loss. > > the message loss problem with UDP will not be solved completely by an > internal cache. when the source is not in the cache and you > have to go out > to find it the lookup can take several seconds. > > moving the lookup out of the input module and into the output > module would > address this, anything else would leave you with losses as > the cache gets > populated. That's right and that's one reason why I intend to move this (optionally) over to the "backend" processing. However, even that does not completely solve the message loss problem, as we, in extreme cases, may loose messages when the queue is full - and for a myriad of other reasons, like routers discarding frames and such. Of course, you know that, but I'd like to mention if for those folks that at some time find our conversation via Google ;) Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-bounces at lists.adiscon.com > >> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of > david at lang.hm > >> Sent: Wednesday, September 30, 2009 8:54 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] DNS cache and expiration > >> > >> On Wed, 30 Sep 2009, Kenneth Marshall wrote: > >> > >>>>> mechanism to map IP addresses to names for the purposes > of logging > >>>>> error messages. As such, pretty much the only piece > that needs to > >>>>> be tracked within rsyslog is the TTL for the entry and the ip -> > >>>>> name mapping. A thread would be responsible for expiring entries > >>>>> from the cache (or refreshing the timeout) after validating the > >>>>> correctness of the mapping. I think the DNS lookups should be > >>>>> handled by a good resolver like pdns-recursor, djbdns,... The > >>>>> goal here is to allow names in the log entries and not just IP > >>>>> addresses and in a very high performance logging environment. > >>>> > >>>> the trouble is that doing _proper_ TTL expiration isn't as > >> simple as it > >>>> sounds. > >>>> > >>>> and if you are willing to back away from 'proper' > >> expiration to something > >>>> that will work in practice, why not go much further (as I > >> have detailed in > >>>> the other messages) > >>>> > >>>> David Lang > >>> > >>> I agree. I only mention TTL values as a reasonable > upperbound of the > >>> refresh check. The advantage is to remove the large pause > in logging > >>> due to a DNS refresh after a HUP to rsyslog, if that were the only > >>> method to flush/refresh. Like you mention, the IPs/names > of systems > >>> being logged change rarely so we should tune this for speed and > >>> not worry about expiration correctness. I do not agree so other > >>> statements that an external cache will perform as well as > >> an internal > >>> cache. Too many software products that I work with have > >> needed exactly > >>> that functionality to support very high levels of performance. > >> > >> actually, you could have the cache be configurable in three modes > >> > >> 1. no caching > >> > >> 2. blank the cache on HUP > >> > >> 3. never blank the cache (i.e. require a full restart to clear it) > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com >